Skip to content

armgit / qm

Public Imported from GitHub
0

Multiplayer agent harness for work.

feat: verify admin models before enabling them

Josh France3691a360bfdf
diff --git a/docs/admin-model-verification.md b/docs/admin-model-verification.md
new file mode 100644
index 000000000..606771a6c
--- /dev/null
+++ b/docs/admin-model-verification.md
@@ -0,0 +1,19 @@
+# Admin-managed model verification
+
+In the admin model registry, **Verify and enable** checks a proposed OpenAI or Anthropic model before saving it. Only organization administrators can run the check. API clients must explicitly consent by including `verify: true` with the registry PUT; it is not part of the model definition.
+
+The check uses Pi's model runtime, the selected template's protocol, the exact model ID, and the organization's serving credential and endpoint configuration. It sends only synthetic text and a synthetic tool schema; no tools are executed or conversation data sent. It requires a completed, nonempty text response. The output cap is the lesser of 128 tokens and the model's configured limit. Fast-mode definitions require ordinary and fast checks. Verification has a 15-second deadline; provider retries may apply. These requests may incur provider charges.
+
+A failed check does not create or save the proposed definition. Failed edits preserve the previously saved settings. A failed recheck of the same saved settings removes their earlier verification. Errors distinguish missing credentials, access denial, unavailable models, quota/rate limits, rejected configurations, timeouts, and other provider failures without returning raw provider details.
+
+Successful checks store a timestamp and a private fingerprint of the definition, resolved template metadata, and serving credential/configuration revision. Each durable refresh checks that fingerprint without making another provider request. Relevant changes invalidate verification and remove the model from selectable choices until an administrator verifies it again. Previously stored definitions without verification are unavailable until checked. Existing selected IDs are preserved rather than silently replaced.
+
+Slow probes run outside the registry write lock. Before saving, the store rechecks the definition, serving configuration, and prior stored revision. A late result cannot undo a deletion or overwrite another edit.
+
+## What verification does not promise
+
+- The timestamp records a successful check at that time, not permanent provider access. Entitlements, quota, or provider behavior can subsequently change.
+- Organization verification does not certify any user's personal API key or subscription. Existing credential boundaries still apply.
+- A small probe does not certify prices, the maximum advertised context/output limits, image support, every reasoning level, or correct tool execution. Check provider documentation for those fields.
+- Configuring a model does not add support for a new protocol or native harness. Admin-managed API models remain Pi-only in normal use.
+- Live provider compatibility requires running the check against that provider. Development tests use local streaming provider fixtures.
diff --git a/plugins/admin/public/index.html b/plugins/admin/public/index.html
index 4be72e354..dee8ea46d 100644
--- a/plugins/admin/public/index.html
+++ b/plugins/admin/public/index.html
@@ -5062,8 +5062,15 @@
</label>
</details>
</div>
+ <p class="muted" id="model-registry-verification-notice">
+ Verify and enable sends a synthetic connection check using the organization's serving credentials. It
+ may incur a small provider charge (up to 128 output tokens per request; an additional check with fast
+ mode; provider retries may apply). A successful check confirms access now, not pricing, maximum context,
+ every capability, or personal-key access. Failed checks do not save or enable the new definition.
+ Existing saved settings remain unchanged. A failed recheck of those settings removes their verification.
+ </p>
<div class="foot">
- <button class="primary" id="model-registry-save">Save model</button
+ <button class="primary" id="model-registry-save">Verify and enable</button
><button id="model-registry-new">New model</button><span class="status" id="st-model-registry"></span>
</div>
</section>
@@ -9004,10 +9011,18 @@
rows.replaceChildren();
const enabled = response.data.models.filter((m) => !m.disabled);
$("model-registry-empty").hidden = enabled.length > 0;
- for (const { spec, unavailableReason } of enabled) {
+ for (const { spec, unavailableReason, verifiedAt } of enabled) {
const row = document.createElement("tr");
for (const value of [
- (spec.name || spec.id) + " (" + spec.id + ")" + (unavailableReason ? " — " + unavailableReason : ""),
+ (spec.name || spec.id) +
+ " (" +
+ spec.id +
+ ")" +
+ (unavailableReason
+ ? " — " + unavailableReason
+ : verifiedAt
+ ? " — Verified with organization credentials " + new Date(verifiedAt).toLocaleString()
+ : " — Not verified"),
spec.provider || "Unknown",
spec.template || "Missing",
]) {
@@ -9064,15 +9079,25 @@
for (const field of ["base", "webui", "auxiliary", "fastMode"])
spec[field] = $("model-registry-" + field).checked;
button.disabled = true;
- const result = await api("PUT", "/api/model-registry/" + encodeURIComponent(spec.id), spec);
+ button.textContent = "Verifying…";
+ setStatus("st-model-registry", "Checking the model with organization credentials…", "");
+ const result = await api("PUT", "/api/model-registry/" + encodeURIComponent(spec.id), {
+ ...spec,
+ verify: true,
+ });
if (!result.ok) throw new Error(result.data?.message || "Could not save model.");
await loadOnboarding();
editRegistryModel(spec);
- setStatus("st-model-registry", "Model saved. New turns use this definition.", "ok");
+ setStatus(
+ "st-model-registry",
+ "Model verified and enabled with organization credentials. Personal-key access may differ.",
+ "ok",
+ );
} catch (error) {
setStatus("st-model-registry", error.message, "err");
} finally {
button.disabled = false;
+ button.textContent = "Verify and enable";
}
};
diff --git a/plugins/admin/test/onboarding-view.test.ts b/plugins/admin/test/onboarding-view.test.ts
index ae9bf23c6..c22285976 100644
--- a/plugins/admin/test/onboarding-view.test.ts
+++ b/plugins/admin/test/onboarding-view.test.ts
@@ -133,3 +133,17 @@ test("?view=onboarding resolves to the onboarding view", () => {
test("unknown views still fall back to the default view", () => {
assert.equal(resolveView("/admin/no-such-view", ""), "history");
});
+
+test("model registry verification makes charges and credential scope explicit", () => {
+ assert.match(html, /id="model-registry-save">Verify and enable/);
+ const notice = slice('id="model-registry-verification-notice"', "</p>");
+ assert.match(notice, /provider charge/);
+ assert.match(notice, /personal-key access/);
+ const save = slice('$("model-registry-save").onclick', "let customProvidersLoaded");
+ assert.match(save, /verify: true/);
+ assert.match(save, /button.disabled = true/);
+ assert.match(save, /Verifying…/);
+ assert.match(save, /finally/);
+ assert.match(save, /button.disabled = false/);
+ assert.match(html, /Verified with organization credentials/);
+});
diff --git a/src/api/routes/admin/model-registry.ts b/src/api/routes/admin/model-registry.ts
index 05f87a149..545a1f74a 100644
--- a/src/api/routes/admin/model-registry.ts
+++ b/src/api/routes/admin/model-registry.ts
@@ -1,3 +1,4 @@
+import { ModelVerificationError } from "../../../model/model-verification.ts";
import { MODEL_REGISTRY, safeModelMetadata } from "../../../model/pi-models.ts";
import { errMessage } from "../../../util/errors.ts";
import { sendJson } from "../../http.ts";
@@ -20,15 +21,37 @@ export async function modelRegistry(ctx: ApiCtx): Promise<void> {
}
const id = ctx.params.model;
if (!id) return sendJson(ctx.res, 400, { error: "bad_request" });
+ let verification: { verifiedAt: number; verificationScope: "organization" } | undefined;
try {
if (ctx.method === "DELETE") {
if (!(await store.delete(id, actor.id))) return sendJson(ctx.res, 404, { error: "not_found" });
} else {
if (!isObj(ctx.body) || (ctx.body.id !== undefined && ctx.body.id !== id))
return sendJson(ctx.res, 400, { error: "bad_request" });
- await store.upsert({ ...ctx.body, id }, actor.id);
+ const { verify, ...spec } = ctx.body;
+ if (verify !== true)
+ return sendJson(ctx.res, 400, {
+ error: "verification_consent_required",
+ message:
+ "Verify and enable sends a small billable synthetic request with organization credentials. Set verify: true to continue.",
+ });
+ verification = await store.upsert({ ...spec, id }, actor.id);
}
} catch (error) {
+ if (error instanceof ModelVerificationError) {
+ audit(ctx.deps, {
+ principalId: actor.id,
+ action: "model-registry.verification-failed",
+ resource: id,
+ scopeLabel: scope,
+ });
+ await store.refresh();
+ return sendJson(
+ ctx.res,
+ ["configuration_conflict", "changed_during_verification"].includes(error.code) ? 409 : 422,
+ { error: error.code, message: error.message },
+ );
+ }
return sendJson(ctx.res, 400, { error: "bad_request", message: errMessage(error) });
}
audit(ctx.deps, {
@@ -38,5 +61,5 @@ export async function modelRegistry(ctx: ApiCtx): Promise<void> {
scopeLabel: scope,
});
await store.refresh();
- return sendJson(ctx.res, 200, { ok: true });
+ return sendJson(ctx.res, 200, { ok: true, ...verification });
}
diff --git a/src/harness/pi-harness.ts b/src/harness/pi-harness.ts
index 194350fa2..600f604a0 100644
--- a/src/harness/pi-harness.ts
+++ b/src/harness/pi-harness.ts
@@ -1,3 +1,4 @@
+import { Type } from "typebox";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -1219,6 +1220,44 @@ export async function oneShot(
}
}
+export async function probeModel(
+ model: Model<Api>,
+ keys: ProviderKeys,
+ signal: AbortSignal,
+ fastMode = false,
+ modelGateway?: ModelGatewayTransportConfig,
+): Promise<void> {
+ const runtime = await buildModelRuntime(keys, modelGateway);
+ signal.throwIfAborted();
+ const fastHeader = fastMode && !modelGateway?.models[model.id];
+ const candidate = fastHeader ? withFastModeHeaders(model) : model;
+ const response = await runtime
+ .streamSimple(
+ candidate,
+ {
+ systemPrompt: "This is a connection check. Reply OK. Do not call tools.",
+ messages: [{ role: "user", content: "Reply OK.", timestamp: Date.now() }],
+ tools: [
+ {
+ name: "connection_check",
+ description: "A synthetic tool for verifying request compatibility. Do not call it.",
+ parameters: Type.Object({}),
+ },
+ ],
+ },
+ {
+ maxTokens: Math.min(128, model.maxTokens),
+ signal,
+ maxRetryDelayMs: 1,
+ onPayload: (payload) => applyFastSpeed(payload, fastMode, model.api),
+ },
+ )
+ .result();
+ signal.throwIfAborted();
+ if (response.stopReason !== "stop" || !response.content.some((part) => part.type === "text" && part.text.trim()))
+ throw new Error(response.errorMessage || "Model verification did not produce a completed text response");
+}
+
const FAST_MODE_BETA = "fast-mode-2026-02-01";
export const FAST_COST_MULTIPLIER = 2;
diff --git a/src/model/model-overlay-store.ts b/src/model/model-overlay-store.ts
index e560d8dfd..366f3efb9 100644
--- a/src/model/model-overlay-store.ts
+++ b/src/model/model-overlay-store.ts
@@ -1,3 +1,5 @@
+import { randomUUID } from "node:crypto";
+import { ModelVerificationError, verificationFailure, type ModelVerifier } from "./model-verification.ts";
import type { DurableMap } from "../persistence/durable-map.ts";
import { parseModelOverlay, type ModelOverlay } from "./model-overlay.ts";
import { modelUnavailableReason, setModelOverlays, validateModelOverlay } from "./pi-models.ts";
@@ -7,46 +9,129 @@ export interface StoredModelOverlay {
disabled: boolean;
updatedAt: number;
updatedBy: string;
+ verification?: { fingerprint: string; verifiedAt: number; revision: string };
}
export function createModelOverlayStore(
backing: DurableMap<StoredModelOverlay>,
write: <T>(fn: () => Promise<T>) => Promise<T> = (fn) => fn(),
+ verifier?: ModelVerifier,
) {
+ async function verificationReason(row: StoredModelOverlay, id: string): Promise<string | undefined> {
+ if (!row?.verification || !verifier)
+ return "Model has not been verified; an administrator must verify and enable it.";
+ try {
+ const context = await verifier(parseModelOverlay({ ...row.spec, id }));
+ if (context.fingerprint === row.verification.fingerprint) return undefined;
+ return "Model settings or serving credentials changed; verify and enable this model again.";
+ } catch {
+ return "Model verification is no longer valid; check serving credentials and verify again.";
+ }
+ }
return {
async statuses() {
- return (await backing.entries()).map(([id, row]) => {
- let spec: Partial<ModelOverlay> & { id: string };
- try {
- spec = parseModelOverlay({ ...row?.spec, id });
- } catch {
- spec = { id, name: id };
- }
- return {
- spec,
- disabled: row?.disabled === true,
- updatedAt: row?.updatedAt,
- updatedBy: row?.updatedBy,
- unavailableReason: modelUnavailableReason(id),
- };
- });
+ return Promise.all(
+ (await backing.entries()).map(async ([id, row]) => {
+ let spec: Partial<ModelOverlay> & { id: string };
+ try {
+ spec = parseModelOverlay({ ...row?.spec, id });
+ } catch {
+ spec = { id, name: id };
+ }
+ return {
+ spec,
+ disabled: row?.disabled === true,
+ updatedAt: row?.updatedAt,
+ updatedBy: row?.updatedBy,
+ unavailableReason:
+ modelUnavailableReason(id) ?? (row?.disabled ? undefined : await verificationReason(row, id)),
+ verifiedAt: row?.verification?.verifiedAt,
+ verificationScope: "organization" as const,
+ };
+ }),
+ );
},
async refresh() {
const rows = await backing.entries();
+ const failures = new Map<string, string>();
+ for (const [id, row] of rows) {
+ if (!row || row.disabled) continue;
+ const reason = await verificationReason(row, id);
+ if (reason) failures.set(id, reason);
+ }
setModelOverlays(
rows.filter(([, row]) => row?.disabled !== true).map(([id, row]) => ({ ...row?.spec, id })),
rows.filter(([, row]) => row?.disabled === true).map(([id]) => id),
+ failures,
);
},
async upsert(value: unknown, updatedBy: string) {
- return write(async () => {
- const spec = validateModelOverlay(value);
- if (!updatedBy.trim()) throw new Error("updatedBy is required");
+ const spec = validateModelOverlay(value);
+ if (!updatedBy.trim()) throw new Error("updatedBy is required");
+ if (!verifier)
+ throw new ModelVerificationError(
+ "verification_unavailable",
+ "Provider verification is unavailable; the model was not enabled.",
+ );
+ const previous = await write(async () => {
+ validateModelOverlay(spec);
const existing = await backing.get(spec.id);
if (existing?.spec?.provider && existing.spec.provider !== spec.provider)
throw new Error("provider cannot change for an existing model id");
- await backing.put(spec.id, { spec, disabled: false, updatedAt: Date.now(), updatedBy });
+ return JSON.stringify(existing);
});
+ try {
+ const context = await verifier(spec);
+ const signal = AbortSignal.timeout(15_000);
+ await new Promise<void>((resolve, reject) => {
+ const onAbort = () =>
+ reject(new ModelVerificationError("timeout", "Verification timed out; the model was not enabled."));
+ signal.addEventListener("abort", onAbort, { once: true });
+ context
+ .probe(signal)
+ .then(resolve, reject)
+ .finally(() => signal.removeEventListener("abort", onAbort));
+ });
+ return await write(async () => {
+ try {
+ validateModelOverlay(spec);
+ } catch {
+ throw new ModelVerificationError(
+ "configuration_conflict",
+ "The model ID or template changed during verification. Reload and try again.",
+ );
+ }
+ if (JSON.stringify(await backing.get(spec.id)) !== previous)
+ throw new ModelVerificationError(
+ "changed_during_verification",
+ "The model changed during verification. Reload and try again.",
+ );
+ if ((await verifier(spec)).fingerprint !== context.fingerprint)
+ throw new ModelVerificationError(
+ "changed_during_verification",
+ "Serving credentials changed during verification. Verify again.",
+ );
+ const verification = { fingerprint: context.fingerprint, verifiedAt: Date.now(), revision: randomUUID() };
+ await backing.put(spec.id, { spec, disabled: false, updatedAt: Date.now(), updatedBy, verification });
+ return { verifiedAt: verification.verifiedAt, verificationScope: "organization" as const };
+ });
+ } catch (error) {
+ const failure = verificationFailure(error);
+ await write(async () => {
+ const current = await backing.get(spec.id);
+ let sameSpec: boolean;
+ try {
+ sameSpec = JSON.stringify(parseModelOverlay(current?.spec)) === JSON.stringify(spec);
+ } catch {
+ sameSpec = false;
+ }
+ if (current && !current.disabled && JSON.stringify(current) === previous && sameSpec) {
+ const { verification: _verification, ...saved } = current;
+ await backing.put(spec.id, saved);
+ }
+ });
+ throw failure;
+ }
},
async delete(id: string, updatedBy: string) {
return write(async () => {
diff --git a/src/model/model-verification.ts b/src/model/model-verification.ts
new file mode 100644
index 000000000..7078648e5
--- /dev/null
+++ b/src/model/model-verification.ts
@@ -0,0 +1,102 @@
+import { createHmac } from "node:crypto";
+import { probeModel } from "../harness/pi-harness.ts";
+import { modelFromOverlay } from "./pi-models.ts";
+import type { ModelOverlay } from "./model-overlay.ts";
+import type { ModelCredentialStore } from "./model-credential-store.ts";
+import type { ModelGatewayTransportConfig } from "./provider-endpoints.ts";
+
+export class ModelVerificationError extends Error {
+ readonly code: string;
+ constructor(code: string, message: string) {
+ super(message);
+ this.code = code;
+ }
+}
+
+interface ModelVerificationContext {
+ fingerprint: string;
+ probe(signal: AbortSignal): Promise<void>;
+}
+export type ModelVerifier = (spec: ModelOverlay) => Promise<ModelVerificationContext>;
+
+export function verificationFailure(error: unknown): ModelVerificationError {
+ if (error instanceof ModelVerificationError) return error;
+ const text = error instanceof Error ? error.message : "";
+ if (/timeout|aborted|aborterror/i.test(text))
+ return new ModelVerificationError("timeout", "Verification timed out. Try again; the model was not enabled.");
+ if (/401|403|unauthori[sz]ed|forbidden|permission|authentication|(?:invalid|incorrect).api.key/i.test(text))
+ return new ModelVerificationError(
+ "access_denied",
+ "The serving credential cannot access this model. Check its provider permissions.",
+ );
+ if (/429|quota|rate.limit|billing|credit|resource.exhausted/i.test(text))
+ return new ModelVerificationError(
+ "quota_or_rate_limit",
+ "The provider reports a quota, billing, or rate limit. Resolve it and verify again.",
+ );
+ if (/404|not.found|does not exist|model.*unavailable/i.test(text))
+ return new ModelVerificationError(
+ "model_unavailable",
+ "The provider could not serve this model ID. Check the ID and credential access.",
+ );
+ if (/\b5\d\d\b/.test(text))
+ return new ModelVerificationError(
+ "provider_failure",
+ "The provider is temporarily unable to complete verification. Try again later.",
+ );
+ if (/400|422|unsupported|invalid.*(parameter|request)|not.supported/i.test(text))
+ return new ModelVerificationError(
+ "unsupported_configuration",
+ "The provider rejected this configuration. Check the template and fast-mode capability.",
+ );
+ return new ModelVerificationError(
+ "provider_failure",
+ "The provider did not complete the verification request. Try again or check provider status.",
+ );
+}
+
+export function createModelVerifier(input: {
+ credentials: ModelCredentialStore;
+ keyMaterial: string | Buffer;
+ modelGateway?: ModelGatewayTransportConfig;
+ probe?: typeof probeModel;
+}): ModelVerifier {
+ return async (spec) => {
+ const model = modelFromOverlay(spec);
+ if (!model)
+ throw new ModelVerificationError(
+ "invalid_template",
+ "The model template is unavailable. Choose another template.",
+ );
+ const gateway = input.modelGateway?.models[spec.id] ? input.modelGateway : undefined;
+ const key = gateway?.apiKey ?? (await input.credentials.resolve(spec.provider));
+ if (!key)
+ throw new ModelVerificationError(
+ "missing_credential",
+ "Configure an organization provider credential before verifying this model.",
+ );
+ const status = (await input.credentials.statuses()).find((s) => s.provider === spec.provider);
+ const fingerprint = createHmac("sha256", input.keyMaterial)
+ .update(
+ JSON.stringify({
+ version: 1,
+ spec,
+ model,
+ key,
+ credentialRevision: gateway ? undefined : status,
+ gateway: gateway
+ ? { url: gateway.url, apiKeyHeader: gateway.apiKeyHeader, target: gateway.models[spec.id] }
+ : undefined,
+ }),
+ )
+ .digest("hex");
+ return {
+ fingerprint,
+ async probe(signal) {
+ const run = input.probe ?? probeModel;
+ await run(model, { [spec.provider]: key }, signal, false, gateway);
+ if (spec.fastMode) await run(model, { [spec.provider]: key }, signal, true, gateway);
+ },
+ };
+ };
+}
diff --git a/src/model/pi-models.ts b/src/model/pi-models.ts
index c1a7b11da..470317f0d 100644
--- a/src/model/pi-models.ts
+++ b/src/model/pi-models.ts
@@ -227,7 +227,11 @@ export function validateModelOverlay(value: unknown): ModelOverlay {
return spec;
}
-export function setModelOverlays(specs: readonly unknown[], deleted: string[] = []): void {
+export function setModelOverlays(
+ specs: readonly unknown[],
+ deleted: string[] = [],
+ verificationFailures: ReadonlyMap<string, string> = new Map(),
+): void {
const next = new Map<string, ModelOverlay>();
const unavailable = new Map(deleted.map((id) => [id, "Model has been deleted; select another model"]));
for (const value of specs) {
@@ -250,7 +254,8 @@ export function setModelOverlays(specs: readonly unknown[], deleted: string[] =
);
} else if (isCustomModelId(id) || OPENROUTER_CATALOG_MODELS.has(id)) {
unavailable.set(id, "Model id conflicts with another provider; ask an administrator to repair this model");
- } else next.set(id, spec);
+ } else if (verificationFailures.has(id)) unavailable.set(id, verificationFailures.get(id)!);
+ else next.set(id, spec);
} catch {
unavailable.set(id, "Stored model definition is invalid; ask an administrator to repair this model");
}
@@ -388,18 +393,23 @@ function resolveBaseModel(id: string): PiModel | undefined {
if (builtin) return builtin;
const overlay = overlays.get(id);
if (overlay) {
- const template = resolveBuiltinModel(overlay.template);
- return template
- ? cloneModel(template, id, overlay.name, {
- contextWindow: overlay.contextWindow,
- maxTokens: overlay.maxTokens,
- cost: structuredClone(overlay.cost),
- })
- : undefined;
+ return modelFromOverlay(overlay, false);
}
return (resolveCustomModel(id) as unknown as PiModel | undefined) ?? OPENROUTER_CATALOG_MODELS.get(id);
}
+export function modelFromOverlay(spec: ModelOverlay, useOrgEndpoints = true): PiModel | undefined {
+ const template = resolveBuiltinModel(spec.template);
+ if (!template || template.provider !== spec.provider) return undefined;
+ const model = cloneModel(template, spec.id, spec.name, {
+ contextWindow: spec.contextWindow,
+ maxTokens: spec.maxTokens,
+ cost: structuredClone(spec.cost),
+ });
+ const override = useOrgEndpoints ? providerBaseUrl(spec.provider) : undefined;
+ return override ? { ...model, baseUrl: override } : model;
+}
+
export function resolveModel(id: string, useOrgEndpoints = true): PiModel | undefined {
const model = resolveBaseModel(id);
if (!model || !useOrgEndpoints) return model;
diff --git a/src/wiring.ts b/src/wiring.ts
index 9e623bbfc..255994c24 100644
--- a/src/wiring.ts
+++ b/src/wiring.ts
@@ -1,3 +1,5 @@
+import { createModelVerifier } from "./model/model-verification.ts";
+import type { probeModel } from "./harness/pi-harness.ts";
import { createAwsRoleBroker, type AwsRoleBroker } from "./auth/aws-role-broker.ts";
import type { SessionShare, SessionShareStore } from "./sessions/session-share.ts";
import { createModelOverlayStore, type ModelOverlayStore } from "./model/model-overlay-store.ts";
@@ -469,6 +471,7 @@ export function buildApp(
securityScreener?: SecurityScreener;
credentialBrokers?: Record<string, AwsRoleBroker>;
modelCredentialFetch?: typeof fetch;
+ modelVerificationProbe?: typeof probeModel;
} = {},
): BuiltApp {
if (config.databaseUrl && !config.connectorSecretKey) {
@@ -992,7 +995,16 @@ export function buildApp(
backing: artifactMap("custom_model_providers"),
keyMaterial: config.connectorSecretKey ?? randomBytes(32),
});
- const modelRegistry = createModelOverlayStore(artifactMap("model_registry"), writeModelRegistry);
+ const modelRegistry = createModelOverlayStore(
+ artifactMap("model_registry"),
+ writeModelRegistry,
+ createModelVerifier({
+ credentials: modelCredentials,
+ keyMaterial: config.connectorSecretKey ?? randomBytes(32),
+ modelGateway: config.modelGateway,
+ probe: overrides.modelVerificationProbe,
+ }),
+ );
const refreshCustomProviders = async () => {
setCustomProviders(await customProviders.enabled());
};
diff --git a/test/model-overlay-postgres.test.ts b/test/model-overlay-postgres.test.ts
index d817abe67..7e89f18f7 100644
--- a/test/model-overlay-postgres.test.ts
+++ b/test/model-overlay-postgres.test.ts
@@ -1,3 +1,4 @@
+import { verificationUpstream } from "./support/model-verification-upstream.ts";
import assert from "node:assert/strict";
import { fork, type ChildProcess } from "node:child_process";
import { once } from "node:events";
@@ -11,10 +12,10 @@ const databaseUrl = process.env.MODEL_OVERLAY_TEST_DATABASE_URL;
const headers = { "content-type": "application/json", "x-admin-actor": "admin-alice@default-org" };
const runtimePath = "/v1/runtime-config?principalId=admin-alice@default-org&scopeId=personal:admin-alice@default-org";
-async function start(databaseUrl: string) {
+async function start(databaseUrl: string, upstream: string) {
const child = fork(new URL("./support/model-overlay-server.ts", import.meta.url), {
stdio: ["ignore", "ignore", "ignore", "ipc"],
- env: { ...process.env, MODEL_OVERLAY_TEST_DATABASE_URL: databaseUrl },
+ env: { ...process.env, MODEL_OVERLAY_TEST_DATABASE_URL: databaseUrl, MODEL_OVERLAY_TEST_UPSTREAM: upstream },
});
const timeout = setTimeout(() => child.kill(), 30_000);
const message = await Promise.race([
@@ -42,6 +43,8 @@ test(
["127.0.0.1", "localhost"].includes(new URL(databaseUrl).hostname),
"use a dedicated local test database",
);
+ const upstream = await verificationUpstream();
+ t.after(() => upstream.close());
const pool = new pg.Pool({ connectionString: databaseUrl });
const schema = `model_overlay_${randomUUID().replaceAll("-", "")}`;
t.after(async () => {
@@ -58,9 +61,9 @@ test(
const factory = createPostgresMapFactory(isolatedDatabaseUrl);
const processes: ChildProcess[] = [];
try {
- const first = await start(isolatedDatabaseUrl);
+ const first = await start(isolatedDatabaseUrl, upstream.url);
processes.push(first.child);
- const second = await start(isolatedDatabaseUrl);
+ const second = await start(isolatedDatabaseUrl, upstream.url);
processes.push(second.child);
const read = async (base: string) => {
const response = await fetch(base + runtimePath, { headers });
@@ -73,6 +76,7 @@ test(
};
assert.equal((await read(second.base)).modelCatalog["overlay-pg-model"], undefined);
const spec = {
+ verify: true,
name: "Future PG model",
provider: "openai",
template: "gpt-5.5",
@@ -107,10 +111,13 @@ test(
}),
}),
]);
- assert.deepEqual(raced.map((response) => response.status).sort(), [200, 400]);
+ assert.deepEqual(
+ raced.map((response) => response.status),
+ raced[0]!.status === 200 ? [200, 400] : [409, 200],
+ );
await read(first.base);
await read(second.base);
- const cold = await start(isolatedDatabaseUrl);
+ const cold = await start(isolatedDatabaseUrl, upstream.url);
processes.push(cold.child);
const coldConfig = await read(cold.base);
assert.equal(coldConfig.effective.modelId, "overlay-pg-model");
diff --git a/test/model-overlay.test.ts b/test/model-overlay.test.ts
index e9bbbddce..f649c66e3 100644
--- a/test/model-overlay.test.ts
+++ b/test/model-overlay.test.ts
@@ -13,7 +13,10 @@ const MODEL_ID = "overlay-future-model";
test("unknown builtin-provider model fails resolution and a live runtime selection request", async () => {
assert.throws(() => getRequiredModel(MODEL_ID), /Unsupported model/);
const config = testConfig({ harness: "pi" });
- const built = buildApp(config, { modelCredentialFetch: async () => Response.json({ data: [] }) });
+ const built = buildApp(config, {
+ modelCredentialFetch: async () => Response.json({ data: [] }),
+ modelVerificationProbe: async () => {},
+ });
const server = createInsecureTestServer(built.app, serverDeps(config, built));
server.listen(0, "127.0.0.1");
await new Promise<void>((resolve) => server.once("listening", resolve));
@@ -40,7 +43,7 @@ test("unknown builtin-provider model fails resolution and a live runtime selecti
import { afterEach } from "node:test";
import { createMemoryMap } from "../src/persistence/durable-map.ts";
-import { createModelOverlayStore, type StoredModelOverlay } from "../src/model/model-overlay-store.ts";
+import { createModelOverlayStore as createStore, type StoredModelOverlay } from "../src/model/model-overlay-store.ts";
import {
setModelOverlays,
validateModelOverlay,
@@ -57,6 +60,9 @@ import { selectableModelCatalog } from "../src/model/model-catalog.ts";
import { resolveIndividualAuthRouting } from "../src/core/individual-auth-routing.ts";
import { validateWebTurnModelOptions } from "../src/core/turn-options.ts";
+const createModelOverlayStore: typeof createStore = (backing, write) =>
+ createStore(backing, write, async () => ({ fingerprint: "fixture", probe: async () => {} }));
+
const spec = {
id: MODEL_ID,
name: "Future API model",
@@ -203,12 +209,25 @@ test("collisions are rejected in both registration orders and OpenRouter failure
test("live admin lifecycle is authorized, audited, immediately selectable and removed on delete", async () => {
const config = testConfig({ harness: "pi", openaiApiKey: "local-test-key" });
- const built = buildApp(config, { modelCredentialFetch: async () => Response.json({ data: [] }) });
+ const built = buildApp(config, {
+ modelCredentialFetch: async () => Response.json({ data: [] }),
+ modelVerificationProbe: async () => {},
+ });
const server = createInsecureTestServer(built.app, serverDeps(config, built));
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
const api = (path: string, method = "GET", body?: unknown, headers = ADMIN) =>
- fetch(`${base}${path}`, { method, headers, ...(body === undefined ? {} : { body: JSON.stringify(body) }) });
+ fetch(`${base}${path}`, {
+ method,
+ headers,
+ ...(body === undefined
+ ? {}
+ : {
+ body: JSON.stringify(
+ method === "PUT" && path.includes("model-registry") ? { ...(body as object), verify: true } : body,
+ ),
+ }),
+ });
const path = `/v1/admin/model-registry/${MODEL_ID}`;
const runtime = {
principalId: "admin-alice@default-org",
@@ -429,7 +448,10 @@ test("hydration isolates promoted builtins, incompatible providers, missing temp
test("model refresh is limited to dependent routes and admin repair survives stale persisted definitions", async () => {
const config = testConfig({ harness: "pi" });
- const built = buildApp(config, { modelCredentialFetch: async () => Response.json({ data: [] }) });
+ const built = buildApp(config, {
+ modelCredentialFetch: async () => Response.json({ data: [] }),
+ modelVerificationProbe: async () => {},
+ });
const backing = createMemoryMap<StoredModelOverlay>();
await backing.put("stale", {
spec: { ...spec, id: "stale", template: "missing" },
diff --git a/test/model-verification.test.ts b/test/model-verification.test.ts
new file mode 100644
index 000000000..3d3ec9628
--- /dev/null
+++ b/test/model-verification.test.ts
@@ -0,0 +1,433 @@
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { createModelOverlayStore } from "../src/model/model-overlay-store.ts";
+import { createMemoryMap } from "../src/persistence/durable-map.ts";
+
+const spec = {
+ id: "verify-future",
+ name: "Verify future",
+ provider: "openai",
+ template: "gpt-5.5",
+ contextWindow: 200_000,
+ maxTokens: 16_000,
+ cost: { input: 1, output: 2, cacheRead: 0, cacheWrite: 0 },
+};
+
+test("no model can be enabled without provider verification", async () => {
+ const store = createModelOverlayStore(createMemoryMap());
+ await assert.rejects(store.upsert(spec, "admin"), /verification/i);
+ assert.deepEqual(await store.statuses(), []);
+});
+
+import { afterEach } from "node:test";
+import {
+ ModelVerificationError,
+ verificationFailure,
+ createModelVerifier,
+ type ModelVerifier,
+} from "../src/model/model-verification.ts";
+import { setModelOverlays, resolveModel, modelUnavailableReason } from "../src/model/pi-models.ts";
+import type { StoredModelOverlay } from "../src/model/model-overlay-store.ts";
+
+const verified: ModelVerifier = async () => ({ fingerprint: "test-context", probe: async () => {} });
+afterEach(() => setModelOverlays([]));
+
+test("legacy unverified rows remain unavailable and statuses never expose proof fingerprints", async () => {
+ const backing = createMemoryMap<StoredModelOverlay>();
+ const store = createModelOverlayStore(backing, undefined, verified);
+ await backing.put(spec.id, { spec, disabled: false, updatedAt: 1, updatedBy: "admin" } as StoredModelOverlay);
+ await store.refresh();
+ assert.equal(resolveModel(spec.id), undefined);
+ assert.match(modelUnavailableReason(spec.id)!, /not been verified/);
+ await store.upsert(spec, "admin");
+ await store.refresh();
+ assert.ok(resolveModel(spec.id));
+ const status = await store.statuses();
+ assert.equal(status[0]?.verificationScope, "organization");
+ assert.ok(status[0]?.verifiedAt);
+ assert.doesNotMatch(JSON.stringify(status), /fingerprint|test-context/);
+});
+
+test("failed probes never publish new models or replace previously verified definitions", async () => {
+ let fail = false;
+ const store = createModelOverlayStore(createMemoryMap(), undefined, async () => ({
+ fingerprint: "context",
+ probe: async () => {
+ if (fail) throw Error("403 secret-credential-value");
+ },
+ }));
+ await store.upsert(spec, "admin");
+ fail = true;
+ await assert.rejects(store.upsert({ ...spec, name: "Unverified edit" }, "admin"), /cannot access/);
+ await assert.rejects(store.upsert({ ...spec, id: "unverified-new" }, "admin"), /cannot access/);
+ const statuses = await store.statuses();
+ assert.equal(statuses.length, 1);
+ assert.equal(statuses[0]?.spec.name, spec.name);
+ assert.doesNotMatch(JSON.stringify(statuses), /secret-credential-value/);
+});
+
+test("credential rotation invalidates availability and a successful recheck restores it", async () => {
+ let fingerprint = "key-a";
+ const store = createModelOverlayStore(createMemoryMap(), undefined, async () => ({
+ fingerprint,
+ probe: async () => {},
+ }));
+ await store.upsert(spec, "admin");
+ await store.refresh();
+ assert.ok(resolveModel(spec.id));
+ fingerprint = "key-b";
+ await store.refresh();
+ assert.equal(resolveModel(spec.id), undefined);
+ assert.match(modelUnavailableReason(spec.id)!, /credentials changed/);
+ await store.upsert(spec, "admin");
+ await store.refresh();
+ assert.ok(resolveModel(spec.id));
+});
+
+test("a deletion while verification runs cannot be undone by its late success", async () => {
+ let release: () => void = () => {};
+ let entered: () => void = () => {};
+ const started = new Promise<void>((r) => {
+ entered = r;
+ });
+ let hold = false;
+ const store = createModelOverlayStore(createMemoryMap(), undefined, async () => ({
+ fingerprint: "context",
+ probe: async () => {
+ if (hold) {
+ entered();
+ await new Promise<void>((r) => {
+ release = r;
+ });
+ }
+ },
+ }));
+ await store.upsert(spec, "admin");
+ hold = true;
+ const update = store.upsert({ ...spec, name: "Late edit" }, "admin");
+ await started;
+ await store.delete(spec.id, "admin");
+ release();
+ await assert.rejects(update, /changed during verification/);
+ assert.equal((await store.statuses())[0]?.disabled, true);
+});
+
+test("a changed serving context during a probe is not certified", async () => {
+ let fingerprint = "old";
+ const store = createModelOverlayStore(createMemoryMap(), undefined, async () => ({
+ fingerprint,
+ probe: async () => {
+ fingerprint = "new";
+ },
+ }));
+ await assert.rejects(store.upsert(spec, "admin"), /credentials changed/);
+ assert.deepEqual(await store.statuses(), []);
+});
+
+test("provider errors are actionable without exposing raw provider text", () => {
+ for (const [raw, code] of [
+ ["401 secret", "access_denied"],
+ ["403 secret", "access_denied"],
+ ["404 secret", "model_unavailable"],
+ ["429 secret", "quota_or_rate_limit"],
+ ["400 unsupported secret", "unsupported_configuration"],
+ ["timeout secret", "timeout"],
+ ["500 secret", "provider_failure"],
+ ]) {
+ const error = verificationFailure(new Error(raw));
+ assert.equal(error.code, code);
+ assert.doesNotMatch(error.message, /secret/);
+ assert.ok(error instanceof ModelVerificationError);
+ }
+});
+
+import "./support/auto-fake-sprites.ts";
+import { verificationUpstream } from "./support/model-verification-upstream.ts";
+import { buildApp, serverDeps } from "../src/wiring.ts";
+import { testConfig } from "./support/test-config.ts";
+import { createInsecureTestServer } from "../src/api/server.ts";
+import type { AddressInfo } from "node:net";
+import { probeModel } from "../src/harness/pi-harness.ts";
+import { modelFromOverlay } from "../src/model/pi-models.ts";
+import { parseModelOverlay } from "../src/model/model-overlay.ts";
+import { setProviderBaseUrls } from "../src/model/provider-endpoints.ts";
+
+test("live API verifies exact serving credentials, blocks denied models, rechecks edits, and invalidates rotated keys", async () => {
+ const upstream = await verificationUpstream();
+ const config = testConfig({
+ openaiApiKey: "verification-test-key",
+ harness: "pi",
+ providerBaseUrls: { openai: upstream.url + "/v1", anthropic: upstream.url },
+ });
+ const built = buildApp(config, { modelCredentialFetch: async () => Response.json({ data: [] }) });
+ const server = createInsecureTestServer(built.app, serverDeps(config, built));
+ await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));
+ const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
+ const headers = { "content-type": "application/json", "x-admin-actor": "admin-alice@default-org" };
+ const path = base + "/v1/admin/model-registry/" + spec.id;
+ const put = (body: object, actor = headers["x-admin-actor"]) =>
+ fetch(path, { method: "PUT", headers: { ...headers, "x-admin-actor": actor }, body: JSON.stringify(body) });
+ const runtime = async () =>
+ (
+ await fetch(
+ base + "/v1/runtime-config?principalId=admin-alice@default-org&scopeId=personal:admin-alice@default-org",
+ { headers },
+ )
+ ).json() as Promise<{
+ effective: { modelId: string };
+ modelCatalog: Record<string, unknown>;
+ unavailableReason: string;
+ }>;
+ try {
+ assert.equal((await put({ ...spec, verify: true }, "bob@default-org")).status, 403);
+ assert.equal((await put(spec)).status, 400);
+ assert.equal(upstream.requests.length, 0);
+ upstream.behavior.status = 403;
+ const denied = await put({ ...spec, verify: true });
+ assert.equal(denied.status, 422);
+ const deniedBody = (await denied.json()) as { error: string };
+ assert.equal(deniedBody.error, "access_denied");
+ assert.doesNotMatch(JSON.stringify(deniedBody), /private-provider-detail|verification-test-key/);
+ assert.equal((await runtime()).modelCatalog[spec.id], undefined);
+ for (const [status, code] of [
+ [401, "access_denied"],
+ [404, "model_unavailable"],
+ [400, "unsupported_configuration"],
+ [429, "quota_or_rate_limit"],
+ [503, "provider_failure"],
+ ] as const) {
+ upstream.behavior.status = status;
+ const rejected = await put({ ...spec, verify: true });
+ assert.equal(rejected.status, 422);
+ const result = (await rejected.json()) as { error: string };
+ assert.equal(result.error, code);
+ assert.doesNotMatch(JSON.stringify(result), /private-provider-detail/);
+ assert.equal((await runtime()).modelCatalog[spec.id], undefined);
+ }
+ upstream.behavior.status = 200;
+ upstream.behavior.empty = true;
+ assert.equal((await put({ ...spec, verify: true })).status, 422);
+ upstream.behavior.empty = false;
+ const accepted = await put({ ...spec, verify: true });
+ assert.equal(accepted.status, 200);
+ const proof = (await accepted.json()) as { verifiedAt: number; verificationScope: string };
+ assert.ok(proof.verifiedAt);
+ assert.equal(proof.verificationScope, "organization");
+ const request = upstream.requests.at(-1)!;
+ assert.equal(request.path, "/v1/responses");
+ assert.equal(request.body.model, spec.id);
+ assert.equal(request.body.max_output_tokens, 128);
+ assert.ok(Array.isArray(request.body.tools));
+ assert.equal(request.authorization, "Bearer verification-test-key");
+ assert.ok((await runtime()).modelCatalog[spec.id]);
+ const beforeEdit = upstream.requests.length;
+ assert.equal((await put({ ...spec, name: "Verified edit", verify: true })).status, 200);
+ assert.equal(upstream.requests.length, beforeEdit + 1);
+ await built.config.setRuntimeSelectionLatest("personal:admin-alice@default-org", {
+ harnessId: "pi",
+ modelId: spec.id,
+ });
+ await built.modelCredentials.set("openai", "rotated-test-key", "admin-alice");
+ const stale = await runtime();
+ assert.equal(stale.effective.modelId, spec.id);
+ assert.equal(stale.modelCatalog[spec.id], undefined);
+ assert.match(stale.unavailableReason, /credentials changed/);
+ assert.equal((await put({ ...spec, verify: true })).status, 200);
+ assert.equal(upstream.requests.at(-1)?.authorization, "Bearer rotated-test-key");
+ await built.modelCredentials.delete("openai", "admin-alice");
+ const calls = upstream.requests.length;
+ const missing = await put({ ...spec, verify: true });
+ assert.equal(missing.status, 422);
+ assert.equal(((await missing.json()) as { error: string }).error, "missing_credential");
+ assert.equal(upstream.requests.length, calls);
+ } finally {
+ server.closeAllConnections();
+ await new Promise<void>((r) => server.close(() => r()));
+ await upstream.close();
+ setProviderBaseUrls({});
+ }
+});
+
+test("Anthropic verification exercises ordinary and fast requests, and gateway uses its serving target and credential", async () => {
+ const upstream = await verificationUpstream();
+ const config = testConfig({ anthropicApiKey: "anthropic-test-key", providerBaseUrls: { anthropic: upstream.url } });
+ const built = buildApp(config);
+ try {
+ upstream.behavior.rejectFast = true;
+ await assert.rejects(
+ built.modelRegistry.upsert(
+ { ...spec, provider: "anthropic", template: "claude-opus-4-8", fastMode: true },
+ "admin",
+ ),
+ /rejected this configuration/,
+ );
+ assert.equal((await built.modelRegistry.statuses()).length, 0);
+ upstream.behavior.rejectFast = false;
+ upstream.requests.length = 0;
+ await built.modelRegistry.upsert(
+ { ...spec, provider: "anthropic", template: "claude-opus-4-8", fastMode: true },
+ "admin",
+ );
+ assert.equal(upstream.requests.length, 2);
+ assert.equal(upstream.requests[0]?.body.model, spec.id);
+ assert.equal(upstream.requests[0]?.body.max_tokens, 128);
+ assert.equal(upstream.requests[0]?.key, "anthropic-test-key");
+ assert.equal(upstream.requests[0]?.body.speed, undefined);
+ assert.equal(upstream.requests[1]?.body.speed, "fast");
+ assert.match(upstream.requests[1]!.beta!, /fast-mode/);
+ const verifier = createModelVerifier({
+ credentials: built.modelCredentials,
+ keyMaterial: "test-hmac-key",
+ modelGateway: {
+ url: upstream.url + "/v1",
+ apiKey: "gateway-test-key",
+ apiKeyHeader: "x-gateway-key",
+ models: { [spec.id]: "gateway-model-target" },
+ },
+ });
+ const context = await verifier(parseModelOverlay(spec));
+ await context.probe(AbortSignal.timeout(5000));
+ assert.equal(upstream.requests.at(-1)?.body.model, "gateway-model-target");
+ assert.equal(upstream.requests.at(-1)?.authorization, "Bearer gateway-test-key");
+ assert.equal(upstream.requests.at(-1)?.gatewayKey, "gateway-test-key");
+ } finally {
+ await upstream.close();
+ setProviderBaseUrls({});
+ }
+});
+
+test("the streaming probe aborts a nonresponsive provider and never accepts an incomplete response", async () => {
+ const upstream = await verificationUpstream();
+ upstream.behavior.hang = true;
+ setProviderBaseUrls({ openai: upstream.url + "/v1" });
+ try {
+ const model = modelFromOverlay(parseModelOverlay(spec))!;
+ await assert.rejects(probeModel(model, { openai: "test-key" }, AbortSignal.timeout(200)));
+ } finally {
+ await upstream.close();
+ setProviderBaseUrls({});
+ }
+});
+
+test("a failed recheck of the saved definition removes its prior certification", async () => {
+ let fail = false;
+ const store = createModelOverlayStore(createMemoryMap(), undefined, async () => ({
+ fingerprint: "context",
+ probe: async () => {
+ if (fail) throw Error("403");
+ },
+ }));
+ await store.upsert(spec, "admin");
+ await store.refresh();
+ assert.ok(resolveModel(spec.id));
+ fail = true;
+ await assert.rejects(store.upsert(spec, "admin"), /cannot access/);
+ await store.refresh();
+ assert.equal(resolveModel(spec.id), undefined);
+ assert.equal((await store.statuses())[0]?.spec.name, spec.name);
+});
+
+test("endpoint, template metadata, credential revision, and model edits change verification fingerprints", async () => {
+ const built = buildApp(testConfig({ openaiApiKey: "test-key" }));
+ const verifier = createModelVerifier({ credentials: built.modelCredentials, keyMaterial: "stable-test-hmac" });
+ try {
+ const original = (await verifier(parseModelOverlay(spec))).fingerprint;
+ assert.equal((await verifier(parseModelOverlay(spec))).fingerprint, original);
+ assert.notEqual((await verifier(parseModelOverlay({ ...spec, template: "gpt-5.6-sol" }))).fingerprint, original);
+ assert.notEqual((await verifier(parseModelOverlay({ ...spec, maxTokens: 8000 }))).fingerprint, original);
+ setProviderBaseUrls({ openai: "http://127.0.0.1:19999/v1" });
+ assert.notEqual((await verifier(parseModelOverlay(spec))).fingerprint, original);
+ setProviderBaseUrls({});
+ await built.modelCredentials.set("openai", "test-key", "admin");
+ assert.notEqual((await verifier(parseModelOverlay(spec))).fingerprint, original);
+ } finally {
+ setProviderBaseUrls({});
+ }
+});
+
+test("corrupt persisted rows and copied proofs cannot enable a different model ID", async () => {
+ const backing = createMemoryMap<StoredModelOverlay>();
+ const verifier: ModelVerifier = async (value) => ({ fingerprint: JSON.stringify(value), probe: async () => {} });
+ const store = createModelOverlayStore(backing, undefined, verifier);
+ await store.upsert(spec, "admin");
+ const saved = (await backing.get(spec.id))!;
+ await backing.put("copied-model", saved);
+ await backing.put("null-model", null as unknown as StoredModelOverlay);
+ await store.refresh();
+ assert.ok(resolveModel(spec.id));
+ assert.equal(resolveModel("copied-model"), undefined);
+ assert.equal(resolveModel("null-model"), undefined);
+ assert.equal((await store.statuses()).length, 3);
+});
+
+test("concurrent successful probes cannot silently overwrite a newer saved definition", async () => {
+ let release: () => void = () => {};
+ let entered: () => void = () => {};
+ const started = new Promise<void>((r) => {
+ entered = r;
+ });
+ let holding = false;
+ const store = createModelOverlayStore(createMemoryMap(), undefined, async (value) => ({
+ fingerprint: JSON.stringify(value),
+ probe: async () => {
+ if (holding && value.name === "Slow edit") {
+ entered();
+ await new Promise<void>((r) => {
+ release = r;
+ });
+ }
+ },
+ }));
+ await store.upsert(spec, "admin");
+ holding = true;
+ const slow = store.upsert({ ...spec, name: "Slow edit" }, "admin");
+ await started;
+ await store.upsert({ ...spec, name: "Newer edit" }, "admin");
+ release();
+ await assert.rejects(slow, /changed during verification/);
+ assert.equal((await store.statuses())[0]?.spec.name, "Newer edit");
+});
+
+test("a model stays unavailable until its pending probe completes", async () => {
+ let release: () => void = () => {};
+ let entered: () => void = () => {};
+ const started = new Promise<void>((r) => {
+ entered = r;
+ });
+ const store = createModelOverlayStore(createMemoryMap(), undefined, async () => ({
+ fingerprint: "context",
+ probe: async () => {
+ entered();
+ await new Promise<void>((r) => {
+ release = r;
+ });
+ },
+ }));
+ const pending = store.upsert(spec, "admin");
+ await started;
+ await store.refresh();
+ assert.equal(resolveModel(spec.id), undefined);
+ assert.deepEqual(await store.statuses(), []);
+ release();
+ await pending;
+ await store.refresh();
+ assert.ok(resolveModel(spec.id));
+});
+
+test("the store deadline leaves a hanging provider unverified", { timeout: 25_000 }, async () => {
+ const upstream = await verificationUpstream();
+ upstream.behavior.hang = true;
+ const built = buildApp(testConfig({ openaiApiKey: "test-key", providerBaseUrls: { openai: upstream.url + "/v1" } }));
+ try {
+ const start = Date.now();
+ await assert.rejects(
+ built.modelRegistry.upsert(spec, "admin"),
+ (error: unknown) => error instanceof ModelVerificationError && error.code === "timeout",
+ );
+ assert.ok(Date.now() - start < 20_000);
+ assert.deepEqual(await built.modelRegistry.statuses(), []);
+ } finally {
+ await upstream.close();
+ setProviderBaseUrls({});
+ }
+});
diff --git a/test/support/model-overlay-server.ts b/test/support/model-overlay-server.ts
index b34a6a759..79f9469a3 100644
--- a/test/support/model-overlay-server.ts
+++ b/test/support/model-overlay-server.ts
@@ -5,6 +5,7 @@ import { createInsecureTestServer } from "../../src/api/server.ts";
import { testConfig } from "./test-config.ts";
const config = testConfig({
+ providerBaseUrls: { openai: `${process.env.MODEL_OVERLAY_TEST_UPSTREAM}/v1` },
databaseUrl: process.env.MODEL_OVERLAY_TEST_DATABASE_URL,
adminGrants: "admin-alice:org_admin",
harness: "pi",
diff --git a/test/support/model-verification-upstream.ts b/test/support/model-verification-upstream.ts
new file mode 100644
index 000000000..616dd60d7
--- /dev/null
+++ b/test/support/model-verification-upstream.ts
@@ -0,0 +1,100 @@
+import { createServer } from "node:http";
+import type { AddressInfo } from "node:net";
+
+export async function verificationUpstream() {
+ const requests: {
+ path: string;
+ body: Record<string, unknown>;
+ authorization?: string;
+ key?: string;
+ beta?: string;
+ gatewayKey?: string;
+ }[] = [];
+ const behavior = { status: 200, empty: false, hang: false, rejectFast: false };
+ const server = createServer((req, res) => {
+ let raw = "";
+ req.on("data", (chunk) => {
+ raw += chunk;
+ });
+ req.on("end", () => {
+ const body = JSON.parse(raw);
+ requests.push({
+ path: req.url!,
+ body,
+ authorization: req.headers.authorization,
+ key: req.headers["x-api-key"] as string,
+ beta: req.headers["anthropic-beta"] as string,
+ gatewayKey: req.headers["x-gateway-key"] as string,
+ });
+ if (behavior.hang) return;
+ const status = behavior.rejectFast && body.speed === "fast" ? 400 : behavior.status;
+ if (status !== 200) {
+ res.writeHead(status, { "content-type": "application/json" });
+ return res.end(
+ JSON.stringify({
+ error: { message: `${status} private-provider-detail`, type: "invalid_request_error" },
+ }),
+ );
+ }
+ res.writeHead(200, { "content-type": "text/event-stream" });
+ const text = behavior.empty ? "" : "VERIFIED MODEL REPLY";
+ const send = (type: string, data: object) =>
+ res.write(`event: ${type}\ndata: ${JSON.stringify({ type, ...data })}\n\n`);
+ if (req.url?.endsWith("/messages")) {
+ send("message_start", {
+ message: {
+ id: "msg_probe",
+ type: "message",
+ role: "assistant",
+ content: [],
+ model: body.model,
+ stop_reason: null,
+ usage: { input_tokens: 5, output_tokens: 0 },
+ },
+ });
+ send("content_block_start", { index: 0, content_block: { type: "text", text: "" } });
+ send("content_block_delta", { index: 0, delta: { type: "text_delta", text } });
+ send("content_block_stop", { index: 0 });
+ send("message_delta", { delta: { stop_reason: "end_turn" }, usage: { output_tokens: 3 } });
+ send("message_stop", {});
+ } else {
+ const item = {
+ id: "msg_probe",
+ type: "message",
+ role: "assistant",
+ status: "completed",
+ content: [{ type: "output_text", text, annotations: [] }],
+ };
+ send("response.created", { response: { id: "resp_probe", status: "in_progress", output: [] } });
+ send("response.output_item.added", { output_index: 0, item: { ...item, status: "in_progress", content: [] } });
+ send("response.content_part.added", {
+ output_index: 0,
+ item_id: item.id,
+ content_index: 0,
+ part: { type: "output_text", text: "", annotations: [] },
+ });
+ send("response.output_text.delta", { output_index: 0, item_id: item.id, content_index: 0, delta: text });
+ send("response.output_item.done", { output_index: 0, item });
+ send("response.completed", {
+ response: {
+ id: "resp_probe",
+ status: "completed",
+ output: [item],
+ usage: { input_tokens: 5, output_tokens: 3, total_tokens: 8 },
+ },
+ });
+ }
+ res.end();
+ });
+ });
+ await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));
+ return {
+ url: `http://127.0.0.1:${(server.address() as AddressInfo).port}`,
+ requests,
+ behavior,
+ async close() {
+ server.closeAllConnections();
+ await new Promise<void>((r) => server.close(() => r()));
+ },
+ };
+}