Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

fix: a model is offered when something says it works, never when nothing does

Jack Rudenko4c48c63fe307
diff --git a/ai-docs/reports/BACKEND_v1414_contract_verification-20260923.md b/ai-docs/reports/BACKEND_v1414_contract_verification-20260923.md
index 7f3fecf8..369f427e 100644
--- a/ai-docs/reports/BACKEND_v1414_contract_verification-20260923.md
+++ b/ai-docs/reports/BACKEND_v1414_contract_verification-20260923.md
@@ -243,3 +243,63 @@ model access, not a missing model or a wrong route. Claudish reports it as an
account-level denial and now shows the provider's own sentence rather than an
inference about it (commit `196e6f8f`). Note also that Alibaba returns a *different
status* for one fact depending on `stream`.
+
+---
+
+## 6. Addendum, 2026-09-23: modality filtering went strict, and what that asks of the backend
+
+Section 3.5 asked whether "do not infer from names" covers names the catalog never published.
+The answer chosen was **yes, everywhere**, and claudish now implements it:
+
+- `NON_CHAT_PATTERNS` (17 regexes) and `VIDEO_OUTPUT_NAME_PATTERNS` (4) are **deleted**. No model
+ is classified by how its id is spelled.
+- `isChatCapable` requires a positive `"chat"` verdict. `"unknown"` is no longer offered.
+- Provider-published capability is read and outranks the catalog, because it describes the
+ deployment rather than the canonical model: Ollama's per-model `capabilities`
+ (`["completion","tools","vision"]` against `["embedding"]`) and LM Studio's `type`.
+- An explicit `provider@model` spec bypasses all of it. The user named the model; claudish sends
+ it. Verified: `routeExplicit` has no capability gate.
+
+**`supportsVision` was removed from claudish's chat inference.** It is an INPUT capability, and
+reading it as evidence of text output was the same class of guess as the name regexes. Measured:
+19 rows had no published output modality and `supportsVision` as their only chat signal, and 17
+were image, video, audio or moderation models. `sora-2` is the clean case — its sibling
+`sora-2-pro` publishes `outputModalities: ["video"]` and was excluded correctly, while `sora-2`
+has the field absent, so the vision flag was the only thing speaking for it.
+
+### The ask: 87 published models carry no `outputModalities`
+
+Measured on generation `g-20260922053223487-99e80e4b`, 87 of 1,136 catalog entries are now
+unavailable by bare name purely because nothing describes their output. Every one has
+`outputModalities` absent — not empty, absent.
+
+Most are correctly unavailable (`imagen-4.0-*`, `flux*`, `gpt-4o-*-tts`, `*-transcribe`,
+`omni-moderation-*`, `hailuo-02`, `lyria-3.5`, `nova-3-*`, `llama-rank-v1`). **These are not:**
+
+ chatgpt-image-latest is correctly excluded, but these look like genuine chat models —
+ gemini-2.0-flash-001, gemini-2.0-flash-lite-001, gemma-3-27b-pt, gemma-3-1b-pt,
+ gpt-3.5-turbo-0125, gpt-3.5-turbo-1106, gpt-4-0613, mixtral-8x7b-v0.1,
+ meta-llama-3.1-8b, llama-3.2-3b, qwen-2-72b, qwen-2-7b, qwen-2-1.5b,
+ molmo-7b-d-0924, kimi-k2.8-preview, nemotron-3-super, cwm, omen-alpha, big-pickle,
+ and the qwen3-*-base family (0.6b/1.7b/4b/8b/14b/30b-a3b)
+
+ plus two legacy completion models whose correct classification is a judgement call:
+ babbage-002, davinci-002
+
+Publishing `outputModalities` for those restores them. claudish will not guess them back.
+
+### What claudish reports instead of guessing
+
+`unavailableForMissingCapability(names)` returns `{catalogSilent, providerSilent}` and the
+discovery failure messages now name the silent party rather than saying "no chat-capable model":
+
+ no capability data for 3 of 12 listed models —
+ this endpoint publishes no capability field
+ (text-embedding-3-small, tts-1, whisper-1)
+
+`catalogSilent` is the list above — models-index work. `providerSilent` is an endpoint that
+publishes no capability field at all, which no backend work reaches: a plain OpenAI-compatible
+`/v1/models` returns `{id, object, created, owned_by}`. LiteLLM, vLLM, MLX and custom endpoints
+are in that bucket, and their models are reachable only by explicit `provider@model` until either
+the catalog describes them or the endpoint starts publishing. LiteLLM's `/model/info` does
+publish a `mode` field; reading it is not yet implemented and would shrink that bucket.
diff --git a/packages/cli/src/providers/catalog-client.ts b/packages/cli/src/providers/catalog-client.ts
index 070eadaa..0d90e31f 100644
--- a/packages/cli/src/providers/catalog-client.ts
+++ b/packages/cli/src/providers/catalog-client.ts
@@ -9,6 +9,7 @@ import {
} from "./all-models-cache.js";
import { catalogRouteMatchesProvider } from "./catalog-route-bindings.js";
import { CATALOG_V3_ACCEPT, type CatalogV3Envelope, parseCatalogV3Envelope } from "./catalog-v3.js";
+import { _clearChatCapabilityIndex } from "./transport/probe-discovery.js";
const DEFAULT_CATALOG_URL =
"https://us-central1-claudish-6da10.cloudfunctions.net/queryModels?status=all&catalog=slim&includeRouteVariants=true&limit=1000";
@@ -420,6 +421,13 @@ export async function refreshCatalog(
};
writeAllModelsCache(cache, options.cachePath);
+ // The chat-capability projection is memoized for 60s and keyed by cache path,
+ // so without this a refresh leaves it serving the PREVIOUS generation for up
+ // to a minute. That was survivable while an unrecognised model was offered
+ // anyway; now that admission requires positive evidence, a stale projection
+ // withholds models the refresh just described. Its doc comment already said
+ // "after a catalog refresh" — nothing had ever called it.
+ _clearChatCapabilityIndex();
_memCache = entries;
_warmPromise = Promise.resolve();
return {
diff --git a/packages/cli/src/providers/ollama-discovery.ts b/packages/cli/src/providers/ollama-discovery.ts
index 72c9180e..b3840acd 100644
--- a/packages/cli/src/providers/ollama-discovery.ts
+++ b/packages/cli/src/providers/ollama-discovery.ts
@@ -34,10 +34,14 @@ export function ollamaBaseUrl(): string {
interface FetchOllamaOptions {
/**
- * When true (default), enrich each model with capabilities via `/api/show`
- * (adds the tools indicator but one extra request per model). Pass false to
- * skip enrichment for a snappier interactive picker — embedding models are
- * still filtered out by name in that case.
+ * When true (default), fall back to `POST /api/show` for any model whose
+ * `/api/tags` row carried no `capabilities` array — one extra request per
+ * such model. Pass false to skip that and accept the silence.
+ *
+ * Ollama publishes `capabilities` inline in `/api/tags` (verified against a
+ * live daemon, 2026-09-23: every one of 20 installed models carried it), so
+ * on a current daemon this fallback never fires and the fan-out costs
+ * nothing. It stays for older daemons, which omit the field.
*/
enrichCapabilities?: boolean;
}
@@ -62,9 +66,11 @@ export async function fetchOllamaModels(options: FetchOllamaOptions = {}): Promi
const enriched = await Promise.all(
models.map(async (m) => {
- let capabilities: string[] = [];
+ // `/api/tags` publishes `capabilities` inline on a current daemon, so
+ // ask the list before spending a request per model on `/api/show`.
+ let capabilities: string[] = Array.isArray(m.capabilities) ? m.capabilities : [];
- if (enrichCapabilities) {
+ if (enrichCapabilities && capabilities.length === 0) {
try {
const showResponse = await fetch(`${host}/api/show`, {
method: "POST",
@@ -77,13 +83,19 @@ export async function fetchOllamaModels(options: FetchOllamaOptions = {}): Promi
capabilities = showData.capabilities || [];
}
} catch {
- // Ignore capability-fetch errors — fall back to name heuristics.
+ // Ignore capability-fetch errors — the model stays undescribed, and
+ // that is reported rather than guessed at from its name.
}
}
- const nameLower = String(m.name).toLowerCase();
const supportsTools = capabilities.includes("tools");
- const isEmbeddingModel = capabilities.includes("embedding") || nameLower.includes("embed");
+ // Ollama'"'"'s own word, and only Ollama'"'"'s word. This used to read
+ // `|| nameLower.includes("embed")`, which is the same class of guess as
+ // the deleted NON_CHAT_PATTERNS: it hides any model whose name happens to
+ // contain "embed" and misses every embedding model that does not say so.
+ // A daemon that reports nothing leaves `capabilities` empty, which reads
+ // as "not stated" everywhere downstream.
+ const isEmbeddingModel = capabilities.includes("embedding");
const sizeInfo = m.details?.parameter_size || "unknown size";
const toolsIndicator = supportsTools ? "✓ tools" : "✗ no tools";
diff --git a/packages/cli/src/providers/transport/probe-discovery-catalog-chat-capability.test.ts b/packages/cli/src/providers/transport/probe-discovery-catalog-chat-capability.test.ts
index 1df92ba6..e309e835 100644
--- a/packages/cli/src/providers/transport/probe-discovery-catalog-chat-capability.test.ts
+++ b/packages/cli/src/providers/transport/probe-discovery-catalog-chat-capability.test.ts
@@ -25,6 +25,7 @@ let tempDir = "";
let cachePath = "";
beforeEach(() => {
+ // The classifier memoizes its catalog projection across files; drop it so this fixture is read.
_clearChatCapabilityIndex();
tempDir = mkdtempSync(join(tmpdir(), "claudish-chat-capability-"));
cachePath = join(tempDir, "cloud-models-catalog-v3.json");
@@ -45,9 +46,6 @@ afterEach(() => {
});
describe("chat capability from the catalog", () => {
- // Mutation targets: probe-discovery.ts:150, :181 and :195. Removing the catalog
- // videoOutput denial breaks Omni; treating videoInput as output breaks 3.8 Flash;
- // requiring literal "chat" instead of anything but "not-chat" breaks unknowns.
test("video output is not chat", () => {
expect(classifyChatCapability("gemini-omni-1.1-flash", cachePath)).toBe("not-chat");
});
@@ -56,14 +54,17 @@ describe("chat capability from the catalog", () => {
expect(classifyChatCapability("gemini-3.8-flash", cachePath)).not.toBe("not-chat");
});
- test("the video-name fallback applies to names absent from the catalog", () => {
- expect(classifyChatCapability("unpublished-model-t2v", cachePath)).toBe("not-chat");
+ test("does not guess capability from non-chat-looking names", () => {
+ for (const name of ["nomic-embed-text", "whisper-large", "dall-e-3", "sora-2-lookalike"]) {
+ expect(classifyChatCapability(name, cachePath)).toBe("unknown");
+ }
});
test.each(["unpublished-model-t2v", "unpublished-chat-model"])(
"isChatCapable is the boolean projection for %s",
(name) => {
- expect(isChatCapable(name)).toBe(classifyChatCapability(name) !== "not-chat");
+ expect(isChatCapable(name)).toBe(classifyChatCapability(name) === "chat");
+ expect(isChatCapable(name)).toBeFalse();
}
);
});
diff --git a/packages/cli/src/providers/transport/probe-discovery-modalities.test.ts b/packages/cli/src/providers/transport/probe-discovery-modalities.test.ts
index 3c0ee6bf..f2b47809 100644
--- a/packages/cli/src/providers/transport/probe-discovery-modalities.test.ts
+++ b/packages/cli/src/providers/transport/probe-discovery-modalities.test.ts
@@ -3,7 +3,12 @@ import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { type DiskCacheV3, type SlimModelEntry, writeAllModelsCache } from "../all-models-cache.js";
-import { _clearChatCapabilityIndex, classifyChatCapability } from "./probe-discovery.js";
+import {
+ _clearChatCapabilityIndex,
+ classifyChatCapability,
+ isReportedChatCapable,
+ unavailableForMissingCapability,
+} from "./probe-discovery.js";
const CATALOG_GENERATION_ID = "g-20260920154425586-418ad3dd";
@@ -101,12 +106,29 @@ const contractRows: SlimModelEntry[] = [
outputModalities: ["text"],
videoOutput: true,
},
+ {
+ modelId: "contract-vision-only",
+ aliases: [],
+ supportsVision: true,
+ supportsTools: false,
+ },
+ {
+ modelId: "contract-tools-with-vision",
+ aliases: [],
+ supportsVision: true,
+ supportsTools: true,
+ },
+ {
+ modelId: "contract-catalog-silent",
+ aliases: [],
+ },
];
let tempDir = "";
let cachePath = "";
beforeEach(() => {
+ // The classifier memoizes its catalog projection across files; drop it so this fixture is read.
_clearChatCapabilityIndex();
tempDir = mkdtempSync(join(tmpdir(), "claudish-output-modalities-"));
cachePath = join(tempDir, "cloud-models-catalog-v3.json");
@@ -168,20 +190,46 @@ describe("chat capability from output modalities", () => {
expect(classifyChatCapability("google/flash-image-3.1", cachePath)).toBe("chat");
});
- // Contract-only route/name fallbacks: these strings are deliberately absent
- // from the cited live generation, as required by steps 1, 5, 6 and 8.
test("a wildcard route remains not-chat before catalog lookup", () => {
expect(classifyChatCapability("gemini/*", cachePath)).toBe("not-chat");
});
test.each(["unpublished-image-generator", "unpublished-model-t2v"])(
- "an absent model still uses the non-chat name rules: %s",
+ "an absent model remains unknown regardless of its name: %s",
(modelId) => {
- expect(classifyChatCapability(modelId, cachePath)).toBe("not-chat");
+ expect(classifyChatCapability(modelId, cachePath)).toBe("unknown");
}
);
test("an absent model with no matching name rule remains unknown", () => {
expect(classifyChatCapability("unpublished-chat-model", cachePath)).toBe("unknown");
});
+
+ test("provider evidence decides and outranks the catalog", () => {
+ expect(classifyChatCapability("gpt-5-image", cachePath)).toBe("chat");
+ expect(classifyChatCapability("gpt-5-image", cachePath, "not-chat")).toBe("not-chat");
+ expect(isReportedChatCapable("gpt-5-image", "not-chat")).toBeFalse();
+
+ expect(classifyChatCapability("nano-banana-pro", cachePath)).toBe("not-chat");
+ expect(classifyChatCapability("nano-banana-pro", cachePath, "chat")).toBe("chat");
+ expect(isReportedChatCapable("nano-banana-pro", "chat")).toBeTrue();
+ });
+
+ test("supportsVision is not chat evidence but supportsTools is", () => {
+ expect(classifyChatCapability("contract-vision-only", cachePath)).toBe("unknown");
+ expect(classifyChatCapability("contract-tools-with-vision", cachePath)).toBe("chat");
+ });
+
+ test("unavailable capability data separates catalog and provider silence", () => {
+ expect(
+ unavailableForMissingCapability(
+ ["contract-catalog-silent", "provider-only-undescribed"],
+ undefined,
+ cachePath
+ )
+ ).toEqual({
+ catalogSilent: ["contract-catalog-silent"],
+ providerSilent: ["provider-only-undescribed"],
+ });
+ });
});
diff --git a/packages/cli/src/providers/transport/probe-discovery.test.ts b/packages/cli/src/providers/transport/probe-discovery.test.ts
index 3d31ae23..3f780f71 100644
--- a/packages/cli/src/providers/transport/probe-discovery.test.ts
+++ b/packages/cli/src/providers/transport/probe-discovery.test.ts
@@ -6,14 +6,51 @@
* Run: bun test src/providers/transport/probe-discovery.test.ts
*/
-import { afterEach, beforeEach, describe, expect, mock, test } from "bun:test";
+import { afterAll, afterEach, beforeEach, describe, expect, mock, test } from "bun:test";
+import type { DiskCacheV3, SlimModelEntry } from "../all-models-cache.js";
+import * as __realAllModelsCache from "../all-models-cache.js";
+
+const __realAllModelsCacheExports = { ...__realAllModelsCache };
+
+function chatCatalogEntry(modelId: string): SlimModelEntry {
+ return { modelId, aliases: [], outputModalities: ["text"] };
+}
+
+const catalogFixture: DiskCacheV3 = {
+ version: 3,
+ lastUpdated: "2026-09-23T00:00:00.000Z",
+ catalogGenerationId: "probe-discovery-test",
+ entries: [
+ chatCatalogEntry("gpt-4o"),
+ chatCatalogEntry("gpt-4o-mini"),
+ chatCatalogEntry("claude-haiku-4"),
+ chatCatalogEntry("lite-x"),
+ chatCatalogEntry("model-a"),
+ ],
+ models: [],
+ plans: [],
+};
+
+// OpenAI-compatible listings carry no capability field, so these tests supply
+// the catalog evidence that admission uses instead of reading the user's cache.
+mock.module("../all-models-cache.js", () => ({
+ ...__realAllModelsCacheExports,
+ readAllModelsCache: (path?: string) =>
+ path ? __realAllModelsCacheExports.readAllModelsCache(path) : catalogFixture,
+}));
+
+afterAll(() => {
+ mock.module("../all-models-cache.js", () => __realAllModelsCacheExports);
+});
import {
+ _clearChatCapabilityIndex,
_clearProbeDiscoveryCache,
discoverViaLMStudio,
discoverViaOllama,
discoverViaOpenAIModels,
invalidateProbeDiscovery,
+ isReportedChatCapable,
rankProbeCandidates,
} from "./probe-discovery.js";
@@ -49,24 +86,17 @@ describe("rankProbeCandidates", () => {
expect(ranked[2]).toBe("model-70b"); // doesn't match the small pattern
});
- test("filters out non-chat-capable model names", () => {
- const ranked = rankProbeCandidates([
- "gpt-4o-mini",
- "dall-e-3",
- "text-embedding-3-small",
- "whisper-1",
- "tts-1",
- "gemini-2.0-flash-exp-image-generation",
- "voxtral-mini-latest",
- "claude-haiku-4",
- ]);
- // Only chat models survive. Order: small-name first, then alphabetical.
- expect(ranked).toEqual(["claude-haiku-4", "gpt-4o-mini"]);
+ test("orders candidates without re-judging capability and drops wildcards", () => {
+ expect(rankProbeCandidates(["b-model", "a-model", "gemini/*"])).toEqual(["a-model", "b-model"]);
});
- test("returns empty when all candidates are non-chat", () => {
- const ranked = rankProbeCandidates(["text-embedding-3-large", "whisper-1", "tts-1-hd"]);
- expect(ranked).toEqual([]);
+ test("provider admission excludes an embedding model before ranking", () => {
+ const admitted = [
+ isReportedChatCapable("vector-model", "not-chat") ? "vector-model" : null,
+ isReportedChatCapable("chat-model", "chat") ? "chat-model" : null,
+ ].filter((name): name is string => name !== null);
+
+ expect(rankProbeCandidates(admitted)).toEqual(["chat-model"]);
});
test("drops wildcard route patterns", () => {
@@ -98,9 +128,15 @@ describe("rankProbeCandidates", () => {
describe("discoverViaOpenAIModels", () => {
const realFetch = globalThis.fetch;
+ beforeEach(() => {
+ _clearProbeDiscoveryCache();
+ // The classifier memoizes its catalog projection across files; drop it so this fixture is read.
+ _clearChatCapabilityIndex();
+ });
afterEach(() => {
globalThis.fetch = realFetch;
_clearProbeDiscoveryCache();
+ _clearChatCapabilityIndex();
});
test("returns smallest model from /v1/models response", async () => {
@@ -236,132 +272,109 @@ describe("discoverViaOllama", () => {
const realFetch = globalThis.fetch;
beforeEach(() => {
_clearProbeDiscoveryCache();
+ // The classifier memoizes its catalog projection across files; drop it so this fixture is read.
+ _clearChatCapabilityIndex();
});
afterEach(() => {
globalThis.fetch = realFetch;
_clearProbeDiscoveryCache();
+ _clearChatCapabilityIndex();
});
const embedderOnlyPsBody =
- '{"models":[{"name":"nomic-embed-text:latest","model":"nomic-embed-text:latest","size":370031984,"digest":"0a109f422b47e3a30ba2b10eca18548e944e8a23073ee3f3e947efcf3c45e59f","details":{"parent_model":"","format":"gguf","family":"nomic-bert","families":["nomic-bert"],"parameter_size":"137M","quantization_level":"F16"},"expires_at":"2026-08-03T15:49:20.497918+10:00","size_vram":370031984,"context_length":2048}]}';
+ '{"models":[{"name":"nomic-embed-text:latest","model":"nomic-embed-text:latest","size":370031984,"capabilities":["embedding"],"digest":"0a109f422b47e3a30ba2b10eca18548e944e8a23073ee3f3e947efcf3c45e59f","details":{"parent_model":"","format":"gguf","family":"nomic-bert","families":["nomic-bert"],"parameter_size":"137M","quantization_level":"F16"},"expires_at":"2026-08-03T15:49:20.497918+10:00","size_vram":370031984,"context_length":2048}]}';
const tagsBody = `{"models":[
- {"name":"nomic-embed-text:latest","size":370031984},
- {"name":"gemma4:31b-cloud","size":312},
- {"name":"qwen3.5:0.8b-mlx","size":1244127078},
- {"name":"vl-bu-30b-a3b-preview:Q4_K_M","size":19715685721}
+ {"name":"nomic-embed-text:latest","size":370031984,"capabilities":["embedding"]},
+ {"name":"gemma4:31b-cloud","size":312,"capabilities":["completion"]},
+ {"name":"qwen3.5:0.8b-mlx","size":1244127078,"capabilities":["completion","tools"]},
+ {"name":"vl-bu-30b-a3b-preview:Q4_K_M","size":19715685721,"capabilities":["completion","vision"]}
]}`;
const loadedChatPsBody =
- '{"models":[{"name":"vl-bu-30b-a3b-preview:Q4_K_M","size":19715685721}]}';
+ '{"models":[{"name":"vl-bu-30b-a3b-preview:Q4_K_M","size":19715685721,"capabilities":["completion","vision"]}]}';
const emptyModelsBody = '{"models":[]}';
- const embedderOnlyTagsBody = '{"models":[{"name":"nomic-embed-text:latest","size":370031984}]}';
-
- function serveOllama(psBody: string, allTagsBody: string) {
- return Bun.serve({
- port: 0,
- fetch(req) {
- const path = new URL(req.url).pathname;
- if (path === "/api/ps") {
- return new Response(psBody, {
- status: 200,
- headers: { "Content-Type": "application/json" },
- });
- }
- if (path === "/api/tags") {
- return new Response(allTagsBody, {
- status: 200,
- headers: { "Content-Type": "application/json" },
- });
- }
- return new Response(null, { status: 404 });
- },
- });
+ const embedderOnlyTagsBody =
+ '{"models":[{"name":"nomic-embed-text:latest","size":370031984,"capabilities":["embedding"]}]}';
+
+ function mockOllama(psBody: string, allTagsBody: string) {
+ globalThis.fetch = mock(async (input: string | URL | Request) => {
+ const path = new URL(String(input)).pathname;
+ if (path === "/api/ps") {
+ return new Response(psBody, {
+ status: 200,
+ headers: { "Content-Type": "application/json" },
+ });
+ }
+ if (path === "/api/tags") {
+ return new Response(allTagsBody, {
+ status: 200,
+ headers: { "Content-Type": "application/json" },
+ });
+ }
+ return new Response(null, { status: 404 });
+ }) as unknown as typeof fetch;
}
test("falls through from a loaded embedder to chat models on disk", async () => {
- const server = serveOllama(embedderOnlyPsBody, tagsBody);
- try {
- const outcome = await discoverViaOllama(`http://localhost:${server.port}`, {
- key: "embedder-loaded-tags-chat",
- });
- expect(outcome.model).not.toBeNull();
- expect(outcome.reason ?? "").not.toMatch(/only embedding\/non-chat/);
- } finally {
- server.stop(true);
- }
+ mockOllama(embedderOnlyPsBody, tagsBody);
+ const outcome = await discoverViaOllama("http://localhost:11434", {
+ key: "embedder-loaded-tags-chat",
+ });
+ expect(outcome.model).not.toBeNull();
+ expect(outcome.reason ?? "").not.toMatch(/only embedding\/non-chat/);
});
test("prefers a loaded chat model over smaller models on disk", async () => {
- const server = serveOllama(loadedChatPsBody, tagsBody);
- try {
- const outcome = await discoverViaOllama(`http://localhost:${server.port}`, {
- key: "loaded-before-smaller-tags",
- });
- expect(outcome.model).toBe("vl-bu-30b-a3b-preview:Q4_K_M");
- } finally {
- server.stop(true);
- }
+ mockOllama(loadedChatPsBody, tagsBody);
+ const outcome = await discoverViaOllama("http://localhost:11434", {
+ key: "loaded-before-smaller-tags",
+ });
+ expect(outcome.model).toBe("vl-bu-30b-a3b-preview:Q4_K_M");
});
test("falls through the cached ranked list when the loaded model is excluded", async () => {
- const server = serveOllama(loadedChatPsBody, tagsBody);
- try {
- const baseUrl = `http://localhost:${server.port}`;
- const cacheKey = "exclude-loaded-from-cached-list";
- expect((await discoverViaOllama(baseUrl, { key: cacheKey })).model).toBe(
- "vl-bu-30b-a3b-preview:Q4_K_M"
- );
-
- // Reuse the same key: exclude must walk the ranked list cached above.
- const outcome = await discoverViaOllama(baseUrl, {
- key: cacheKey,
- exclude: new Set(["vl-bu-30b-a3b-preview:Q4_K_M"]),
- });
- expect(outcome.model).not.toBeNull();
- expect(["gemma4:31b-cloud", "qwen3.5:0.8b-mlx", "vl-bu-30b-a3b-preview:Q4_K_M"]).toContain(
- outcome.model!
- );
- expect(outcome.model).not.toBe("vl-bu-30b-a3b-preview:Q4_K_M");
- } finally {
- server.stop(true);
- }
+ mockOllama(loadedChatPsBody, tagsBody);
+ const baseUrl = "http://localhost:11434";
+ const cacheKey = "exclude-loaded-from-cached-list";
+ expect((await discoverViaOllama(baseUrl, { key: cacheKey })).model).toBe(
+ "vl-bu-30b-a3b-preview:Q4_K_M"
+ );
+
+ // Reuse the same key: exclude must walk the ranked list cached above.
+ const outcome = await discoverViaOllama(baseUrl, {
+ key: cacheKey,
+ exclude: new Set(["vl-bu-30b-a3b-preview:Q4_K_M"]),
+ });
+ expect(outcome.model).not.toBeNull();
+ expect(["gemma4:31b-cloud", "qwen3.5:0.8b-mlx", "vl-bu-30b-a3b-preview:Q4_K_M"]).toContain(
+ outcome.model!
+ );
+ expect(outcome.model).not.toBe("vl-bu-30b-a3b-preview:Q4_K_M");
});
test("prefers the smallest chat model within the unloaded tier", async () => {
- const server = serveOllama(emptyModelsBody, tagsBody);
- try {
- const outcome = await discoverViaOllama(`http://localhost:${server.port}`, {
- key: "smallest-unloaded-chat",
- });
- expect(outcome.model).toBe("gemma4:31b-cloud");
- } finally {
- server.stop(true);
- }
+ mockOllama(emptyModelsBody, tagsBody);
+ const outcome = await discoverViaOllama("http://localhost:11434", {
+ key: "smallest-unloaded-chat",
+ });
+ expect(outcome.model).toBe("gemma4:31b-cloud");
});
test("reports a clear error when both Ollama endpoints are empty", async () => {
- const server = serveOllama(emptyModelsBody, emptyModelsBody);
- try {
- const outcome = await discoverViaOllama(`http://localhost:${server.port}`, {
- key: "genuinely-empty",
- });
- expect(outcome.model).toBeNull();
- expect(outcome.reason).toContain("no models on");
- } finally {
- server.stop(true);
- }
+ mockOllama(emptyModelsBody, emptyModelsBody);
+ const outcome = await discoverViaOllama("http://localhost:11434", {
+ key: "genuinely-empty",
+ });
+ expect(outcome.model).toBeNull();
+ expect(outcome.reason).toContain("no models on");
});
test("keeps the non-chat reason when every Ollama model is an embedder", async () => {
- const server = serveOllama(embedderOnlyPsBody, embedderOnlyTagsBody);
- try {
- const outcome = await discoverViaOllama(`http://localhost:${server.port}`, {
- key: "embedder-only-everywhere",
- });
- expect(outcome.model).toBeNull();
- expect(outcome.reason).toMatch(/only embedding\/non-chat/);
- } finally {
- server.stop(true);
- }
+ mockOllama(embedderOnlyPsBody, embedderOnlyTagsBody);
+ const outcome = await discoverViaOllama("http://localhost:11434", {
+ key: "embedder-only-everywhere",
+ });
+ expect(outcome.model).toBeNull();
+ expect(outcome.reason).toMatch(/described as non-chat/);
});
test("prefers smaller loaded model from /api/ps", async () => {
@@ -371,8 +384,8 @@ describe("discoverViaOllama", () => {
return new Response(
JSON.stringify({
models: [
- { name: "llama3-70b", size: 70_000_000_000 },
- { name: "llama3-3b", size: 3_000_000_000 },
+ { name: "llama3-70b", size: 70_000_000_000, capabilities: ["completion"] },
+ { name: "llama3-3b", size: 3_000_000_000, capabilities: ["completion"] },
],
}),
{ status: 200 }
@@ -395,8 +408,8 @@ describe("discoverViaOllama", () => {
return new Response(
JSON.stringify({
models: [
- { name: "qwen-7b", size: 7_000_000_000 },
- { name: "tinyllama-1b", size: 1_000_000_000 },
+ { name: "qwen-7b", size: 7_000_000_000, capabilities: ["completion"] },
+ { name: "tinyllama-1b", size: 1_000_000_000, capabilities: ["completion"] },
],
}),
{ status: 200 }
@@ -428,9 +441,13 @@ describe("discoverViaOllama", () => {
return new Response(
JSON.stringify({
models: [
- { name: "all-minilm:latest", size: 45_000_000 }, // smallest BUT embedding
- { name: "nomic-embed-text", size: 274_000_000 }, // embedding
- { name: "llama-3.2-3b", size: 3_000_000_000 }, // chat, bigger
+ {
+ name: "vectorizer-small",
+ size: 45_000_000,
+ capabilities: ["embedding"],
+ }, // smallest BUT embedding; its name gives no hint
+ { name: "retrieval-base", size: 274_000_000, capabilities: ["embedding"] },
+ { name: "llama-3.2-3b", size: 3_000_000_000, capabilities: ["completion"] },
],
}),
{ status: 200 }
@@ -448,9 +465,15 @@ describe("discoverViaOllama", () => {
describe("discoverViaLMStudio", () => {
const realFetch = globalThis.fetch;
+ beforeEach(() => {
+ _clearProbeDiscoveryCache();
+ // The classifier memoizes its catalog projection across files; drop it so this fixture is read.
+ _clearChatCapabilityIndex();
+ });
afterEach(() => {
globalThis.fetch = realFetch;
_clearProbeDiscoveryCache();
+ _clearChatCapabilityIndex();
});
test("prefers loaded models over not-loaded ones", async () => {
diff --git a/packages/cli/src/providers/transport/probe-discovery.ts b/packages/cli/src/providers/transport/probe-discovery.ts
index 53124b62..7eb39b19 100644
--- a/packages/cli/src/providers/transport/probe-discovery.ts
+++ b/packages/cli/src/providers/transport/probe-discovery.ts
@@ -39,51 +39,38 @@ const SMALL_MODEL_PATTERNS = [
];
/**
- * Models that cannot answer a chat turn: image generation, embeddings, TTS,
- * speech-to-text. They appear in `/v1/models` lists beside chat models and 404
- * or 400 a probe.
- *
- * A NEGATIVE name rule: it can deny, never confirm. It is now the FALLBACK, not
- * the rule: the cloud models catalog publishes output modalities, so for a model
- * it describes the published list decides and this list is never consulted. It
- * still decides for what the catalog leaves without an output modality — local
- * servers (Ollama, LM Studio), custom endpoints, LiteLLM deployments, and catalog
- * rows that carry no modality field.
+ * What a provider's OWN listing said about a model, when it said anything.
+ *
+ * This is evidence, not a guess. Ollama publishes a `capabilities` array per
+ * model in `GET /api/tags` — `["completion","tools","vision"]` for a chat model,
+ * `["embedding"]` for an embedding one — and LM Studio publishes a `type`. It
+ * describes THIS deployment rather than the canonical model, which is why it
+ * outranks the catalog below: a local embedding build of a name the catalog
+ * knows as a chat model is an embedding model here.
+ *
+ * `undefined` means the provider's listing carried no capability field. That is
+ * a silence, and it stays a silence — see {@link classifyChatCapability}.
*/
-const NON_CHAT_PATTERNS = [
- /\bimage\b/i,
- /\bembed/i, // embedding, embeddings
- /\bminilm\b/i, // sentence-transformers MiniLM family (Ollama lists these)
- /\bnomic-embed/i, // nomic embedding models (Ollama)
- /\bbge-/i, // BAAI BGE embeddings
- /\bmxbai-embed/i, // MixedBread AI embeddings
- /\btts\b/i,
- /\bwhisper\b/i,
- /\baudio\b/i,
- /\bvoxtral\b/i,
- /\bdall-?e\b/i,
- /\bmoderation\b/i,
- /\brerank/i,
- /\bspeech\b/i,
- /-(image|tts|audio|embedding|vision-only)(-|$)/i,
-];
+export type ReportedCapability = "chat" | "not-chat" | undefined;
/**
- * Ids that LOOK like video generators — a fallback the catalog overrides.
- *
- * Kept apart from {@link NON_CHAT_PATTERNS} because the catalog publishes
- * `videoOutput` (a defined boolean, `false` included), so for any model it knows
- * the fact replaces this guess. The guess still matters for models the catalog
- * does not cover. One word, two directions: `\bvideo\b` cannot tell a video
- * GENERATOR from a model that READS video, and video input alone never excludes a
- * chat model — which is why a published `videoOutput: false` switches this off.
+ * Ollama's `/api/tags` and `/api/ps` rows carry `capabilities`. `completion`
+ * means it answers chat turns; `embedding` means it does not. An empty or
+ * missing array is a silence, never a denial — older daemons omit the field.
*/
-const VIDEO_OUTPUT_NAME_PATTERNS = [
- /\bvideo\b/i, // video-01, wan2.2-video, hunyuan-video
- /(^|[-_.])(t2v|i2v|r2v|v2v)([-_.]|$)/i, // happyhorse-1.1-t2v / -i2v / -r2v; MiniMax T2V-01
- /\bveo\b/i, // Google Veo
- /\bsora\b/i, // OpenAI Sora
-];
+function ollamaReported(row: { capabilities?: unknown }): ReportedCapability {
+ const caps = Array.isArray(row.capabilities) ? (row.capabilities as string[]) : [];
+ if (caps.includes("completion")) return "chat";
+ if (caps.includes("embedding")) return "not-chat";
+ return undefined;
+}
+
+/** LM Studio's `/api/v0/models` rows carry `type`: `llm`/`vlm` chat, `embeddings` not. */
+function lmStudioReported(row: { type?: unknown }): ReportedCapability {
+ if (row.type === "llm" || row.type === "vlm") return "chat";
+ if (row.type === "embeddings" || row.type === "embedding") return "not-chat";
+ return undefined;
+}
function isSmallName(name: string): boolean {
return SMALL_MODEL_PATTERNS.some((re) => re.test(name));
@@ -92,19 +79,18 @@ function isSmallName(name: string): boolean {
/**
* What is known about whether a model can answer a chat turn.
*
- * The cloud models catalog's OUTPUT MODALITY is the evidence; the name rules are
- * the fallback for models it does not describe.
- *
- * - `"not-chat"` — the catalog's published output modalities exclude `"text"`
- * (image, audio, video, embeddings, transcription, speech, rerank, decisions);
- * or, for a model with no published output modality, `videoOutput: true`, a
- * {@link NON_CHAT_PATTERNS} match, or a LiteLLM wildcard route. Never offered.
- * - `"chat"` — the catalog's published output modalities include `"text"`; or,
- * with none published, the catalog declares a chat-shaped capability (tools,
- * thinking or vision).
- * - `"unknown"` — neither. Still offered. Rounding it to `"chat"` would assert
- * what nothing established; rounding it to `"not-chat"` would hide a newly
- * shipped chat model with no error and no trace.
+ * Two sources of evidence, and no guesses: the provider's own published
+ * capability for THIS deployment, and the cloud models catalog's published
+ * output modality for the canonical model.
+ *
+ * - `"not-chat"` — a provider or catalog statement that it produces something
+ * other than text, or a LiteLLM wildcard route. Never offered.
+ * - `"chat"` — a provider or catalog statement that it answers chat turns.
+ * - `"unknown"` — NOBODY said. Not offered either, and that is the point: an
+ * unknown used to be treated as chat, so an embedding model nothing described
+ * was offered as one. A model is now shown when it is known to work, and its
+ * absence is counted and attributed rather than guessed around — see
+ * {@link unavailableForMissingCapability}.
*/
export type ChatCapability = "chat" | "not-chat" | "unknown";
@@ -125,6 +111,14 @@ interface CatalogCapabilityIndex {
videoOutput: Set<string>;
/** Ids with ANY published `videoOutput`, `false` included: a statement, not a silence. */
videoOutputKnown: Set<string>;
+ /**
+ * Every id the catalog has a row for, whatever it says about it.
+ *
+ * Membership separates the two silences {@link unavailableForMissingCapability}
+ * has to tell apart: a model the backend publishes but has not described, and a
+ * name the backend has never heard of.
+ */
+ known: Set<string>;
}
const _catalogChatIndex = new Map<string, { index: CatalogCapabilityIndex; expiresAt: number }>();
@@ -171,9 +165,11 @@ function catalogCapabilityIndex(cachePath?: string): CatalogCapabilityIndex {
nonTextOutput: new Set(),
videoOutput: new Set(),
videoOutputKnown: new Set(),
+ known: new Set(),
};
for (const entry of readAllModelsCache(cachePath)?.entries ?? []) {
const keys = [catalogKey(entry.modelId), ...(entry.aliases ?? []).map(catalogKey)];
+ for (const k of keys) index.known.add(k);
// Filed BEFORE the `videoOutput: true` `continue` below, so a video generator
// still contributes its published output modality — and where the two disagree
// the modality list wins, being the more specific statement.
@@ -188,10 +184,22 @@ function catalogCapabilityIndex(cachePath?: string): CatalogCapabilityIndex {
continue;
}
// Positive flags only: `undefined` means no opinion and must stay `unknown`.
- const chatShaped =
- entry.supportsTools === true ||
- entry.supportsThinking === true ||
- entry.supportsVision === true;
+ //
+ // `supportsVision` is NOT among them, and used to be. Vision is an INPUT
+ // capability, and reading it as evidence of TEXT OUTPUT is the same kind of
+ // guess as the deleted name regexes — it just wore a catalog field. Measured
+ // on generation `g-20260922053223487-99e80e4b`: 19 rows had no published
+ // output modality and `supportsVision` as their only chat signal, and 17 of
+ // them were image, video, audio or moderation models — `sora-2`,
+ // `gpt-image-2.5-*`, `chatgpt-image-latest`, `seedream-4.5`, `lyria-3.5`,
+ // `omni-moderation-latest`, `hailuo-02`. `sora-2-pro` publishes
+ // `outputModalities: ["video"]` and was excluded correctly; `sora-2` is the
+ // same model family with the field missing, so the flag was all that spoke.
+ //
+ // `supportsTools` and `supportsThinking` stay: tool calling and reasoning are
+ // behaviours of a model that emits text, so they are evidence ABOUT output.
+ // 207 rows rely on `supportsTools` alone, and none on `supportsThinking`.
+ const chatShaped = entry.supportsTools === true || entry.supportsThinking === true;
if (chatShaped) for (const k of keys) index.chat.add(k);
}
_catalogChatIndex.set(key, { index, expiresAt: Date.now() + CATALOG_CHAT_INDEX_TTL_MS });
@@ -201,45 +209,164 @@ function catalogCapabilityIndex(cachePath?: string): CatalogCapabilityIndex {
/**
* Classify whether a model can answer a chat turn — see {@link ChatCapability}.
*
- * The cloud models catalog's published OUTPUT MODALITY outranks every name rule,
- * in both directions: a model named like an image generator whose published output
- * is `["text"]` IS a chat model, and a model with a blameless name whose output is
- * `["image"]` is NOT. Only where the catalog publishes no output modality — local
- * servers, custom endpoints, LiteLLM deployments, catalog rows without the field —
- * do the older `videoOutput` boolean and the name rules decide.
+ * NO NAME RULES. There used to be two lists of regexes — `/\bembed/`, `/\btts\b/`,
+ * `/\bwhisper\b/`, `/\bvideo\b/`, `/\bsora\b/` and so on — consulted whenever the
+ * catalog published no output modality. They are deleted. A regex over a model id
+ * is a guess, it silently miscategorises every name that does not follow the
+ * convention it encodes, and it produced a filter nobody could reason about: a
+ * model was hidden because of how it was spelled.
+ *
+ * What replaced them is evidence the providers were already publishing and
+ * claudish was not reading. Ollama returns `capabilities` per model in
+ * `GET /api/tags` (`["embedding"]` for `nomic-embed-text`), LM Studio returns a
+ * `type`. Callers that hold such a listing pass it as `reported`.
*
- * Order: wildcard route, non-text output, text output, `videoOutput: true`,
- * {@link NON_CHAT_PATTERNS}, the video name guess (only while `videoOutput` is
- * unpublished), the chat-shaped capability flags, then `"unknown"`.
+ * `reported` outranks the catalog because it is the more specific statement: the
+ * catalog describes a canonical model, the provider describes the build it will
+ * actually serve.
+ *
+ * Order: wildcard route, the provider's own statement, non-text output, text
+ * output, `videoOutput: true`, the chat-shaped capability flags, then
+ * `"unknown"` — which is now a refusal, not a pass.
*
* @param cachePath Override the catalog cache path. Tests only.
+ * @param reported The provider's own capability for this model, when its
+ * listing published one.
*/
-export function classifyChatCapability(name: string, cachePath?: string): ChatCapability {
+export function classifyChatCapability(
+ name: string,
+ cachePath?: string,
+ reported?: ReportedCapability
+): ChatCapability {
// Wildcard entries ("gemini/*") are LiteLLM route patterns, not models.
if (name.includes("*")) return "not-chat";
+ if (reported) return reported;
+
const index = catalogCapabilityIndex(cachePath);
const key = catalogKey(name);
- // Catalog evidence, ahead of every name rule.
if (index.nonTextOutput.has(key)) return "not-chat";
if (index.textOutput.has(key)) return "chat";
-
if (index.videoOutput.has(key)) return "not-chat";
- if (NON_CHAT_PATTERNS.some((re) => re.test(name))) return "not-chat";
- if (!index.videoOutputKnown.has(key) && VIDEO_OUTPUT_NAME_PATTERNS.some((re) => re.test(name))) {
- return "not-chat";
- }
if (index.chat.has(key)) return "chat";
return "unknown";
}
/**
- * Whether a model may be offered as a chat model: everything but `"not-chat"`.
+ * Whether a model may be offered as a chat model: ONLY a known `"chat"`.
+ *
+ * This used to be `!== "not-chat"`, which offered every `"unknown"` as well.
+ * That is the wrong default in the only case that matters: nothing had
+ * established the model answers chat turns, so the list included whatever the
+ * filter failed to recognise — the reason a name-regex fallback had to exist at
+ * all, and the reason embedding models kept reappearing whenever a name did not
+ * match one. Requiring positive evidence removes the need to guess.
+ *
+ * The cost is visible rather than silent: {@link unavailableForMissingCapability}
+ * counts what this excludes and says whose data is missing.
+ *
* A projection of {@link classifyChatCapability}, so every caller shares one rule.
*/
export function isChatCapable(name: string): boolean {
- return classifyChatCapability(name) !== "not-chat";
+ return classifyChatCapability(name) === "chat";
+}
+
+/**
+ * The same rule for a caller that holds the provider's own listing.
+ *
+ * Separate from {@link isChatCapable} rather than an optional second parameter,
+ * because every caller filters with `.filter(isChatCapable)` and `Array.filter`
+ * passes `(value, index, array)` — an optional second parameter would silently
+ * receive the array index as the provider's capability.
+ */
+export function isReportedChatCapable(name: string, reported: ReportedCapability): boolean {
+ return classifyChatCapability(name, undefined, reported) === "chat";
+}
+
+/** Why a model could not be offered, when the reason is absent data. */
+export interface MissingCapabilityReport {
+ /** Ids the catalog publishes but leaves without an output modality. */
+ catalogSilent: string[];
+ /** Ids no catalog row describes and whose provider listing published no capability. */
+ providerSilent: string[];
+}
+
+/**
+ * Split the models this filter excluded for LACK OF DATA from the ones it
+ * excluded on evidence, and say which source was silent.
+ *
+ * The two buckets are different people's work. `catalogSilent` is a models-index
+ * crawler gap: the backend publishes the model but not its
+ * `outputModalities` — 52 rows on generation `g-20260922053223487-99e80e4b`,
+ * almost all of them genuinely not chat (`imagen-4.0-*`, `flux.1-kontext-*`,
+ * `gemini-3.5-transcribe`). `providerSilent` is an endpoint gap: a plain
+ * OpenAI-compatible `/v1/models` returns `{id, object, created, owned_by}` and
+ * nothing about capability, so nobody has ever described those models.
+ *
+ * Reported rather than guessed around. A count with names attached is something
+ * a backend engineer can act on; a regex that hides the row is not.
+ */
+export function unavailableForMissingCapability(
+ names: readonly string[],
+ reportedFor?: (name: string) => ReportedCapability,
+ cachePath?: string
+): MissingCapabilityReport {
+ const catalogSilent: string[] = [];
+ const providerSilent: string[] = [];
+ for (const name of names) {
+ const reported = reportedFor?.(name);
+ if (classifyChatCapability(name, cachePath, reported) !== "unknown") continue;
+ const index = catalogCapabilityIndex(cachePath);
+ if (index.known.has(catalogKey(name))) catalogSilent.push(name);
+ else providerSilent.push(name);
+ }
+ return { catalogSilent, providerSilent };
+}
+
+/**
+ * Say WHY a listing produced no probe candidate, and whose data is missing.
+ *
+ * The old message was `no chat-capable model among N listed`, which is now
+ * ambiguous in a way that matters. Two different endpoints produce it:
+ *
+ * - one serving nothing but image and embedding models, which is working
+ * correctly and has nothing to fix;
+ * - one whose models nobody has ever described, which is a data gap — and
+ * since claudish stopped guessing from names, this is the case that grew.
+ *
+ * Naming the silent party is the whole point. `catalogSilent` is a models-index
+ * row published without `outputModalities`, which a backend engineer can fix
+ * from this list. `providerSilent` is an endpoint that publishes no capability
+ * field at all — a plain OpenAI-compatible `/v1/models` returns
+ * `{id, object, created, owned_by}` — which no amount of backend work reaches.
+ *
+ * Names are included, capped, because "3 models" sends nobody anywhere and
+ * `text-embedding-3-small, tts-1, whisper-1` answers the question on sight.
+ */
+function describeNoCandidates(
+ ids: readonly string[],
+ reportedFor?: (name: string) => ReportedCapability,
+ cachePath?: string
+): string {
+ const { catalogSilent, providerSilent } = unavailableForMissingCapability(
+ ids,
+ reportedFor,
+ cachePath
+ );
+ const undescribed = catalogSilent.length + providerSilent.length;
+ if (undescribed === 0) {
+ return `all ${ids.length} listed models are described as non-chat (image, embedding, audio)`;
+ }
+ const sample = [...catalogSilent, ...providerSilent].slice(0, 3).join(", ");
+ const more = undescribed > 3 ? `, +${undescribed - 3} more` : "";
+ const whose =
+ providerSilent.length === 0
+ ? "the models catalog publishes them without modalities"
+ : catalogSilent.length === 0
+ ? "this endpoint publishes no capability field"
+ : "neither the models catalog nor this endpoint describes them";
+ return `no capability data for ${undescribed} of ${ids.length} listed models — ${whose} (${sample}${more})`;
}
/**
@@ -274,16 +401,31 @@ function isStandardName(name: string): boolean {
}
/**
- * Rank models for probe selection. Layered preference (highest priority
- * decides first):
- * 1. Chat-capable (filter): drop image/embedding/audio/wildcard rows.
- * 2. Standard names: prefer bare or recognized-vendor-prefixed IDs
+ * ORDER models for probe selection. It no longer filters — see below.
+ *
+ * 1. Standard names: prefer bare or recognized-vendor-prefixed IDs
* over deployment-specific aliases (e.g. `gem-mad/...`).
- * 3. Small variants: prefer mini/nano/flash/lite/haiku/Nb.
- * 4. Tiebreak: alphabetical for determinism.
+ * 2. Small variants: prefer mini/nano/flash/lite/haiku/Nb.
+ * 3. Tiebreak: alphabetical for determinism.
+ *
+ * It used to open with `.filter(isChatCapable)`, which was harmless while that
+ * predicate judged names and became destructive the moment capability came from
+ * the provider instead. Its callers filter FIRST, holding the provider's listing
+ * — Ollama's `capabilities`, LM Studio's `type` — and then passed bare strings
+ * in here, where a second pass re-judged them with the evidence stripped off and
+ * discarded every one. Measured: LM Studio rows carrying `type: "llm"` survived
+ * their own filter and then vanished here, and the endpoint reported no probe
+ * model at all.
+ *
+ * Ranking and admission are now separate jobs, and only the caller holding the
+ * evidence decides admission.
*/
export function rankProbeCandidates(names: string[]): string[] {
- return names.filter(isChatCapable).sort((a, b) => {
+ // One exclusion survives here, and it is not a capability judgement: a name
+ // containing `*` is a LiteLLM ROUTE PATTERN (`gemini/*`), not a model id, so
+ // there is nothing to send it. Dropping it is rejecting a non-identifier, not
+ // guessing what it can do — which is why it stays after the name rules went.
+ return names.filter((name) => !name.includes("*")).sort((a, b) => {
const aStd = isStandardName(a);
const bStd = isStandardName(b);
if (aStd !== bStd) return aStd ? -1 : 1;
@@ -424,9 +566,18 @@ export async function discoverViaOpenAIModels(
return { model: null, reason };
}
- const ranked = rankProbeCandidates(ids);
+ // A plain OpenAI-compatible `/v1/models` row is `{id, object, created,
+ // owned_by}` — there is no capability field to read, so admission here rests
+ // on the catalog alone.
+ const ranked = rankProbeCandidates(ids.filter(isChatCapable));
if (ranked.length === 0) {
- const reason = `no chat-capable model among ${ids.length} listed`;
+ // "No chat-capable model" is now two different facts and the reader has to
+ // be told which. Every model DESCRIBED as non-chat is a working endpoint
+ // full of image or embedding models. Every model described by NOBODY is a
+ // data gap, and naming whose gap it is turns a dead end into a fixable
+ // one — a models-index row missing its modalities, or an endpoint that
+ // publishes no capability field at all.
+ const reason = describeNoCandidates(ids);
cacheSetFailure(cacheKey.key, reason);
return { model: null, reason };
}
@@ -519,6 +670,13 @@ function extractModelIds(body: unknown): string[] {
interface OllamaModel {
name: string;
size?: number;
+ /**
+ * The daemon's own statement of what this model does: `["completion","tools",
+ * "vision"]` for a chat model, `["embedding"]` for an embedding one. Optional
+ * because daemons older than the field omit it, and an omission is a silence
+ * rather than a denial — see {@link ollamaReported}.
+ */
+ capabilities?: string[];
}
/**
@@ -585,16 +743,36 @@ export async function discoverViaOllama(
// Filter out embedding/image/TTS models — they're listed in /api/tags
// alongside chat models but will 404 on /v1/chat/completions.
- const loaded = loadedRaw.filter((m) => isChatCapable(m.name));
+ //
+ // Ollama says which is which itself: each row carries `capabilities`,
+ // `["completion","tools","vision"]` against `["embedding"]`. Passing it as
+ // `reported` is what let the name regexes go — `nomic-embed-text` is excluded
+ // because the daemon called it an embedding model, not because of how it is
+ // spelled. A row from an older daemon that carries no array stays unstated.
+ const loaded = loadedRaw.filter((m) => isReportedChatCapable(m.name, ollamaReported(m)));
const loadedNames = new Set(loaded.map((m) => m.name));
- const rest = tagsRaw.filter((m) => isChatCapable(m.name) && !loadedNames.has(m.name));
+ const rest = tagsRaw.filter(
+ (m) => isReportedChatCapable(m.name, ollamaReported(m)) && !loadedNames.has(m.name)
+ );
if (loaded.length === 0 && rest.length === 0) {
+ // A daemon old enough to omit `capabilities` lands here with a full model
+ // list and no way to tell chat from embedding, which is a different problem
+ // from a daemon that genuinely only holds embedders. `describeNoCandidates`
+ // separates them and names the fix; `ollama pull` is not it.
+ const listed = [...loadedRaw, ...tagsRaw];
+ const reportedFor = (name: string): ReportedCapability => {
+ const row = listed.find((m) => m.name === name);
+ return row ? ollamaReported(row) : undefined;
+ };
const reason =
connectionError ??
- (loadedRaw.length === 0 && tagsRaw.length === 0
+ (listed.length === 0
? `no models on ${baseUrl} (pull one: ollama pull llama3.2)`
- : `only embedding/non-chat models on ${baseUrl}`);
+ : `${describeNoCandidates(
+ listed.map((m) => m.name),
+ reportedFor
+ )} on ${baseUrl}`);
cacheSetFailure(cacheKey.key, reason);
return { model: null, reason };
}
@@ -680,9 +858,9 @@ export async function discoverViaLMStudio(
// Filter out non-chat models (embeddings, etc) and rank: loaded first,
// then by the standard small-name heuristic among each tier.
- const chatModels = models.filter(
- (m) => isChatCapable(m.id) && m.type !== "embeddings" && m.type !== "embedding"
- );
+ // LM Studio publishes `type` per model, so its own word decides here too and
+ // the separate `type !== "embeddings"` test folds into the shared rule.
+ const chatModels = models.filter((m) => isReportedChatCapable(m.id, lmStudioReported(m)));
const loaded = chatModels.filter((m) => m.state === "loaded");
const notLoaded = chatModels.filter((m) => m.state !== "loaded");
@@ -734,13 +912,19 @@ async function fetchOllamaModels(url: string): Promise<OllamaModel[]> {
});
if (!response.ok) return [];
const body = (await response.json().catch(() => null)) as {
- models?: Array<{ name?: unknown; size?: unknown }>;
+ models?: Array<{ name?: unknown; size?: unknown; capabilities?: unknown }>;
} | null;
if (!body?.models) return [];
return body.models
.map((m) => ({
name: typeof m.name === "string" ? m.name : "",
size: typeof m.size === "number" ? m.size : undefined,
+ // Carried through, not projected away. This mapping previously kept only
+ // name and size, so the daemon's own `capabilities` array was discarded
+ // here and every caller downstream had nothing but the name to go on.
+ capabilities: Array.isArray(m.capabilities)
+ ? m.capabilities.filter((c): c is string => typeof c === "string")
+ : undefined,
}))
.filter((m) => m.name.length > 0);
}