The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Add an OpenAI-compatible API provider for Metabot (#83783)
Andrei Simionescue8fb520c3ebe
diff --git a/docs/ai/providers.md b/docs/ai/providers.mdindex 683848110b5..6a26d439576 100644--- a/docs/ai/providers.md+++ b/docs/ai/providers.md@@ -386,6 +386,45 @@ Credentials:Ollama can load a model with a smaller context window than the model supports. Metabot needs at least 16,384 tokens, and Metabase checks the window when you connect. See Ollama's documentation on [context length](https://docs.ollama.com/context-length).+## OpenAI-compatible API++- Provider key: `openai-compatible`++Supported models:++Whichever model your server serves. Metabase runs the one you enter in **Model ID**.++Credentials:++- **API base URL** (required). Any server that implements OpenAI's Chat Completions API with tool calling. Enter the URL that comes before /chat/completions. You can also set it with the environment variable `MB_LLM_OPENAI_COMPATIBLE_API_BASE_URL`.+- **API key**. Only needed if your server asks for one. You can also set it with the environment variable `MB_LLM_OPENAI_COMPATIBLE_API_KEY`.+- **Model ID** (required). The model to run, exactly as your server names it.++### What connecting checks++When you connect, Metabase sends the model a few short test requests. The model has to call a tool, call a tool when the request requires one (`tool_choice` set to `required`), and return its reasoning separately from its answer. If the server's `/models` list gives the model's context window, as `max_model_len` or `context_length`, it has to be at least 16,384 tokens. If a check fails, Metabase doesn't save the connection and tells you which check failed.++The checks only test what Metabot needs to run. Metabase doesn't benchmark these models, so how well Metabot works depends on the model you choose. Models that need their reasoning sent back with every request can fail partway through a conversation, because Metabase doesn't send it back to them.++### Servers this connection has been tried with++- xAI, `https://api.x.ai/v1`, with `grok-4.3`: connects, and Metabot answers questions on it.+- OpenRouter, `https://openrouter.ai/api/v1`, with `anthropic/claude-haiku-4.5`: passed the checks when tried. OpenRouter sends each request to one of the providers that serve a model, so a model can pass the checks on one try and fail them on the next, as `openai/gpt-oss-120b` did.+- DeepSeek, `https://api.deepseek.com/v1`, with `deepseek-v4-pro`: doesn't connect. In thinking mode, DeepSeek rejects requests that require a tool call, so the forced tool call check fails.++### API base URL examples++Enter the URL that comes before `/chat/completions`. These examples show the format, and weren't tried:++- Scaleway: `https://api.scaleway.ai/v1`+- A vLLM, llama.cpp, or LM Studio server: `http://<host>:<port>/v1`++On a self-hosted Metabase, a server on your private network or on the same machine as Metabase needs [`MB_LLM_ALLOWED_NETWORKS`](../configuring-metabase/environment-variables.md#mb_llm_allowed_networks).++### Set the model with environment variables++If you configure this connection with environment variables, it has no **Model ID**. Set `MB_LLM_METABOT_PROVIDER` to `openai-compatible/` followed by the model ID instead, like `openai-compatible/gpt-oss-120b`.+## Metabase AI service- Provider key: `metabase`diff --git a/e2e/test/scenarios/onboarding/setup/setup.cy.spec.ts b/e2e/test/scenarios/onboarding/setup/setup.cy.spec.tsindex 017cc2b09b9..85b3930752a 100644--- a/e2e/test/scenarios/onboarding/setup/setup.cy.spec.ts+++ b/e2e/test/scenarios/onboarding/setup/setup.cy.spec.ts@@ -502,13 +502,15 @@ describe("scenarios > setup > AI config step", () => {navigateToAiConfigStep();cy.findByLabelText("Connect to an AI provider").within(() => {- [/OpenAI/, /OpenRouter/, /Microsoft Azure/, /Amazon Bedrock/].forEach(- (name) => {- cy.findByRole("button", { name })- .scrollIntoView()- .should("be.visible");- },- );+ [+ "OpenAI",+ "OpenAI-compatible API",+ /OpenRouter/,+ /Microsoft Azure/,+ /Amazon Bedrock/,+ ].forEach((name) => {+ cy.findByRole("button", { name }).scrollIntoView().should("be.visible");+ });// the managed provider is offered but not connectable without the LLM proxy,// which e2e does not configurecy.findByRole("button", { name: /Metabase/ }).should("be.disabled");diff --git a/frontend/src/metabase-types/api/llm.ts b/frontend/src/metabase-types/api/llm.tsindex c60e9eaab9b..6a0659d6e1d 100644--- a/frontend/src/metabase-types/api/llm.ts+++ b/frontend/src/metabase-types/api/llm.ts@@ -47,6 +47,7 @@ export type LlmProviderTypeName =| "bedrock"| "vllm"| "ollama"+ | "openai-compatible"| "metabase";export type LlmProviderFieldType =diff --git a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.tsx b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.tsxindex ddc084fe760..9a8e533dfc1 100644--- a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.tsx+++ b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.tsx@@ -22,8 +22,8 @@ const GENERIC_PROVIDER_ICON = "ai";// Every mark is inlined rather than loaded by URL: it lets the single-colour ones take the theme's text// colour the way their brands publish them, and it keeps them out of the emitted-asset pipeline. Metabase-// is null because it is drawn from its own logo component, and vLLM because it falls back to the generic-// AI icon.+// is null because it is drawn from its own logo component, and vLLM and OpenAI-compatible servers because+// they fall back to the generic AI icon.const PROVIDER_LOGOS: Record<LlmProviderTypeName,ComponentType<SVGProps<SVGSVGElement>> | null@@ -41,6 +41,7 @@ const PROVIDER_LOGOS: Record<azure: AzureMark,vllm: null,ollama: OllamaMark,+ "openai-compatible": null,metabase: null,};diff --git a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.unit.spec.tsx b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.unit.spec.tsxindex 529f825f643..2d08e1b5b82 100644--- a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.unit.spec.tsx+++ b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.unit.spec.tsx@@ -42,12 +42,15 @@ describe("ProviderTypeIcon", () => {expect(screen.getByRole("presentation")).toBeInTheDocument();});- it("falls back to a generic icon for vLLM, which is a server rather than a vendor", () => {- renderWithProviders(<ProviderTypeIcon type="vllm" />);+ it.each(["vllm", "openai-compatible"] as const)(+ "falls back to a generic icon for %s, which names no vendor",+ (type) => {+ renderWithProviders(<ProviderTypeIcon type={type} />);- expect(screen.queryByTestId("main-logo")).not.toBeInTheDocument();- expect(screen.queryByRole("presentation")).not.toBeInTheDocument();- });+ expect(screen.queryByTestId("main-logo")).not.toBeInTheDocument();+ expect(screen.queryByRole("presentation")).not.toBeInTheDocument();+ },+ );it("falls back to a generic icon for a type the frontend does not know yet", () => {// the registry is the backend's, so it can serve a type this union does not list yet;diff --git a/src/metabase/cmd/ai_provider_dox.clj b/src/metabase/cmd/ai_provider_dox.cljindex b19bcedcc1f..554ab221715 100644--- a/src/metabase/cmd/ai_provider_dox.clj+++ b/src/metabase/cmd/ai_provider_dox.clj@@ -21,16 +21,23 @@"Hand-written prose appended to a provider type's section, keyed by type, for what an admin needs to know that theregistry doesn't hold. The managed provider's covers that it's a Metabase Cloud offering, how it's billed, and how itauthenticates. Bedrock's covers the IAM actions the mantle endpoint needs and why the model picker is region-scoped.- Google's covers connecting a Model Garden endpoint, which its fixed catalog cannot list."- {"metabase" "metabase/cmd/resources/ai-provider-metabase.md"- "bedrock" "metabase/cmd/resources/ai-provider-bedrock.md"- "google" "metabase/cmd/resources/ai-provider-google.md"- "ollama" "metabase/cmd/resources/ai-provider-ollama.md"})+ Google's covers connecting a Model Garden endpoint, which its fixed catalog cannot list. The OpenAI-compatible+ type's covers what connecting checks, what it doesn't promise, the servers it was tried with and example base+ URLs."+ {"metabase" "metabase/cmd/resources/ai-provider-metabase.md"+ "bedrock" "metabase/cmd/resources/ai-provider-bedrock.md"+ "google" "metabase/cmd/resources/ai-provider-google.md"+ "ollama" "metabase/cmd/resources/ai-provider-ollama.md"+ "openai-compatible" "metabase/cmd/resources/ai-provider-openai-compatible.md"})(def ^:private dynamic-catalog-types"Provider types that serve whatever models the operator loaded, so there is no list to publish."#{"vllm" "ollama"})+(def ^:private named-model-types+ "Provider types whose connection runs whichever model the admin names, on a server Metabase has no list for."+ #{"openai-compatible"})+(def ^:private max-enumerated-options"Above this many `:options`, a field's choices are pointed at rather than listed. Bedrock's dozens of regions arenoise on a docs page, and are right there in the dropdown."@@ -190,6 +197,10 @@(seq models)(fixed-models-table models)+ (contains? named-model-types type)+ (str "Whichever model your server serves. Metabase runs the one you enter in "+ (field-labels fields model-fields) ".")+(seq model-fields)(str "Whichever model your deployment serves. " provider-label" serves the deployments you create, not a fixed catalog, so there's no list to pick from — Metabase "diff --git a/src/metabase/cmd/resources/ai-provider-openai-compatible.md b/src/metabase/cmd/resources/ai-provider-openai-compatible.mdnew file mode 100644index 00000000000..d13cf5db710--- /dev/null+++ b/src/metabase/cmd/resources/ai-provider-openai-compatible.md@@ -0,0 +1,24 @@+### What connecting checks++When you connect, Metabase sends the model a few short test requests. The model has to call a tool, call a tool when the request requires one (`tool_choice` set to `required`), and return its reasoning separately from its answer. If the server's `/models` list gives the model's context window, as `max_model_len` or `context_length`, it has to be at least 16,384 tokens. If a check fails, Metabase doesn't save the connection and tells you which check failed.++The checks only test what Metabot needs to run. Metabase doesn't benchmark these models, so how well Metabot works depends on the model you choose. Models that need their reasoning sent back with every request can fail partway through a conversation, because Metabase doesn't send it back to them.++### Servers this connection has been tried with++- xAI, `https://api.x.ai/v1`, with `grok-4.3`: connects, and Metabot answers questions on it.+- OpenRouter, `https://openrouter.ai/api/v1`, with `anthropic/claude-haiku-4.5`: passed the checks when tried. OpenRouter sends each request to one of the providers that serve a model, so a model can pass the checks on one try and fail them on the next, as `openai/gpt-oss-120b` did.+- DeepSeek, `https://api.deepseek.com/v1`, with `deepseek-v4-pro`: doesn't connect. In thinking mode, DeepSeek rejects requests that require a tool call, so the forced tool call check fails.++### API base URL examples++Enter the URL that comes before `/chat/completions`. These examples show the format, and weren't tried:++- Scaleway: `https://api.scaleway.ai/v1`+- A vLLM, llama.cpp, or LM Studio server: `http://<host>:<port>/v1`++On a self-hosted Metabase, a server on your private network or on the same machine as Metabase needs [`MB_LLM_ALLOWED_NETWORKS`](../configuring-metabase/environment-variables.md#mb_llm_allowed_networks).++### Set the model with environment variables++If you configure this connection with environment variables, it has no **Model ID**. Set `MB_LLM_METABOT_PROVIDER` to `openai-compatible/` followed by the model ID instead, like `openai-compatible/gpt-oss-120b`.diff --git a/src/metabase/llm/provider.clj b/src/metabase/llm/provider.cljindex a547d692565..3287fed6bdb 100644--- a/src/metabase/llm/provider.clj+++ b/src/metabase/llm/provider.clj@@ -73,6 +73,7 @@:required? true:placeholder "sk-proj-...":prefix "sk-"+ :prefix-help (deferred-tru "For another server that implements OpenAI''s API, add an OpenAI-compatible API provider instead."):docs-url "https://platform.openai.com/api-keys"}{:key :base-url:normalize strip-trailing-slashes@@ -403,6 +404,30 @@;; not required: a self-hosted server takes none, and a base URL on its own is a complete;; configuration. Ollama Cloud refuses a generation without one, which connecting finds out.:help (deferred-tru "Required for Ollama Cloud. Leave blank if your server doesn''t require one.")}]}+ {:type "openai-compatible"+ :label (deferred-tru "OpenAI-compatible API")+ ;; served by vLLM's adapter, whose connect check adds :model-reasoning to the stored config+ :stored-config-fields [:model-reasoning]+ :default-model nil+ :model-fields [:model-id]+ :fields [{:key :base-url+ :normalize strip-trailing-slashes+ :validate llm.provider.settings/llm-url-problem+ :label (deferred-tru "API base URL")+ :type :text+ :required? true+ :placeholder "https://api.example.com/v1"+ :help (deferred-tru "Any server that implements OpenAI''s Chat Completions API with tool calling. Enter the URL that comes before /chat/completions.")}+ {:key :api-key+ :label (deferred-tru "API key")+ :type :password+ :help (deferred-tru "Only needed if your server asks for one.")}+ {:key :model-id+ :label (deferred-tru "Model ID")+ :type :text+ :required? true+ :placeholder "gpt-oss-120b"+ :help (deferred-tru "The model to run, exactly as your server names it.")}]}{:type "metabase":label (deferred-tru "Metabase AI service"):managed? true@@ -562,13 +587,14 @@(:fields (provider-type type-name))))(defn- validate-field!- [type-name {:keys [key label required? prefix default options] :as field} config]+ [type-name {:keys [key label required? prefix prefix-help default options] :as field} config](let [value (u/trimmed-string (get config key))](when (and required? (not value) (not default))(throw (ex-info (tru "{0} is required for {1}." (str label) type-name){:status-code 400 :field key})))(when (and value prefix (not (str/starts-with? value prefix)))- (throw (ex-info (tru "Invalid {0} for {1}. It must start with ''{2}''." (str label) type-name prefix)+ (throw (ex-info (cond-> (tru "Invalid {0} for {1}. It must start with ''{2}''." (str label) type-name prefix)+ prefix-help (str " " prefix-help)){:status-code 400 :field key})))(when (and value (seq options) (not-any? #(= value (:value %)) options))(throw (ex-info (tru "Invalid {0} for {1}." (str label) type-name)@@ -747,7 +773,11 @@;; as for vLLM, the base URL is the credential: a self-hosted server takes no key, so the URL alone;; brings a usable connection into existence. A key alone, with no address to send it to, does not.:settings {:base-url {:setting :llm-ollama-api-base-url :credential? true}- :api-key {:setting :llm-ollama-api-key}}}})+ :api-key {:setting :llm-ollama-api-key}}}+ "openai-compatible" {:type "openai-compatible"+ ;; the base URL is the credential, as for vLLM; the model comes from MB_LLM_METABOT_PROVIDER+ :settings {:base-url {:setting :llm-openai-compatible-api-base-url :credential? true}+ :api-key {:setting :llm-openai-compatible-api-key}}}})(defn connection-env-vars"The environment variables that configure a connection of `type-name`, as `{config-field \"MB_LLM_...\"}`.diff --git a/src/metabase/llm/settings.clj b/src/metabase/llm/settings.cljindex b03f984b34a..6e95191187a 100644--- a/src/metabase/llm/settings.clj+++ b/src/metabase/llm/settings.clj@@ -440,6 +440,26 @@:visibility :settings-manager:export? false)+;;; --------------------------------------------- OpenAI-compatible ---------------------------------------------++(defsetting llm-openai-compatible-api-base-url+ (deferred-tru "The base URL of a server that implements OpenAI''s Chat Completions API, e.g. `https://api.scaleway.ai/v1`. Set `MB_LLM_METABOT_PROVIDER` to `openai-compatible/` followed by the model ID to choose its model.")+ :encryption :when-encryption-key-set+ :visibility :settings-manager+ :export? false+ :getter (connection-field-getter :llm-openai-compatible-api-base-url)+ :setter (connection-field-setter :llm-openai-compatible-api-base-url)+ :doc "Backed by the openai-compatible connection in the admin AI settings provider list: reads and writes go through the llm-providers connection list. A value set by this environment variable shadows that connection's base URL.")++(defsetting llm-openai-compatible-api-key+ (deferred-tru "The API key for the server that the OpenAI-compatible connection talks to, if it asks for one.")+ :sensitive? true+ :visibility :settings-manager+ :export? false+ :getter (connection-field-getter :llm-openai-compatible-api-key)+ :setter (connection-field-setter :llm-openai-compatible-api-key)+ :doc "Backed by the openai-compatible connection in the admin AI settings provider list: reads and writes go through the llm-providers connection list, and a value set by this environment variable shadows this one field of that connection.")+;;; The per-provider credential settings above are read-only at runtime: they configure a connection only when set;;; by an environment variable, which [[metabase.llm.provider/connections]] resolves on every read. Editing one in;;; the app DB would not reach the connection serving requests, so a write is rejected rather than silently ignored.diff --git a/src/metabase/metabot/self.clj b/src/metabase/metabot/self.cljindex c9fa26bb30b..d29812df1eb 100644--- a/src/metabase/metabot/self.clj+++ b/src/metabase/metabot/self.clj@@ -45,8 +45,9 @@This is the allow-list [[list-models]] intersects with the provider's live catalog, so a model listed here isavailable only if the connection's credentials can actually reach it. Returns nil for the provider types that have- no allow-list: `azure`, whose model is the deployment name the admin gives it, `vllm` and `ollama`, which serve whatever the- operator loaded, and `google` and `metabase`, whose catalogs are fixed in [[metabase.llm.provider]] instead."+ no allow-list: `azure` and `openai-compatible`, whose model is the one the admin names, `vllm` and `ollama`, which+ serve whatever the operator loaded, and `google` and `metabase`, whose catalogs are fixed in [[metabase.llm.provider]]+ instead."[provider](when-let [models (registry/optional provider :supported-models)](into {}diff --git a/src/metabase/metabot/self/core.clj b/src/metabase/metabot/self/core.cljindex d4753d0e0f2..7ce0ede0b97 100644--- a/src/metabase/metabot/self/core.clj+++ b/src/metabase/metabot/self/core.clj@@ -155,10 +155,12 @@(def ^:private ApiKeyCredentials"The `{:api-key ... :base-url ...}` connection shape shared by most providers. `:model-reasoning` and- `:probed-model` are not admin-entered: a connect-time probe records them on the connection (vLLM)."+ `:probed-model` are not admin-entered: a connect-time probe records them on the connection (vLLM). An+ OpenAI-compatible connection names its model in `:model-id`."[:map {:closed true}[:api-key {:optional true} [:maybe :string]][:base-url {:optional true} [:maybe :string]]+ [:model-id {:optional true} [:maybe :string]][:model-reasoning {:optional true} [:maybe [:or :boolean :string]]][:probed-model {:optional true} [:maybe :string]][:mini-model {:optional true} [:maybe :string]]])diff --git a/src/metabase/metabot/self/google.clj b/src/metabase/metabot/self/google.cljindex e3784c4d26e..901197451d6 100644--- a/src/metabase/metabot/self/google.clj+++ b/src/metabase/metabot/self/google.clj@@ -37,6 +37,7 @@[metabase.metabot.self.google.models :as models][metabase.metabot.self.google.raw-predict :as raw-predict][metabase.metabot.self.google.stream-generate-content :as stream-generate-content]+ [metabase.metabot.self.openai.chat-completions :as chat-completions][metabase.metabot.self.vllm :as vllm][metabase.util :as u][metabase.util.i18n :refer [tru]]@@ -694,8 +695,7 @@"Call the Gemini Enterprise Agent Platform, return AISDK stream."[& args](let [[family raw] (apply google-raw args)]- (eduction (case family- :anthropic (raw-predict/->aisdk-chunks-xf)- :google (stream-generate-content/->aisdk-chunks-xf)- :chat-completions (vllm/vllm->aisdk-chunks-xf))- raw)))+ (case family+ :anthropic (eduction (raw-predict/->aisdk-chunks-xf) raw)+ :google (eduction (stream-generate-content/->aisdk-chunks-xf) raw)+ :chat-completions (eduction (vllm/vllm->aisdk-chunks-xf) (chat-completions/usage-once raw)))))diff --git a/src/metabase/metabot/self/mistral.clj b/src/metabase/metabot/self/mistral.cljindex 90c25426a72..e19f9ca2729 100644--- a/src/metabase/metabot/self/mistral.clj+++ b/src/metabase/metabot/self/mistral.clj@@ -194,4 +194,4 @@"Call the Mistral Chat Completions API, return AISDK stream."[& args](let [raw (apply mistral-raw args)]- (eduction (mistral->aisdk-chunks-xf) raw)))+ (eduction (mistral->aisdk-chunks-xf) (chat-completions/usage-once raw))))diff --git a/src/metabase/metabot/self/moonshot.clj b/src/metabase/metabot/self/moonshot.cljindex 904f5c53f71..01d23403aa1 100644--- a/src/metabase/metabot/self/moonshot.clj+++ b/src/metabase/metabot/self/moonshot.clj@@ -198,4 +198,4 @@"Call the Moonshot Chat Completions API, return AISDK stream."[& args](let [raw (apply moonshot-raw args)]- (eduction (moonshot->aisdk-chunks-xf) raw)))+ (eduction (moonshot->aisdk-chunks-xf) (chat-completions/usage-once raw))))diff --git a/src/metabase/metabot/self/openai/chat_completions.clj b/src/metabase/metabot/self/openai/chat_completions.cljindex 33524d3ce24..bcc660c0202 100644--- a/src/metabase/metabot/self/openai/chat_completions.clj+++ b/src/metabase/metabot/self/openai/chat_completions.clj@@ -37,6 +37,43 @@:cacheCreationTokens (or (:cache_write_tokens details) 0):cacheReadTokens (or (:cached_tokens details) 0)}))+(defn usage-once+ "Move the server's usage in `chunks`, a reducible of Chat Completions chunks, to one final chunk.++ The final `{:usage ...}` chunk carries the last usage the server sent. Some servers repeat their running totals on+ every chunk, and translating each would count the call many times over. A stream that fails partway, or that the+ consumer stops, still passes on the usage it received."+ [chunks]+ (reify clojure.lang.IReduceInit+ (reduce [_ rf init]+ (let [usage (volatile! nil)+ latest (volatile! init)+ flush (fn [acc]+ (cond-> (unreduced acc)+ @usage (rf {:usage @usage})))+ acc (try+ (reduce (fn [acc chunk]+ (when-let [u (:usage chunk)]+ (vreset! usage u))+ (vreset! latest (rf acc (dissoc chunk :usage))))+ init+ chunks)+ (catch Throwable t+ (flush @latest)+ (throw t)))]+ (unreduced (flush acc))))))++(defn count-reasoning-as-output+ "Raise a chunk's `completion_tokens` to everything its `total_tokens` holds beyond the prompt.++ Some servers, xAI among them, report reasoning tokens next to `completion_tokens` rather than inside them, and bill+ them as output: https://docs.x.ai/developers/advanced-api-usage/prompt-caching/usage-and-pricing. A chunk whose+ total holds nothing more, or that has no total, keeps its `completion_tokens`."+ [{:keys [usage] :as chunk}]+ (if-let [total (:total_tokens usage)]+ (update-in chunk [:usage :completion_tokens] (fnil max 0) (- total (:prompt_tokens usage 0)))+ chunk))+;;; AISDK parts → Chat Completions messages(defn- merge-consecutive-assistant-messagesdiff --git a/src/metabase/metabot/self/openai_compatible.clj b/src/metabase/metabot/self/openai_compatible.cljnew file mode 100644index 00000000000..18809c1a90f--- /dev/null+++ b/src/metabase/metabot/self/openai_compatible.clj@@ -0,0 +1,94 @@+(ns metabase.metabot.self.openai-compatible+ "Adapter for any server that implements OpenAI's Chat Completions API with tool calling.++ It is vLLM's adapter under its own name and messages: the same request body less its `temperature`, stream+ translation, timeouts and connect-time checks (see [[vllm/server]]). The connection names its model, so connecting+ checks that model whether or not the server lists it, and reads the server's `/models` only for a context window."+ (:require+ [metabase.metabot.self.adapter :as adapter]+ [metabase.metabot.self.openai.chat-completions :as chat-completions]+ [metabase.metabot.self.vllm :as vllm]+ [metabase.util :as u]+ [metabase.util.i18n :refer [tru]]+ [metabase.util.malli :as mu]))++(set! *warn-on-reflection* true)++(def ^:private server+ (vllm/server+ {:slug "openai-compatible"+ :display-name "OpenAI-compatible"+ :errors {401 #(tru "The server did not accept the API key")+ 404 #(tru "The server has no Chat Completions endpoint at this base URL, or no model with this ID")}}+ {:base-url-missing #(tru "No base URL is set for this OpenAI-compatible connection")+ :model-missing #(tru "No model ID is set for this OpenAI-compatible connection")+ :unreachable #(tru "Could not reach the server at {0}. Check that it is running and that the base URL is correct." %)+ :context-too-small #(tru "{0} has a {1} token context window, which is too small for Metabot. It needs at least {2}." %1 %2 %3)+ :reasoning-as-text #(tru "{0} writes its reasoning into its answers, where it would show up in Metabot''s replies. Use a model or server setting that returns reasoning separately." %)+ :invalid-tool-arguments #(tru "{0} returned a tool call whose arguments are not valid JSON. Check that the server supports tool calling for this model." %)+ :answered-with-text #(tru "{0} answered with text instead of calling a tool. Check that the server supports tool calling for this model." %)+ :forced-call-ignored #(tru "The server answered with text when tool_choice required a tool call. Metabot needs forced tool calls for conversation titles and SQL generation. If this is an Ollama server, add an Ollama provider instead.")+ :connection-test-timeout #(tru "The server did not answer the connection test within {0}ms. A server this slow to answer a short prompt can''t run Metabot." %)+ :request-timeout #(tru "The server did not respond within {0}ms. Check that it is not overloaded." %)+ :stopped-responding #(tru "The server stopped responding after {0}ms. Try again, or use a faster model." %)+ :interrupted #(tru "The connection to the server was interrupted before the response finished.")}+ ;; a server that fixes the temperature per model rejects any other value: Moonshot takes only 1, and Claude 4.7+ ;; and later reject anything below 1 through Anthropic's compatibility layer+ {:temperature? false}))++(def ^:private provider (:provider server))++(defn- catalog-entry+ "The server's `/models` entry for `model`, reduced to what [[vllm/preflight!]] reads.++ The context window goes under vLLM's `max_model_len`, whether the server publishes it under that name or as+ `context_length`, as OpenRouter does. A server with no model list, or one that omits the model, publishes no+ window, and the window goes unchecked."+ [req model]+ (let [entry (try+ (->> (adapter/request! provider (assoc req :method :get :path "/models" :as :json))+ :body+ :data+ (u/seek #(= model (:id %))))+ (catch Exception _+ nil))]+ {:id model+ :max_model_len (or (:max_model_len entry) (:context_length entry))}))++(mu/defn list-models :- adapter/ModelListing+ "List the one model the connection names.++ `:probe?` also runs [[vllm/preflight!]] against it and reports whether it reasons as `:connection-info`, for the+ connect path to store on the connection."+ ([] (list-models {}))+ ([{:keys [credentials ai-proxy? model probe?]} :- adapter/ListOpts]+ (adapter/reject-ai-proxy! provider ai-proxy?)+ (let [req {:credentials credentials :ai-proxy? ai-proxy?}]+ (cond-> {:models (if model [{:id model :display_name model}] [])}+ probe? (assoc :connection-info+ {vllm/reasoning-config-key+ (str (:reasoning? (vllm/preflight! server req (catalog-entry req model))))})))))++(mu/defn streams-reasoning? :- :boolean+ "Registry capability: whether the connection's model streams its reasoning.++ A connection saved through the providers API recorded the answer when it connected (see+ [[vllm/reasoning-config-key]]). One configured by environment variables never ran that check, so it counts as+ reasoning, and the chat shows whatever reasoning arrives."+ [{:keys [credentials]} :- adapter/ResolvedRef]+ (or (vllm/reasoning-connection? credentials)+ (not (contains? credentials vllm/reasoning-config-key))))++(defn openai-compatible->aisdk-chunks-xf+ "Translate an OpenAI-compatible server's Chat Completions chunks into AI SDK v5 protocol chunks.++ The same translation as vLLM's, with reasoning tokens that a server reports next to `completion_tokens` counted+ as output."+ []+ (comp (map chat-completions/count-reasoning-as-output)+ (vllm/vllm->aisdk-chunks-xf)))++(defn openai-compatible+ "Call an OpenAI-compatible server's Chat Completions API, return AISDK stream."+ [opts]+ (eduction (openai-compatible->aisdk-chunks-xf) (chat-completions/usage-once (vllm/server-raw server opts))))diff --git a/src/metabase/metabot/self/openrouter.clj b/src/metabase/metabot/self/openrouter.cljindex 83728f9d7f9..d4beaed9000 100644--- a/src/metabase/metabot/self/openrouter.clj+++ b/src/metabase/metabot/self/openrouter.clj@@ -314,4 +314,4 @@"Call OpenRouter Chat Completions API, return AISDK stream."[& args](let [raw (apply openrouter-raw args)]- (eduction (openrouter->aisdk-chunks-xf) raw)))+ (eduction (openrouter->aisdk-chunks-xf) (chat-completions/usage-once raw))))diff --git a/src/metabase/metabot/self/registry.clj b/src/metabase/metabot/self/registry.cljindex f277f4b239a..1995250572f 100644--- a/src/metabase/metabot/self/registry.clj+++ b/src/metabase/metabot/self/registry.clj@@ -16,6 +16,7 @@[metabase.metabot.self.moonshot :as moonshot][metabase.metabot.self.ollama :as ollama][metabase.metabot.self.openai :as openai]+ [metabase.metabot.self.openai-compatible :as openai-compatible][metabase.metabot.self.openrouter :as openrouter][metabase.metabot.self.vllm :as vllm][metabase.metabot.self.xai :as xai]@@ -118,6 +119,9 @@:supported-models #'openai/supported-models:context-window #'openai/context-window-tokens:reasoning? #'openai/streams-reasoning?}+ "openai-compatible" {:stream #'openai-compatible/openai-compatible+ :list-models #'openai-compatible/list-models+ :reasoning? #'openai-compatible/streams-reasoning?}"openrouter" {:stream #'openrouter/openrouter:list-models #'openrouter/list-models:supported-models #'openrouter/supported-modelsdiff --git a/src/metabase/metabot/self/vllm.clj b/src/metabase/metabot/self/vllm.cljindex 7c5ba338d28..163f2141510 100644--- a/src/metabase/metabot/self/vllm.clj+++ b/src/metabase/metabot/self/vllm.clj@@ -26,41 +26,85 @@(set! *warn-on-reflection* true)-(defn- missing-base-url-ex []- (ex-info (tru "No vLLM base URL is set")- {:api-error true- :error-code :base-url-missing}))--(defn- vllm-auth- "vLLM's `:auth`. A server needs a base URL but not a key — one started without `--api-key` is a complete- configuration — so the map is never nil and [[core/resolve-auth]] cannot reach its `missing-api-key-ex`- branch."- [{:keys [slug display-name]} {:keys [credentials ai-proxy?]}]- (let [base-url (not-empty (:base-url credentials))- api-key (not-empty (:api-key credentials))]- (when-not base-url- (throw (missing-base-url-ex)))- (core/resolve-auth slug display-name- (cond-> {:url base-url}- api-key (assoc :headers {"Authorization" (str "Bearer " api-key)}))- ai-proxy?)))--(def ^:private provider- (adapter/provider- {:slug "vllm"- :display-name "vLLM"- :auth vllm-auth- :error-fallback #(tru "vLLM API error (HTTP {0})" %)- :errors {400 #(tru "vLLM rejected the request — usually an unsupported schema, or a model that cannot compile the tool grammar")- 401 #(tru "vLLM API key expired or invalid — check the key your server was started with via --api-key")- 404 #(tru "vLLM API endpoint was not found — the base URL should end in /v1")- 429 #(tru "The vLLM server''s request queue is full — reduce concurrent load, or restart it with a larger --max-num-seqs")- 500 #(tru "vLLM returned an internal server error")}}))--(defn- missing-model-ex []- (ex-info (tru "No vLLM model is set")- {:api-error true- :error-code :model-missing}))+(defn- base-url-auth+ "The `:auth` of a server that needs a base URL but not an API key.++ One started without `--api-key` is a complete configuration, so the map is never nil and [[core/resolve-auth]]+ cannot reach its `missing-api-key-ex` branch. `missing-base-url-msg` returns the message a connection without a+ base URL fails with."+ [missing-base-url-msg]+ (fn [{:keys [slug display-name]} {:keys [credentials ai-proxy?]}]+ (let [base-url (not-empty (:base-url credentials))+ api-key (not-empty (:api-key credentials))]+ (when-not base-url+ (throw (ex-info (missing-base-url-msg)+ {:api-error true+ :error-code :base-url-missing})))+ (core/resolve-auth slug display-name+ (cond-> {:url base-url}+ api-key (assoc :headers {"Authorization" (str "Bearer " api-key)}))+ ai-proxy?))))++(def ^:private Copy+ "What a server's errors say, each message a fn of the values it names."+ [:map {:closed true}+ [:base-url-missing [:=> :cat :string]]+ [:model-missing [:=> :cat :string]]+ [:unreachable [:=> [:cat :string] :string]]+ [:context-too-small [:=> [:cat :string :string :string] :string]]+ [:reasoning-as-text [:=> [:cat :string] :string]]+ [:invalid-tool-arguments [:=> [:cat :string] :string]]+ [:answered-with-text [:=> [:cat :string] :string]]+ [:forced-call-ignored [:=> :cat :string]]+ [:connection-test-timeout [:=> [:cat :string] :string]]+ [:request-timeout [:=> [:cat :string] :string]]+ [:stopped-responding [:=> [:cat :string] :string]]+ [:interrupted [:=> :cat :string]]])++(def ^:private Server+ [:map {:closed true}+ [:provider adapter/Provider]+ [:copy Copy]+ [:temperature? :boolean]])++(mu/defn server :- Server+ "Describe a kind of server the code below talks to.++ `spec` builds its adapter descriptor, with an `:auth` that needs a base URL but not a key, and `copy` holds the+ messages its errors use. `:temperature?` says whether its requests carry a `temperature` at all. vLLM is one kind;+ [[metabase.metabot.self.openai-compatible]] is another."+ [spec :- adapter/ProviderSpec+ copy :- Copy+ {:keys [temperature?]} :- [:map {:closed true} [:temperature? :boolean]]]+ {:provider (adapter/provider (assoc spec :auth (base-url-auth (:base-url-missing copy))))+ :copy copy+ :temperature? temperature?})++(def ^:private vllm-server+ (server+ {:slug "vllm"+ :display-name "vLLM"+ :error-fallback #(tru "vLLM API error (HTTP {0})" %)+ :errors {400 #(tru "vLLM rejected the request — usually an unsupported schema, or a model that cannot compile the tool grammar")+ 401 #(tru "vLLM API key expired or invalid — check the key your server was started with via --api-key")+ 404 #(tru "vLLM API endpoint was not found — the base URL should end in /v1")+ 429 #(tru "The vLLM server''s request queue is full — reduce concurrent load, or restart it with a larger --max-num-seqs")+ 500 #(tru "vLLM returned an internal server error")}}+ {:base-url-missing #(tru "No vLLM base URL is set")+ :model-missing #(tru "No vLLM model is set")+ :unreachable #(tru "Could not reach the vLLM server at {0}. Check that it is running and that the base URL is correct." %)+ :context-too-small #(tru "{0} is served with a {1} token context window, which is too small for Metabot — it needs at least {2}. Restart vLLM with a larger --max-model-len." %1 %2 %3)+ :reasoning-as-text #(tru "{0} streamed its reasoning as chat text. Restart vLLM with --reasoning-parser so thinking doesn''t appear inside Metabot''s answers." %)+ :invalid-tool-arguments #(tru "The vLLM server returned a tool call whose arguments are not valid JSON. The --tool-call-parser most likely does not match {0}''s output format." %)+ :answered-with-text #(tru "The vLLM server answered with text instead of calling a tool. Restart it with --enable-auto-tool-choice and a --tool-call-parser matching {0}''s output format." %)+ :forced-call-ignored #(tru "The vLLM server did not honor a forced tool call. Metabot needs structured output support for conversation titles and SQL generation.")+ :connection-test-timeout #(tru "The vLLM server did not answer the connection test within {0}ms. Check that it is not overloaded — a server this slow to answer a trivial prompt cannot drive Metabot." %)+ :request-timeout #(tru "The vLLM server did not respond within {0}ms. Check that it is not overloaded, or raise the vLLM request timeout." %)+ :stopped-responding #(tru "The vLLM server stopped responding after {0}ms. Raise the vLLM request timeout, or serve a faster model." %)+ :interrupted #(tru "The connection to the vLLM server was interrupted before the response finished.")}+ {:temperature? true}))++(def ^:private provider (:provider vllm-server))(def reasoning-config-key"The connection `:config` key [[preflight!]]'s reasoning observation is recorded under. It is not an@@ -97,10 +141,9 @@;;; ----------------------------------------------- Transport errors ---------------------------------------------(defn- unreachable-ex- "The vLLM error for a non-timeout transport failure. `extra` is the caller's own ex-data tags."- [^IOException e base-url extra]- (ex-info (tru "Could not reach the vLLM server at {0}. Check that it is running and that the base URL is correct."- (str base-url))+ "The error for a non-timeout transport failure. `extra` is the caller's own ex-data tags."+ [copy ^IOException e base-url extra]+ (ex-info ((:unreachable copy) (str base-url))(merge {:api-error true :error-code :vllm-unreachable} extra)e))@@ -116,7 +159,7 @@:status-code 400:error-code :vllm-timeout}e)- (unreachable-ex e base-url {:status-code 400})))+ (unreachable-ex (:copy vllm-server) e base-url {:status-code 400})));;; ------------------------------------------------ Model listing -----------------------------------------------@@ -180,6 +223,12 @@catalog is cheap to serve, and the chat request waits for this lookup."5000)+(def ^:private forced-tool-call-messages+ "The prompt of the forced tool call check.++ Nothing in it calls for a tool, so a server that ignores `tool_choice` answers it with text."+ [{:role "user" :content "Say hello."}])+(defn- preflight-ex"A preflight failure, tagged so `metabase.metabot.api` surfaces the message verbatim, not as a 500."[msg]@@ -191,27 +240,26 @@"Run one non-streaming Chat Completions turn against `model` and return the first choice. The`finish_reason` is part of the return value because a generation truncated at[[adapter/probe-max-tokens]] and a server that will not call tools both produce empty `tool_calls`."- [req model tool-choice]+ [{:keys [provider temperature?]} req model tool-choice messages](let [res (adapter/request! provider(assoc req:method :post:path "/chat/completions":as :json- :body (json/encode {:model model- :messages adapter/probe-messages- :tools [adapter/probe-tool]- :tool_choice tool-choice- :temperature 0- :max_tokens adapter/probe-max-tokens}))+ :body (json/encode (cond-> {:model model+ :messages messages+ :tools [adapter/probe-tool]+ :tool_choice tool-choice+ :max_tokens adapter/probe-max-tokens}+ temperature? (assoc :temperature 0))))(probe-timeouts))](get-in res [:body :choices 0])))(defn- check-context-budget!- [{:keys [id max_model_len]}]+ [copy {:keys [id max_model_len]}](when (and max_model_len (< (long max_model_len) adapter/min-context-window-tokens))(throw (preflight-ex- (tru "{0} is served with a {1} token context window, which is too small for Metabot — it needs at least {2}. Restart vLLM with a larger --max-model-len."- (str id) (str max_model_len) (str adapter/min-context-window-tokens))))))+ ((:context-too-small copy) (str id) (str max_model_len) (str adapter/min-context-window-tokens))))))(defn- check-tool-calling!"Check that the server was started with `--enable-auto-tool-choice` and a `--tool-call-parser` whose@@ -219,8 +267,8 @@`content` as prose and Metabot chats without ever acting.Returns whether the model emitted reasoning, the only signal anywhere that it is a reasoning model."- [req model]- (let [{:keys [message finish_reason]} (probe-chat! req model "auto")+ [{:keys [copy] :as server} req model]+ (let [{:keys [message finish_reason]} (probe-chat! server req model "auto" adapter/probe-messages)content (str (:content message));; `reasoning` since vLLM 0.26; `reasoning_content` is the deprecated spelling older builds;; and other OpenAI-compatible servers still use.@@ -229,9 +277,7 @@truncated? (= "length" finish_reason)](cond(str/includes? content "<think>")- (throw (preflight-ex- (tru "{0} streamed its reasoning as chat text. Restart vLLM with --reasoning-parser so thinking doesn''t appear inside Metabot''s answers."- (str model))))+ (throw (preflight-ex ((:reasoning-as-text copy) (str model))))(seq tool-calls)(let [arguments (get-in (first tool-calls) [:function :arguments])]@@ -242,8 +288,7 @@(if truncated?(tru "{0} reached the {1} token connection-test ceiling partway through a tool call. A model that generates this much before calling a tool is too slow to drive Metabot."(str model) (str adapter/probe-max-tokens))- (tru "The vLLM server returned a tool call whose arguments are not valid JSON. The --tool-call-parser most likely does not match {0}''s output format."- (str model))))))+ ((:invalid-tool-arguments copy) (str model))))))(not (str/blank? reasoning)))(and truncated? (not (str/blank? reasoning)))@@ -264,21 +309,19 @@(str model) (str adapter/probe-max-tokens)))):else- (throw (preflight-ex- (tru "The vLLM server answered with text instead of calling a tool. Restart it with --enable-auto-tool-choice and a --tool-call-parser matching {0}''s output format."- (str model)))))))+ (throw (preflight-ex ((:answered-with-text copy) (str model)))))))(defn- check-structured-output!"Check that guided decoding works. A different failure from [[check-tool-calling!]]: a model whosegrammar the server cannot compile chats fine but breaks titling and the whole `sql` profile."- [req model]- (let [{:keys [message finish_reason]} (probe-chat! req model "required")]+ [{:keys [copy] :as server} req model]+ (let [{:keys [message finish_reason]} (probe-chat! server req model "required" forced-tool-call-messages)](when (empty? (:tool_calls message))(throw (preflight-ex(if (= "length" finish_reason)(tru "{0} reached the {1} token connection-test ceiling without producing a forced tool call. Metabot needs structured output support for conversation titles and SQL generation."(str model) (str adapter/probe-max-tokens))- (tru "The vLLM server did not honor a forced tool call. Metabot needs structured output support for conversation titles and SQL generation.")))))))+ ((:forced-call-ignored copy))))))))(defn- no-models-ex [](preflight-ex (tru "The vLLM server is reachable but is not serving any models.")))@@ -324,10 +367,10 @@They run concurrently; deref order fixes the verdict, tool calling being the more actionablediagnosis when a server fails both. The loser is cancelled rather than left generating against theoperator's server long after anyone is listening."- [req model]+ [{:keys [provider copy] :as server} req model](try- (let [tool-calling (future (check-tool-calling! req model))- structured (future (check-structured-output! req model))]+ (let [tool-calling (future (check-tool-calling! server req model))+ structured (future (check-structured-output! server req model))](try(let [reasoning? (await-probe! tool-calling)](await-probe! structured)@@ -336,26 +379,27 @@(future-cancel tool-calling)(future-cancel structured))))(catch SocketTimeoutException _- (throw (preflight-ex- (tru "The vLLM server did not answer the connection test within {0}ms. Check that it is not overloaded — a server this slow to answer a trivial prompt cannot drive Metabot."- (str (:socket-timeout (probe-timeouts)))))))+ (throw (preflight-ex ((:connection-test-timeout copy) (str (:socket-timeout (probe-timeouts)))))))+ (catch IOException e+ (throw (unreachable-ex copy e (get-in req [:credentials :base-url]) {:status-code 400})))(catch Exception e(adapter/rethrow! provider e))))-(defn- preflight!- "Exercise the contract the agent loop depends on, against the model that will actually be used, and- return `{:model id :reasoning? bool}`. The connect path must adopt exactly this model rather than- re-deriving it from the listing, which agrees only while nothing reorders the catalog.+(defn preflight!+ "Exercise the contract the agent loop depends on against the model that `entry` names.++ `entry` is the model's catalog entry, with its `:id` and, when the server publishes one, its `:max_model_len`;+ `server` is the kind of server it runs on. Returns `{:model id :reasoning? bool}`. The connect path must adopt+ exactly this model rather than re-deriving it from the listing, which agrees only while nothing reorders the+ catalog.`:reasoning?` reports whether the probed model streamed reasoning. Only the probe can answer that,and the answer drives which renderer the frontend picks, so the connection records it (see[[reasoning-config-key]])."- [req entries requested-model]- (let [entry (probe-target entries requested-model)- model (:id entry)]- (check-context-budget! entry)- {:model model- :reasoning? (run-probes! req model)}))+ [server req {:keys [id] :as entry}]+ (check-context-budget! (:copy server) entry)+ {:model id+ :reasoning? (run-probes! server req id)})(mu/defn list-models :- adapter/ModelListing"List the models the connection's vLLM server is serving. Pass-through: there is nothing to@@ -374,7 +418,7 @@proposed (when (some #(= proposed-model (:id %)) entries)proposed-model)probed (when probe?- (preflight! req entries (or model proposed)))]+ (preflight! vllm-server req (probe-target entries (or model proposed))))](cond-> {:models (mapv (fn [{:keys [id] :as entry}]{:id id :display_name (or (:name entry) id)})entries)}@@ -414,7 +458,7 @@(https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/serve/engine/protocol.py, `ModelCard`).Nil when the catalog lists no entry with the model's id or no window for it (Ollama, LM Studio, TGI).Throws when the request fails."- [credentials model]+ [provider credentials model](let [res (adapter/request! provider{:credentials credentials :method :get :path "/models" :as :json}{:socket-timeout window-lookup-timeout-ms@@ -429,27 +473,27 @@An answer, with or without a window, lives for [[answered-lookup-ttl-ms]]. A failed lookup gives a nilwindow that lives for [[failed-lookup-ttl-ms]]: an unknown window only means no default cap, and thechat request that follows reports a real failure in its own words."- [credentials model]+ [provider credentials model](try- (window-entry (fetch-max-model-len credentials model) answered-lookup-ttl-ms)+ (window-entry (fetch-max-model-len provider credentials model) answered-lookup-ttl-ms)(catch Exception e- (log/debugf e "Could not read the context window of %s from the vLLM server" model)+ (log/debugf e "Could not read the context window of %s from the %s server" model (:display-name provider))(window-entry nil failed-lookup-ttl-ms))))(defn- served-max-model-len- "The context window the vLLM server serves the request's model with, or nil when it is unknown.+ "The context window `provider`'s server serves the request's model with, or nil when it is unknown.Read from the server rather than stored on the connection, so a restart with a different`--max-model-len` is followed within [[answered-lookup-ttl-ms]]. Lookups are cached per base URL andmodel, nil windows and failures too, so a server costs one lookup per [[answered-lookup-ttl-ms]], or per[[failed-lookup-ttl-ms]] while its lookup fails, not one per request. Two requests that miss at the sametime both look up; the later answer wins."- [{:keys [model credentials ai-proxy?]}]+ [provider {:keys [model credentials ai-proxy?]}](let [base-url (:base-url credentials)k [base-url model]](when-not (or ai-proxy? (str/blank? base-url))(:window (or (live-entry k)- (let [entry (lookup-entry credentials model)]+ (let [entry (lookup-entry provider credentials model)](swap! window-cache assoc k entry)entry))))))@@ -505,7 +549,7 @@falls back to [[adapter/default-temperature]]. Both stay adapter-local rather than moving into the sharedbuilder, which would also change Z.AI, Mistral, and OpenRouter.- Pure: [[vllm-raw]] looks the window up. The 1-arity, which Model Garden endpoints use, has none."+ Pure: [[server-raw]] looks the window up. The 1-arity, which Model Garden endpoints use, has none."([opts :- core/LLMRequestOpts](vllm-request-body opts nil))([{:keys [max-tokens temperature schema tool_choice credentials] :as opts} :- core/LLMRequestOpts@@ -518,64 +562,71 @@cap (assoc :max_tokens cap)))))(defn- stream-io-ex- "The vLLM error for a transport failure while *consuming* a response stream. Tagged+ "The error for a transport failure while *consuming* a response stream. Tagged`:retryable? false`: on a self-hosted server a stalled or severed response means \"too slow\" or\"it died\", not \"transient\", and a retry replays a full cold prefill at up to`llm-vllm-request-timeout-ms` (300s) apiece. The tag is required — `retryable-error?` walks thecause chain and would otherwise match the `IOException` below."- [^IOException e timeout-ms]+ [copy ^IOException e timeout-ms](if (instance? SocketTimeoutException e)- (ex-info (tru "The vLLM server stopped responding after {0}ms. Raise the vLLM request timeout, or serve a faster model."- (str timeout-ms))+ (ex-info ((:stopped-responding copy) (str timeout-ms)){:api-error true:error-code :vllm-timeout:retryable? false}e)- (ex-info (tru "The connection to the vLLM server was interrupted before the response finished.")+ (ex-info ((:interrupted copy)){:api-error true:error-code :vllm-stream-interrupted:retryable? false}e)))(defn- request-io-ex- "The vLLM error for a transport failure while *establishing* a request. `core/rethrow-api-error!`+ "The error for a transport failure while *establishing* a request. `core/rethrow-api-error!`would render these as \"vllm API request failed: Read timed out\", naming neither the server'sslowness nor the setting that governs it.Tagged `:retryable? false` for the same reason as [[stream-io-ex]], and more importantly: nothinghas been emitted yet, so `call-llm`'s own \"nothing emitted\" predicate would not stop a replay."- [^IOException e base-url timeout-ms]+ [copy ^IOException e base-url timeout-ms](if (instance? SocketTimeoutException e)- (ex-info (tru "The vLLM server did not respond within {0}ms. Check that it is not overloaded, or raise the vLLM request timeout."- (str timeout-ms))+ (ex-info ((:request-timeout copy) (str timeout-ms)){:api-error true:error-code :vllm-timeout:retryable? false}e)- (unreachable-ex e base-url {:retryable? false})))+ (unreachable-ex copy e base-url {:retryable? false})))-(mu/defn vllm-raw- "Perform a streaming request to a vLLM server's Chat Completions API.+(mu/defn server-raw+ "Perform a streaming request to the Chat Completions API of the kind of server `server` describes.Opts map takes `:credentials` (`{:base-url ... :api-key ...}`) from the connection serving thisrequest, and throws without a base URL.- `:ai-proxy?` is not supported for vLLM and throws when true.+ `:ai-proxy?` is not supported and throws when true.Reads the served context window first, through [[served-max-model-len]]'s cache, to size `max_tokens`."- [{:keys [model credentials] :as opts} :- core/LLMRequestOpts]+ [{:keys [provider copy temperature?]} :- Server+ {:keys [model credentials] :as opts} :- core/LLMRequestOpts](when (str/blank? model)- (throw (missing-model-ex)))+ (throw (ex-info ((:model-missing copy))+ {:api-error true+ :error-code :model-missing})))(let [timeout-ms (llm/llm-vllm-request-timeout-ms)](adapter/stream! provider opts{:path "/chat/completions"- :body (vllm-request-body opts (served-max-model-len opts))+ :body (cond-> (vllm-request-body opts (served-max-model-len provider opts))+ (not temperature?) (dissoc :temperature)):request-options (inference-timeouts)- :wrap-stream #(adapter/io-guarded % (fn [e] (stream-io-ex e timeout-ms)))+ :wrap-stream #(adapter/io-guarded % (fn [e] (stream-io-ex copy e timeout-ms)));; clj-http raises an `IOException` only when there is no response at all, so the;; IO branch cannot swallow a failure the provider's own messages would have translated.:on-request-error (fn [e](if (instance? IOException e)- (throw (request-io-ex e (:base-url credentials) timeout-ms))+ (throw (request-io-ex copy e (:base-url credentials) timeout-ms))(adapter/rethrow! provider e)))})))+(mu/defn vllm-raw+ "Perform a streaming request to a vLLM server's Chat Completions API. See [[server-raw]]."+ [opts :- core/LLMRequestOpts]+ (server-raw vllm-server opts))+(defn vllm->aisdk-chunks-xf"Translates vLLM Chat Completions streaming chunks into AI SDK v5 protocol chunks.@@ -592,4 +643,4 @@"Call a vLLM server's Chat Completions API, return AISDK stream."[& args](let [raw (apply vllm-raw args)]- (eduction (vllm->aisdk-chunks-xf) raw)))+ (eduction (vllm->aisdk-chunks-xf) (chat-completions/usage-once raw))))diff --git a/src/metabase/metabot/self/xai.clj b/src/metabase/metabot/self/xai.cljindex 7a20c554592..3685dc82620 100644--- a/src/metabase/metabot/self/xai.clj+++ b/src/metabase/metabot/self/xai.clj@@ -94,22 +94,12 @@{:path "/chat/completions":body (xai-request-body opts)})))-(defn- count-reasoning-as-output- "Rewrite a chunk's `completion_tokens` as everything `total_tokens` holds beyond the prompt.-- xAI reports reasoning tokens next to `completion_tokens` rather than inside them, unlike OpenAI, and bills them- as output: https://docs.x.ai/developers/advanced-api-usage/prompt-caching/usage-and-pricing"- [{:keys [usage] :as chunk}]- (if-let [total (:total_tokens usage)]- (assoc-in chunk [:usage :completion_tokens] (- total (:prompt_tokens usage 0)))- chunk))-(defn xai->aisdk-chunks-xf"Translates xAI Chat Completions streaming chunks into AI SDK v5 protocol chunks.Reasoning summaries arrive as `delta.reasoning_content` and are forwarded as reasoning chunks."[]- (comp (map count-reasoning-as-output)+ (comp (map chat-completions/count-reasoning-as-output)(chat-completions/chat-completions->aisdk-chunks-xf chat-completions/stop-reasons{:forward-reasoning? true})))@@ -117,4 +107,4 @@"Call the xAI Chat Completions API, return AISDK stream."[& args](let [raw (apply xai-raw args)]- (eduction (xai->aisdk-chunks-xf) raw)))+ (eduction (xai->aisdk-chunks-xf) (chat-completions/usage-once raw))))diff --git a/src/metabase/metabot/self/zai.clj b/src/metabase/metabot/self/zai.cljindex b866d910ca1..2651ae57e40 100644--- a/src/metabase/metabot/self/zai.clj+++ b/src/metabase/metabot/self/zai.clj@@ -154,4 +154,4 @@"Call the Z.AI Chat Completions API, return AISDK stream."[& args](let [raw (apply zai-raw args)]- (eduction (zai->aisdk-chunks-xf) raw)))+ (eduction (zai->aisdk-chunks-xf) (chat-completions/usage-once raw))))diff --git a/test/metabase/cmd/ai_provider_dox_test.clj b/test/metabase/cmd/ai_provider_dox_test.cljindex 2d2a236659a..cb9618b8afd 100644--- a/test/metabase/cmd/ai_provider_dox_test.clj+++ b/test/metabase/cmd/ai_provider_dox_test.clj@@ -191,7 +191,9 @@(is (str/includes? (#'ai-provider-dox/models-markdown (registry-entry "google"))"| Model "))(is (str/includes? (#'ai-provider-dox/models-markdown (registry-entry "vllm"))- "whichever models your vLLM server has available")))+ "whichever models your vLLM server has available"))+ (is (= "Whichever model your server serves. Metabase runs the one you enter in **Model ID**."+ (#'ai-provider-dox/models-markdown (registry-entry "openai-compatible")))))(testing "Azure explains that the model comes from the deployment instead, naming the fields by their labels";; answer first: the block sits under a "Supported models:" label, so it leads with what you get rather than;; with the catalog it hasn't gotdiff --git a/test/metabase/llm/api/provider_test.clj b/test/metabase/llm/api/provider_test.cljindex 890b21997ad..96b74b2e150 100644--- a/test/metabase/llm/api/provider_test.clj+++ b/test/metabase/llm/api/provider_test.clj@@ -62,10 +62,10 @@(testing "every provider type is listed with the credential fields a connection needs"(let [types (mt/user-http-request :crowberto :get 200 "llm/provider-types")](is (= #{"anthropic" "openai" "openrouter" "mistral" "zai" "moonshot" "deepseek" "xai" "google" "azure"- "bedrock" "vllm" "ollama" "metabase"}+ "bedrock" "vllm" "ollama" "openai-compatible" "metabase"}(set (map :type types))))(is (= ["anthropic" "openai" "openrouter" "mistral" "zai" "moonshot" "deepseek" "xai" "google" "azure"- "bedrock" "vllm" "ollama"]+ "bedrock" "vllm" "ollama" "openai-compatible"](remove #{"metabase"} (map :type types)))"the bring-your-own-key providers keep their registry order")(is (=? {:type "anthropic"@@ -402,6 +402,30 @@:config {:base-url "http://vllm.internal:8000/v1"}}))))(is (= [] (llm.provider/connections)))))))+(deftest create-openai-compatible-connection-test+ (testing "connecting checks the model the connection names, which Metabot then runs on, on a server with no model list"+ (let [checked (atom #{})]+ (mt/with-dynamic-fn-redefs [http/request (fn [{:keys [url body]}]+ (if (re-find #"/models$" (str url))+ (throw (ex-info "clj-http: status 404" {:status 404 :body "Not Found"}))+ (do (swap! checked conj (:model (json/decode+kw (str body))))+ {:status 200+ :body {:choices [{:message {:content ""+ :tool_calls [{:id "call-1"+ :type "function"+ :function {:name "record_table_name"+ :arguments "{\"table_name\": \"orders\"}"}}]}+ :finish_reason "tool_calls"}]}})))]+ (mt/with-temporary-setting-values [llm-providers []]+ (mt/with-temporary-raw-setting-values [llm-metabot-provider nil]+ (mt/user-http-request :crowberto :post 200 "llm/providers"+ {:type "openai-compatible"+ :config {:base-url "https://inference.internal/v1" :model-id "gpt-oss-120b"}})+ (is (= #{"gpt-oss-120b"} @checked))+ (is (= {:base-url "https://inference.internal/v1" :model-id "gpt-oss-120b" :model-reasoning "false"}+ (stored-config "openai-compatible")))+ (is (= "openai-compatible/gpt-oss-120b" (metabot.settings/llm-metabot-provider)))))))))+(deftest create-rejects-a-malformed-model-catalog-test(testing (str "a 2xx whose body is not a model list means the base URL reached something that is not the API. ""Failing closed is only useful if the admin sees why, so it comes back as the adapter's message "diff --git a/test/metabase/llm/provider_test.clj b/test/metabase/llm/provider_test.cljindex 64c9d5ceb4b..a1240defdc0 100644--- a/test/metabase/llm/provider_test.clj+++ b/test/metabase/llm/provider_test.clj@@ -154,7 +154,12 @@(is (thrown-with-msg?clojure.lang.ExceptionInfo #"must start with"(llm.provider/validate-config! "openrouter" {:api-key "sk-or-v2-nope"})))- (is (nil? (llm.provider/validate-config! "openrouter" {:api-key "sk-or-v1-valid"}))))+ (is (nil? (llm.provider/validate-config! "openrouter" {:api-key "sk-or-v1-valid"})))+ (testing "and OpenAI's points a key from another server to the OpenAI-compatible type"+ (is (= (str "Invalid API key for openai. It must start with 'sk-'. "+ "For another server that implements OpenAI's API, add an OpenAI-compatible API provider instead.")+ (try (llm.provider/validate-config! "openai" {:api-key "company-gateway-key"})+ (catch clojure.lang.ExceptionInfo e (ex-message e)))))))(testing "a type with several required fields rejects the first one missing"(is (thrown-with-msg?clojure.lang.ExceptionInfo #"API base URL is required for azure"@@ -480,6 +485,16 @@(is (=? {:connection-key "metabase" :type "anthropic" :ai-proxy? true}(llm.provider/resolve-model-ref "metabase/anthropic/claude-sonnet-4-6")))))))+(deftest openai-compatible-connection-from-the-environment-test+ (testing "an OpenAI-compatible server's base URL synthesizes a usable connection, which runs the model a reference names"+ (mt/with-temporary-setting-values [llm-providers []]+ (mt/with-temp-env-var-value! [mb-llm-openai-compatible-api-base-url "https://inference.internal/v1"]+ (is (true? (llm.provider/connection-usable? "openai-compatible")))+ (is (=? {:type "openai-compatible"+ :model "gpt-oss-120b"+ :credentials {:base-url "https://inference.internal/v1"}}+ (llm.provider/resolve-model-ref "openai-compatible/gpt-oss-120b")))))))+(deftest connections-shadowed-by-the-environment-test(testing "the environment shadows a stored connection with the same key field by field, not wholesale"(mt/with-temporary-setting-values [llm-providers [(connection "google" "google"@@ -757,7 +772,7 @@"without updating them ships a provider that silently falls back to the generic icon. Update ""both, then this list.")(is (= #{"anthropic" "openai" "openrouter" "mistral" "zai" "moonshot" "deepseek" "xai" "google" "azure"- "bedrock" "vllm" "ollama" "metabase"}+ "bedrock" "vllm" "ollama" "openai-compatible" "metabase"}(into #{} (map :type) (llm.provider/provider-types))))))(deftest ^:parallel provider-types-test@@ -800,6 +815,7 @@;; connecting adopts the model its probe exercised"vllm" nil"ollama" nil+ "openai-compatible" nil"metabase" "anthropic/claude-sonnet-4-6"}(into {} (map (juxt :type #(llm.provider/default-model (:type %)))) (llm.provider/provider-types))))(is (nil? (llm.provider/default-model "evilai"))))@@ -821,6 +837,7 @@;; to fall back to"vllm" nil"ollama" nil+ "openai-compatible" nil"metabase" nil}(into {} (map (juxt :type #(llm.provider/mini-model (:type %)))) (llm.provider/provider-types))))(is (nil? (llm.provider/mini-model "evilai")))))diff --git a/test/metabase/metabot/self/adapter_test.clj b/test/metabase/metabot/self/adapter_test.cljindex c35b6f5abe9..94c666684d1 100644--- a/test/metabase/metabot/self/adapter_test.clj+++ b/test/metabase/metabot/self/adapter_test.clj@@ -17,6 +17,7 @@[metabase.metabot.self.moonshot :as moonshot][metabase.metabot.self.ollama :as ollama][metabase.metabot.self.openai :as openai]+ [metabase.metabot.self.openai-compatible :as openai-compatible][metabase.metabot.self.openrouter :as openrouter][metabase.metabot.self.registry :as registry][metabase.metabot.self.vllm :as vllm]@@ -47,6 +48,7 @@#'moonshot/provider :metabot.moonshot/request#'ollama/provider :metabot.ollama/request#'openai/provider :metabot.openai/request+ #'openai-compatible/provider :metabot.openai-compatible/request#'openrouter/provider :metabot.openrouter/request#'vllm/provider :metabot.vllm/request#'xai/provider :metabot.xai/request@@ -76,6 +78,7 @@;; the shared msgid, which renders identically#'ollama/provider "Ollama API error (HTTP 418)"#'openai/provider "OpenAI API error (HTTP 418)"+ #'openai-compatible/provider "OpenAI-compatible API error (HTTP 418)"#'openrouter/provider "OpenRouter API error (HTTP 418)"#'vllm/provider "vLLM API error (HTTP 418)"#'xai/provider "xAI API error (HTTP 418)"diff --git a/test/metabase/metabot/self/openai/chat_completions_test.clj b/test/metabase/metabot/self/openai/chat_completions_test.cljindex 84d4cd2fada..82a65328dc2 100644--- a/test/metabase/metabot/self/openai/chat_completions_test.clj+++ b/test/metabase/metabot/self/openai/chat_completions_test.clj@@ -343,6 +343,78 @@:usage dual-location-usage}]}{:choices [] :usage dual-location-usage}]))))))+(def ^:private running-usage-chunks+ "A reply from a server that repeats its running usage totals on every chunk."+ [{:id "chatcmpl-5"+ :model "qwen3-235b"+ :choices [{:index 0 :delta {:role "assistant" :content "Hel"}}]+ :usage {:prompt_tokens 120 :completion_tokens 1}}+ {:choices [{:index 0 :delta {:content "lo"}}]+ :usage {:prompt_tokens 120 :completion_tokens 2}}+ {:choices [{:index 0 :delta {} :finish_reason "stop"}]+ :usage {:prompt_tokens 120 :completion_tokens 3}}])++(deftest ^:parallel usage-once-counts-running-usage-once-test+ (testing "a server that repeats its running totals on every chunk is counted once, with its last totals"+ (is (= [:start :text-start :text-delta :text-delta :text-end :usage]+ (chunk-types (chat-completions/usage-once running-usage-chunks))))+ (is (=? [{:usage {:promptTokens 120 :completionTokens 3}+ :finish-reason "stop"}]+ (usage-parts (chat-completions/usage-once running-usage-chunks))))))++(defn- sse-stream+ "An SSE body that sends `chunks`."+ [chunks]+ (java.io.ByteArrayInputStream.+ (.getBytes ^String (str/join (map #(str "data: " (json/encode %) "\n\n") chunks)) "UTF-8")))++(defn- failing-sse-stream+ "An SSE body that sends `chunks`, then fails the way a reset connection does."+ [chunks]+ (java.io.SequenceInputStream.+ (sse-stream chunks)+ (proxy [java.io.InputStream] []+ (read+ ([] (throw (java.io.IOException. "Connection reset")))+ ([_bytes _offset _length] (throw (java.io.IOException. "Connection reset")))))))++(deftest ^:parallel usage-once-passes-on-usage-when-the-stream-fails-test+ (testing "a stream that fails partway still counts the usage it received, once, before the failure"+ (let [parts (atom [])]+ (is (thrown-with-msg?+ java.io.IOException #"Connection reset"+ (transduce (chat-completions/chat-completions->aisdk-chunks-xf)+ (fn ([acc] acc) ([acc part] (swap! parts conj part) acc))+ nil+ (chat-completions/usage-once+ (self.core/sse-reducible (failing-sse-stream (pop running-usage-chunks)))))))+ (is (=? [{:type :usage :usage {:promptTokens 120 :completionTokens 2}}]+ (filterv #(= :usage (:type %)) @parts))))))++(deftest ^:parallel usage-once-passes-on-usage-when-the-consumer-stops-test+ (testing "a consumer that stops partway, as a disconnected client does, still counts the usage it received, once"+ (let [parts (atom [])]+ (transduce (chat-completions/chat-completions->aisdk-chunks-xf)+ (fn ([acc] acc)+ ([acc part]+ (swap! parts conj part)+ (cond-> acc (= "lo" (:delta part)) reduced)))+ nil+ (chat-completions/usage-once+ (self.core/sse-reducible (sse-stream running-usage-chunks))))+ (is (=? [{:type :usage :usage {:promptTokens 120 :completionTokens 2}}]+ (filterv #(= :usage (:type %)) @parts))))))++(deftest ^:parallel count-reasoning-as-output-test+ (testing "reasoning tokens reported next to completion_tokens count as output, whether or not they're inside it"+ (are [usage completion-tokens] (= completion-tokens+ (get-in (chat-completions/count-reasoning-as-output {:usage usage})+ [:usage :completion_tokens]))+ {:prompt_tokens 32 :completion_tokens 9 :total_tokens 135} 103+ {:prompt_tokens 32 :completion_tokens 103 :total_tokens 135} 103+ {:prompt_tokens 32 :completion_tokens 103 :total_tokens 100} 103+ {:prompt_tokens 32 :completion_tokens 9} 9)))+(deftest ^:parallel chunks-xf-cache-reads-come-from-prompt-tokens-details-test(testing "cacheReadTokens is read from prompt_tokens_details, and cached tokens are a subset of promptTokens"(is (=? {:type :usagediff --git a/test/metabase/metabot/self/openai_compatible_test.clj b/test/metabase/metabot/self/openai_compatible_test.cljnew file mode 100644index 00000000000..c9d3378ba10--- /dev/null+++ b/test/metabase/metabot/self/openai_compatible_test.clj@@ -0,0 +1,163 @@+(ns metabase.metabot.self.openai-compatible-test+ (:require+ [clj-http.client :as http]+ [clojure.string :as str]+ [clojure.test :refer :all]+ [metabase.llm.settings :as llm.settings]+ [metabase.metabot.self.core :as self.core]+ [metabase.metabot.self.debug :as debug]+ [metabase.metabot.self.openai-compatible :as openai-compatible]+ [metabase.metabot.self.vllm :as vllm]+ [metabase.test :as mt]+ [metabase.util.json :as json])+ (:import+ (java.net ConnectException)))++(set! *warn-on-reflection* true)++(def ^:private credentials+ {:base-url "https://inference.internal/v1" :api-key "sk-test" :model-id "gpt-oss-120b"})++(def ^:private tool-call+ {:message {:content ""+ :tool_calls [{:id "call-1"+ :type "function"+ :function {:name "record_table_name" :arguments "{\"table_name\": \"orders\"}"}}]}+ :finish_reason "tool_calls"})++(defn- no-model-list []+ (throw (ex-info "clj-http: status 404" {:status 404 :body "Not Found"})))++(defn- connect!+ "Run the connect-time listing against a stubbed server, returning the listing and the models the checks named.++ The stub's `GET /models` answers `(models-response)`, and its chat completions answer `(choice request-body)`."+ [models-response choice]+ (let [checked (atom [])]+ (mt/with-dynamic-fn-redefs [http/request (fn [{:keys [url body]}]+ (if (re-find #"/models$" (str url))+ (models-response)+ (let [request (json/decode+kw (str body))]+ (swap! checked conj (:model request))+ {:status 200 :body {:choices [(choice request)]}})))]+ {:listing (openai-compatible/list-models {:credentials credentials :model "gpt-oss-120b" :probe? true})+ :checked @checked})))++(deftest connect-checks-the-named-model-test+ (testing "the model the connection names is checked whether or not the server lists it"+ (doseq [[server-lists models-response] [["no models" no-model-list]+ ["other models only" (constantly {:status 200 :body {:data [{:id "other"}]}})]]]+ (testing server-lists+ (is (= {:listing {:models [{:id "gpt-oss-120b" :display_name "gpt-oss-120b"}]+ :connection-info {vllm/reasoning-config-key "false"}}+ :checked ["gpt-oss-120b" "gpt-oss-120b"]}+ (connect! models-response (constantly tool-call))))))))++(deftest connect-checks-a-listed-context-window-test+ (testing "a context window the server lists for the model is checked, under vLLM's name or OpenRouter's"+ (doseq [field [:max_model_len :context_length]]+ (testing field+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"gpt-oss-120b has a 4096 token context window, which is too small for Metabot"+ (connect! (constantly {:status 200 :body {:data [{:id "gpt-oss-120b" field 4096}]}})+ (constantly tool-call))))))))++(deftest ^:parallel streams-reasoning-test+ (testing "a saved connection reports what its connect check saw"+ (is (true? (openai-compatible/streams-reasoning? {:credentials (assoc credentials :model-reasoning "true")})))+ (is (false? (openai-compatible/streams-reasoning? {:credentials (assoc credentials :model-reasoning "false")}))))+ (testing "one configured by environment variables never ran the check, so the chat shows any reasoning"+ (is (true? (openai-compatible/streams-reasoning? {:credentials credentials})))))++(deftest plain-listing-makes-no-request-test+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (throw (ex-info "should never be called" {})))]+ (is (= {:models [{:id "gpt-oss-120b" :display_name "gpt-oss-120b"}]}+ (openai-compatible/list-models {:credentials credentials :model "gpt-oss-120b"})))))++(deftest connect-failures-say-what-to-check-test+ (testing "a model that answers with text instead of calling a tool"+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"^gpt-oss-120b answered with text instead of calling a tool. Check that the server supports tool calling"+ (connect! no-model-list (constantly {:message {:content "Orders it is." :tool_calls []} :finish_reason "stop"})))))+ (testing "a server that ignores tool_choice, as Ollama does"+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"^The server answered with text when tool_choice required a tool call\..* If this is an Ollama server, add an Ollama provider instead\.$"+ (connect! no-model-list (fn [{[{:keys [content]}] :messages}]+ (if (re-find #"table name" content)+ tool-call+ {:message {:content "Hello!" :tool_calls []} :finish_reason "stop"}))))))+ (testing "a model ID the server doesn't serve keeps the server's own reason"+ (mt/with-dynamic-fn-redefs [http/request (fn [_]+ (throw (ex-info "clj-http: status 404"+ {:status 404+ :body "{\"message\":\"The model `gpt-oss-12b` does not exist.\"}"})))]+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"^The server has no Chat Completions endpoint at this base URL, or no model with this ID.*does not exist"+ (openai-compatible/list-models {:credentials credentials :model "gpt-oss-12b" :probe? true})))))+ (testing "a server that can't be reached is a client error naming the base URL"+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (throw (ConnectException. "Connection refused")))]+ (let [e (try (openai-compatible/list-models {:credentials credentials :model "gpt-oss-120b" :probe? true})+ (catch clojure.lang.ExceptionInfo e e))]+ (is (= "Could not reach the server at https://inference.internal/v1. Check that it is running and that the base URL is correct."+ (ex-message e)))+ (is (= 400 (:status-code (ex-data e))))))))++(defn- sent-chat-request+ "The HTTP request that `stream`, an adapter's stream fn, sends for a one-message chat turn on [[credentials]]."+ [stream]+ (let [captured (atom nil)]+ (mt/with-dynamic-fn-redefs [self.core/sse-reducible (fn [_] (reify clojure.lang.IReduceInit+ (reduce [_ _rf init] init)))+ debug/capture-stream (fn [r _] r)+ http/request (fn [req] (reset! captured req) {:body nil})]+ (into [] (stream {:model "gpt-oss-120b"+ :input [{:role :user :content "hi"}]+ :credentials credentials})))+ @captured))++(deftest chat-request-test+ (testing "a chat turn goes to the connection's server, with its key and vLLM's timeout"+ (let [request (sent-chat-request openai-compatible/openai-compatible)]+ (is (=? {:url "https://inference.internal/v1/chat/completions"+ :headers {"Authorization" "Bearer sk-test"}+ :socket-timeout (llm.settings/llm-vllm-request-timeout-ms)}+ request))+ (is (=? {:model "gpt-oss-120b"}+ (json/decode+kw (:body request)))))))++(deftest requests-carry-no-temperature-test+ (testing "a chat turn sends no temperature, where vLLM's sends one"+ (is (not (contains? (json/decode+kw (:body (sent-chat-request openai-compatible/openai-compatible)))+ :temperature)))+ (is (contains? (json/decode+kw (:body (sent-chat-request vllm/vllm)))+ :temperature)))+ (testing "nor do the two calls of the connect check"+ (let [requests (atom [])]+ (connect! no-model-list (fn [request]+ (swap! requests conj request)+ tool-call))+ (is (= 2 (count @requests)))+ (is (not-any? #(contains? % :temperature) @requests)))))++(deftest chat-usage-test+ (testing "a streamed reply counts its usage once, with reasoning tokens the server reports separately as output"+ (let [chunks [{:id "chatcmpl-1"+ :model "grok-4.3"+ :choices [{:index 0 :delta {:role "assistant" :content "Hi"}}]+ :usage {:prompt_tokens 32 :completion_tokens 1 :total_tokens 120}}+ {:choices [{:index 0 :delta {} :finish_reason "stop"}]+ :usage {:prompt_tokens 32 :completion_tokens 9 :total_tokens 135}}]+ body (str/join (map #(str "data: " (json/encode %) "\n\n") chunks))]+ (mt/with-dynamic-fn-redefs [debug/capture-stream (fn [r _] r)+ http/request (fn [_]+ {:body (java.io.ByteArrayInputStream.+ (.getBytes ^String body "UTF-8"))})]+ (is (=? [{:type :usage :usage {:promptTokens 32 :completionTokens 103}}]+ (filterv #(= :usage (:type %))+ (into [] (openai-compatible/openai-compatible {:model "grok-4.3"+ :input [{:role :user :content "hi"}]+ :credentials credentials})))))))))diff --git a/test/metabase/metabot/self/vllm_test.clj b/test/metabase/metabot/self/vllm_test.cljindex 932baf66b7f..df5e70ed0b3 100644--- a/test/metabase/metabot/self/vllm_test.clj+++ b/test/metabase/metabot/self/vllm_test.clj@@ -1225,6 +1225,24 @@{:message {:content "I can record the table name for you." :tool_calls []}:finish_reason "stop"})))))+(deftest preflight-rejects-a-server-that-ignores-tool-choice-test+ (testing "the forced tool call check asks for nothing a tool would do, so a server that treats `required` as+ `auto` answers it with text and fails"+ (mt/with-dynamic-fn-redefs [http/request (fn [{:keys [url body]}]+ (if (re-find #"/models$" (str url))+ {:status 200 :body {:data [{:id "vllm-test" :max_model_len 32768}]}}+ (let [prompt (-> body str json/decode+kw :messages first :content)]+ {:status 200+ :body {:choices [(if (re-find #"table name" prompt)+ {:message tool-calling-message+ :finish_reason "tool_calls"}+ {:message {:content "Hello!" :tool_calls []}+ :finish_reason "stop"})]}})))]+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"did not honor a forced tool call"+ (vllm/list-models {:credentials credentials :probe? true}))))))+(deftest preflight-reports-a-forced-tool-call-that-ran-out-of-budget-test(testing "a forced call truncated at the ceiling is named as such, not as a server refusing to honor it"(is (thrown-with-msg?@@ -1265,7 +1283,7 @@(is (true? (deref interrupted 10000 :never-cancelled)))))))(deftest preflight-probe-request-shape-test- (testing "both probes ask for the same trivial completion, non-streaming and deterministic — the ceiling+ (testing "both probes ask for a trivial completion, non-streaming and deterministic — the ceilingis the one a forced tool call is guaranteed to be given at request time"(let [requests (atom [])](mt/with-dynamic-fn-redefs [http/request (fn [{:keys [url body] :as req}]