Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

[BOT-1850] Ollama metabot support (#82065)

Voytek Pituła6be6d1edcb88
diff --git a/.storybook/preview.tsx b/.storybook/preview.tsx
index b236d75dc37..19958c10ef1 100644
--- a/.storybook/preview.tsx
+++ b/.storybook/preview.tsx
@@ -22,19 +22,16 @@ require("metabase/visualizations/components/EChartsRenderer/EChartsRenderer");
import "@mantine/core/styles.css";
import "@mantine/dates/styles.css";
+import { Global, css, useTheme } from "@emotion/react";
+import { initialize, mswLoader } from "msw-storybook-addon";
import { OverlayStackProvider } from "metabase/ui/components/overlays/overlay-stack";
import { EmotionCacheProvider } from "metabase/ui/components/theme/EmotionCacheProvider";
-
-import { Global, css, useTheme } from "@emotion/react";
-
import {
getSaveDomImageStyles,
loadVisualizationComponents,
} from "metabase/viz-core";
-import { initialize, mswLoader } from "msw-storybook-addon";
-
// Inject @font-face declarations synchronously at preview load so that bundled
// fonts are registered with `document.fonts` before any story's loaders run.
// Without this, the Emotion <Global /> in the decorator below registers them
diff --git a/docs/ai/providers.md b/docs/ai/providers.md
index 9118a613713..365f3787f85 100644
--- a/docs/ai/providers.md
+++ b/docs/ai/providers.md
@@ -231,9 +231,9 @@ Credentials:
- **Project ID**. The Google Cloud project to use. Optional if the service account key provides it. [Where do I find this?](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) You can also set it with the environment variable `MB_LLM_GOOGLE_PROJECT_ID`.
- **Location**. Optional. Defaults to global. You can also set it with the environment variable `MB_LLM_GOOGLE_LOCATION`.
-- **Authentication method** (required). Authenticate with a service account key or an OAuth access token. One of: `Service account key`, `OAuth token`. Defaults to `Service account key`.
-- **Service account key file**. Only when **Authentication method** is **Service account key**. Upload a service account key file to authenticate with. [Where do I find this?](https://docs.cloud.google.com/iam/docs/keys-create-delete) You can also set it with the environment variable `MB_LLM_GOOGLE_SERVICE_ACCOUNT_KEY`.
-- **OAuth access token**. Only when **Authentication method** is **OAuth token**. A short-lived token, e.g. the output of gcloud auth print-access-token. Useful for testing. You can also set it with the environment variable `MB_LLM_GOOGLE_OAUTH_ACCESS_TOKEN`.
+- **Authentication method** (required). Authenticate with a service account key or an OAuth access token. One of: `service-account-key`, `oauth-token`. Defaults to `service-account-key`.
+- **Service account key file**. Only when **Authentication method** is **Service account key** (`service-account-key`). Upload [a service account key file to authenticate with](https://docs.cloud.google.com/iam/docs/keys-create-delete). You can also set it with the environment variable `MB_LLM_GOOGLE_SERVICE_ACCOUNT_KEY`.
+- **OAuth access token**. Only when **Authentication method** is **OAuth token** (`oauth-token`). A short-lived token, e.g. the output of gcloud auth print-access-token. Useful for testing. You can also set it with the environment variable `MB_LLM_GOOGLE_OAUTH_ACCESS_TOKEN`.
- **Model Garden endpoint ID**. Optional. Use an open model you deployed from Model Garden instead of one Google hosts. Set the location to the region you deployed it to.
- **API base URL** (advanced). Derived from the location when left at the global host. Defaults to `https://aiplatform.googleapis.com`. You can also set it with the environment variable `MB_LLM_GOOGLE_API_BASE_URL`.
@@ -263,7 +263,7 @@ Credentials:
- **API key** (required). [Where do I find this?](https://ai.azure.com) You can also set it with the environment variable `MB_LLM_AZURE_API_KEY`.
- **API base URL** (required). You can also set it with the environment variable `MB_LLM_AZURE_API_BASE_URL`.
-- **Model provider** (required). Whether your deployment serves an Anthropic or an OpenAI model. One of: `OpenAI`, `Anthropic`. Defaults to `OpenAI`. You can also set it with the environment variable `MB_LLM_AZURE_MODEL_FAMILY`.
+- **Model provider** (required). Whether your deployment serves an Anthropic or an OpenAI model. One of: `openai`, `anthropic`. Defaults to `openai`. You can also set it with the environment variable `MB_LLM_AZURE_MODEL_FAMILY`.
- **Deployment name** (required). The name of the model deployment on your Azure resource. We recommend naming deployments after the model they serve. You can also set it with the environment variable `MB_LLM_AZURE_DEPLOYMENT_NAME`.
## Amazon Bedrock
@@ -362,13 +362,30 @@ If you configure the Bedrock connection with environment variables, it has no mo
Supported models:
-Metabase lists whichever models your vLLM server is serving, so what you can pick depends on how you started it.
+Metabase lists whichever models your vLLM server has available, so what you can pick depends on how you set it up.
Credentials:
- **API base URL** (required). Your server's OpenAI-compatible API. It should end in /v1. Metabase must be able to reach it: self-hosted, a server on your private network or on this machine needs MB_LLM_ALLOWED_NETWORKS. You can also set it with the environment variable `MB_LLM_VLLM_API_BASE_URL`.
- **API key**. Only needed if you started your server with --api-key. You can also set it with the environment variable `MB_LLM_VLLM_API_KEY`.
+## Ollama
+
+- Provider key: `ollama`
+
+Supported models:
+
+Metabase lists whichever models your Ollama server has available, so what you can pick depends on how you set it up.
+
+Credentials:
+
+- **API base URL** (required). Your Ollama server's address, ending in /v1, or https://ollama.com/v1 for Ollama Cloud. To reach a server on your private network, set `MB_LLM_ALLOWED_NETWORKS=allow-private`; for one on this machine, allow-all. You can also set it with the environment variable `MB_LLM_OLLAMA_API_BASE_URL`.
+- **API key**. Required for Ollama Cloud. Leave blank if your server doesn't require one. You can also set it with the environment variable `MB_LLM_OLLAMA_API_KEY`.
+
+### Context window
+
+Ollama can load a model with a smaller context window than the model supports. Metabot needs at least 16,384 tokens, and Metabase checks the window when you connect. See Ollama's documentation on [context length](https://docs.ollama.com/context-length).
+
## Metabase AI service
- Provider key: `metabase`
diff --git a/docs/ai/settings.md b/docs/ai/settings.md
index e5fb0750f3c..b954dde0012 100644
--- a/docs/ai/settings.md
+++ b/docs/ai/settings.md
@@ -94,6 +94,8 @@ If you're self-hosting, you can configure a provider with [environment variables
An environment variable can also override a single field of a connection you manage in the UI. For example, if you set only `MB_LLM_ANTHROPIC_API_BASE_URL`, the base URL comes from the environment, and the rest of the connection stays editable.
+A connection's credentials and its address have to come from the same place. For Ollama, if `MB_LLM_OLLAMA_API_BASE_URL` points a connection at a different server (such as Ollama Cloud at `https://ollama.com/v1`), Metabase won't send an API key entered in the UI there: set `MB_LLM_OLLAMA_API_KEY` too.
+
To put the whole list under environment control, set [`MB_LLM_PROVIDERS`](../configuring-metabase/environment-variables.md#mb_llm_providers) to a JSON array of connections. The provider list is then read-only, so manage your connections by editing `MB_LLM_PROVIDERS` and restarting.
On Metabase Cloud, [contact support](https://www.metabase.com/help-premium) if you want environment variables set for your instance.
diff --git a/docs/configuring-metabase/config-template.md b/docs/configuring-metabase/config-template.md
index 91b679050a7..e92bbd279f7 100644
--- a/docs/configuring-metabase/config-template.md
+++ b/docs/configuring-metabase/config-template.md
@@ -233,6 +233,9 @@ config:
llm-mistral-api-key: null
llm-moonshot-api-base-url: https://api.moonshot.ai/v1
llm-moonshot-api-key: null
+ llm-ollama-api-base-url: null
+ llm-ollama-api-key: null
+ llm-ollama-request-timeout-ms: 300000
llm-openai-api-base-url: https://api.openai.com
llm-openai-api-key: null
llm-openai-model: gpt-5.4
diff --git a/docs/configuring-metabase/environment-variables.md b/docs/configuring-metabase/environment-variables.md
index c6ef18ce363..d43685ccbef 100644
--- a/docs/configuring-metabase/environment-variables.md
+++ b/docs/configuring-metabase/environment-variables.md
@@ -1579,6 +1579,34 @@ The Moonshot AI API Key.
Backed by the moonshot connection in the admin AI settings provider list: reads and writes go through the llm-providers connection list, and a value set by this environment variable shadows this one field of that connection.
+### `MB_LLM_OLLAMA_API_BASE_URL`
+
+- Type: string
+- Default: `null`
+- [Configuration file name](./config-file.md): `llm-ollama-api-base-url`
+
+The base URL of your Ollama server's OpenAI-compatible API, e.g. `http://localhost:11434/v1`, or `https://ollama.com/v1` for Ollama Cloud.
+
+Backed by the Ollama connection in the admin AI settings provider list: reads and writes go through the llm-providers connection list. A value set by this environment variable shadows that connection's base URL.
+
+### `MB_LLM_OLLAMA_API_KEY`
+
+- Type: string
+- Default: `null`
+- [Configuration file name](./config-file.md): `llm-ollama-api-key`
+
+The API key for Ollama Cloud, with `MB_LLM_OLLAMA_API_BASE_URL` set to https://ollama.com/v1. For self-hosted servers, only needed behind an authenticated proxy.
+
+Backed by the Ollama connection in the admin AI settings provider list: reads and writes go through the llm-providers connection list, and a value set by this environment variable shadows this one field of that connection.
+
+### `MB_LLM_OLLAMA_REQUEST_TIMEOUT_MS`
+
+- Type: integer
+- Default: `300000`
+- [Configuration file name](./config-file.md): `llm-ollama-request-timeout-ms`
+
+Socket timeout in milliseconds for requests to your Ollama server.
+
### `MB_LLM_OPENAI_API_BASE_URL`
- Type: string
diff --git a/frontend/src/metabase-types/api/llm.ts b/frontend/src/metabase-types/api/llm.ts
index 9b4abde19dc..c60e9eaab9b 100644
--- a/frontend/src/metabase-types/api/llm.ts
+++ b/frontend/src/metabase-types/api/llm.ts
@@ -46,6 +46,7 @@ export type LlmProviderTypeName =
| "azure"
| "bedrock"
| "vllm"
+ | "ollama"
| "metabase";
export type LlmProviderFieldType =
diff --git a/frontend/src/metabase-types/api/settings.ts b/frontend/src/metabase-types/api/settings.ts
index cfdc7b13bf9..251a58760ec 100644
--- a/frontend/src/metabase-types/api/settings.ts
+++ b/frontend/src/metabase-types/api/settings.ts
@@ -557,6 +557,9 @@ interface SettingsManagerSettings {
"llm-bedrock-session-token"?: string | null;
"llm-vllm-api-base-url"?: string | null;
"llm-vllm-api-key"?: string | null;
+ "llm-ollama-api-base-url"?: string | null;
+ "llm-ollama-api-key"?: string | null;
+ "llm-ollama-request-timeout-ms"?: number | null;
"openai-api-key": string | null;
"openai-available-models"?: OpenAiModel[];
"openai-model": string | null;
diff --git a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderConnectionForm.tsx b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderConnectionForm.tsx
index 062a0390c83..f279297fcfe 100644
--- a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderConnectionForm.tsx
+++ b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderConnectionForm.tsx
@@ -34,6 +34,11 @@ import { ProviderTypePicker } from "./ProviderTypePicker";
import { findProviderTypeForApiKey } from "./api-key";
import { getHiddenFieldKeys, isVisibleField } from "./visible-fields";
+const OLLAMA_TYPE = "ollama";
+const BASE_URL_FIELD = "base-url";
+// Ollama Cloud is reached like any other Ollama server, at its own address.
+const OLLAMA_CLOUD_BASE_URL = "https://ollama.com/v1";
+
export function ProviderConnectionForm({
providerTypes,
connection,
@@ -247,6 +252,24 @@ export function ProviderConnectionForm({
disabledFields={connection?.env_fields}
autoFocusFirstField
/>
+ {selected.type === OLLAMA_TYPE &&
+ !connection?.env_fields.includes(BASE_URL_FIELD) && (
+ <Group>
+ <Button
+ type="button"
+ variant="subtle"
+ onClick={() =>
+ setConfig((current) => ({
+ ...current,
+ [BASE_URL_FIELD]: OLLAMA_CLOUD_BASE_URL,
+ }))
+ }
+ disabled={isSaving}
+ >
+ {t`Use Ollama Cloud`}
+ </Button>
+ </Group>
+ )}
{selected.models.length > 0 && !hasConfiguredModel && (
<Select
label={t`Model`}
diff --git a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderConnectionForm.unit.spec.tsx b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderConnectionForm.unit.spec.tsx
index d0f06beb10e..6db1ff6e02e 100644
--- a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderConnectionForm.unit.spec.tsx
+++ b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderConnectionForm.unit.spec.tsx
@@ -517,3 +517,93 @@ describe("ProviderConnectionForm editing a fixed-catalog connection", () => {
expect(screen.getByLabelText("OAuth access token")).toBeEnabled();
});
});
+
+const OLLAMA_TYPE = createMockLlmProviderType({
+ type: "ollama",
+ label: "Ollama",
+ default_model: null,
+ models: [],
+ fields: [
+ createMockLlmProviderField({
+ key: "base-url",
+ label: "API base URL",
+ type: "text",
+ required: true,
+ }),
+ createMockLlmProviderField({
+ key: "api-key",
+ label: "API key",
+ type: "password",
+ required: false,
+ }),
+ ],
+});
+
+const setupOllama = () => {
+ fetchMock.removeRoutes();
+ fetchMock.clearHistory();
+ setupCreateLlmProviderEndpoint();
+ const onSaved = jest.fn();
+
+ renderWithProviders(
+ <ProviderConnectionForm providerTypes={[OLLAMA_TYPE]} onSaved={onSaved} />,
+ );
+
+ return { onSaved };
+};
+
+const pickOllama = async () => {
+ await userEvent.click(screen.getByRole("button", { name: /Ollama/ }));
+ await screen.findByLabelText(/API base URL/);
+};
+
+describe("ProviderConnectionForm for Ollama", () => {
+ it("needs a base URL to connect", async () => {
+ setupOllama();
+
+ await pickOllama();
+ await userEvent.type(screen.getByLabelText(/API key/), "k");
+
+ expect(screen.getByRole("button", { name: "Connect" })).toBeDisabled();
+ });
+
+ it("fills in Ollama Cloud's address, and saves nothing else for it", async () => {
+ const { onSaved } = setupOllama();
+
+ await pickOllama();
+ await userEvent.click(
+ screen.getByRole("button", { name: "Use Ollama Cloud" }),
+ );
+ expect(screen.getByLabelText(/API base URL/)).toHaveValue(
+ "https://ollama.com/v1",
+ );
+ await userEvent.type(screen.getByLabelText(/API key/), "k");
+ await userEvent.click(screen.getByRole("button", { name: "Connect" }));
+
+ await waitFor(() => {
+ expect(onSaved).toHaveBeenCalled();
+ });
+ expect(
+ await fetchMock.callHistory
+ .lastCall("path:/api/llm/providers", { method: "POST" })
+ ?.request?.json(),
+ ).toEqual({
+ type: "ollama",
+ name: "Ollama",
+ // no model: Ollama's catalog comes from the server that connecting reaches, so a new
+ // connection adopts whatever its probe exercised
+ config: { "base-url": "https://ollama.com/v1", "api-key": "k" },
+ });
+ });
+
+ it("does not offer Ollama Cloud for another provider type", async () => {
+ setup();
+
+ await userEvent.click(screen.getByRole("button", { name: /Anthropic/ }));
+ await screen.findByLabelText(/API key/);
+
+ expect(
+ screen.queryByRole("button", { name: "Use Ollama Cloud" }),
+ ).not.toBeInTheDocument();
+ });
+});
diff --git a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.tsx b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.tsx
index 050abb10058..ddc084fe760 100644
--- a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.tsx
+++ b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.tsx
@@ -12,6 +12,7 @@ import DeepSeekMark from "./logos/deepseek.svg?component";
import GoogleMark from "./logos/google.svg?component";
import MistralMark from "./logos/mistral.svg?component";
import MoonshotMark from "./logos/moonshot.svg?component";
+import OllamaMark from "./logos/ollama.svg?component";
import OpenAiMark from "./logos/openai.svg?component";
import OpenRouterMark from "./logos/openrouter.svg?component";
import XaiMark from "./logos/xai.svg?component";
@@ -39,6 +40,7 @@ const PROVIDER_LOGOS: Record<
google: GoogleMark,
azure: AzureMark,
vllm: null,
+ ollama: OllamaMark,
metabase: null,
};
diff --git a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.unit.spec.tsx b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.unit.spec.tsx
index 621cf5a688f..529f825f643 100644
--- a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.unit.spec.tsx
+++ b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/ProviderTypeIcon.unit.spec.tsx
@@ -34,6 +34,7 @@ describe("ProviderTypeIcon", () => {
"google",
"azure",
"bedrock",
+ "ollama",
] as const)("renders the vendor logo for %s", (type) => {
renderWithProviders(<ProviderTypeIcon type={type} />);
diff --git a/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/logos/ollama.svg b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/logos/ollama.svg
new file mode 100644
index 00000000000..a1e26284134
--- /dev/null
+++ b/frontend/src/metabase/metabot/components/AIProviderConfigurationForm/logos/ollama.svg
@@ -0,0 +1 @@
+<svg width="24" height="24" xmlns="http://www.w3.org/2000/svg" fill="currentColor" viewBox="0 0 24 24"><g transform="translate(2.922717)scale(.028103)"><path d="M140.629 0.239929C132.66 1.52725 123.097 5.69568 116.354 10.845C95.941 26.3541 80.1253 59.2728 73.4435 100.283C70.9302 115.792 69.2138 137.309 69.2138 153.738C69.2138 173.109 71.4819 197.874 74.7309 214.977C75.4665 218.778 75.8343 222.15 75.5278 222.395C75.2826 222.64 72.2788 225.092 68.9072 227.789C57.3827 236.984 44.2029 251.145 35.1304 264.08C17.7209 288.784 6.44151 316.86 1.72133 347.265C-0.117698 359.28 -0.608106 383.555 0.863118 395.57C4.11207 423.278 12.449 446.695 26.7321 468.151L31.391 475.078L30.0424 477.346C20.4794 493.407 12.3264 516.64 8.52575 538.953C5.522 556.608 5.15419 561.328 5.15419 584.99C5.15419 608.837 5.4607 613.557 8.28054 630.047C11.6521 649.786 18.5178 670.689 26.1804 684.605C28.6938 689.141 34.8239 698.581 35.5595 699.072C35.8047 699.194 35.0691 701.462 33.9044 704.098C25.077 723.408 17.537 749.093 14.4106 770.733C12.2038 785.567 11.8973 790.349 11.8973 805.981C11.8973 825.903 13.0007 835.589 17.1692 851.466L17.7822 853.795H44.019H70.3172L68.6007 850.546C57.9957 830.93 57.0149 794.517 66.1487 758.166C70.3172 741.369 75.0374 729.048 83.8647 712.067L89.1366 701.769V695.455C89.1366 689.57 89.014 688.896 87.1137 685.034C85.6424 682.091 83.6808 679.578 80.1866 676.145C74.2404 670.383 69.9494 664.314 66.5165 656.835C51.4365 624.1 48.494 575.489 59.0991 534.049C63.5128 516.762 70.8076 501.376 78.4702 492.978C83.6808 487.215 86.378 480.779 86.378 474.097C86.378 467.17 83.926 461.469 78.4089 455.523C62.5932 438.604 52.8464 418.006 49.3522 394.038C44.3868 359.893 53.3981 322.683 73.8726 293.198C93.9181 264.263 122.055 245.689 153.503 240.724C160.552 239.559 173.732 239.743 181.088 241.092C189.119 242.502 194.145 242.072 199.295 239.62C205.67 236.617 208.858 232.877 212.597 224.295C215.907 216.633 218.482 212.464 225.409 203.821C233.746 193.461 241.776 186.411 254.649 177.89C269.362 168.266 286.097 161.278 302.771 157.906C308.839 156.68 311.659 156.496 323 156.496C334.341 156.496 337.161 156.68 343.229 157.906C367.688 162.872 391.964 175.5 411.335 193.399C415.503 197.261 425.495 209.644 428.683 214.794C429.909 216.816 432.055 221.108 433.403 224.295C437.142 232.877 440.33 236.617 446.705 239.62C451.671 242.011 456.881 242.502 464.605 241.214C476.804 239.13 486.183 239.314 498.137 241.766C538.841 249.98 574.273 283.512 589.966 328.446C603.636 367.862 599.774 409.118 579.422 440.626C575.989 445.96 572.556 450.251 567.591 455.523C556.863 466.986 556.863 481.208 567.53 492.978C585.062 512.165 596.035 559.367 592.724 600.99C590.518 628.453 583.468 653.035 573.782 666.95C572.066 669.402 568.511 673.57 565.813 676.145C562.319 679.578 560.358 682.091 558.886 685.034C556.986 688.896 556.863 689.57 556.863 695.455V701.769L562.135 712.067C570.963 729.048 575.683 741.369 579.851 758.166C588.863 794.027 588.066 829.704 577.767 849.995C576.909 851.711 576.173 853.305 576.173 853.489C576.173 853.673 587.882 853.795 602.226 853.795H628.218L628.892 851.159C629.26 849.75 629.873 847.604 630.179 846.378C630.854 843.681 632.202 835.712 633.306 828.049C634.348 820.325 634.348 791.881 633.306 783.299C629.383 752.158 622.823 727.454 612.096 704.098C610.931 701.462 610.195 699.194 610.44 699.072C610.747 698.888 612.463 696.436 614.302 693.677C627.666 673.448 635.88 648.008 640.049 614.415C641.152 605.158 641.152 565.374 640.049 556.485C637.106 533.559 633.551 517.988 627.666 502.234C625.214 495.675 618.716 481.821 615.958 477.346L614.609 475.078L619.268 468.151C633.551 446.695 641.888 423.278 645.137 395.57C646.608 383.555 646.118 359.28 644.279 347.265C639.497 316.798 628.279 288.845 610.87 264.08C601.797 251.145 588.617 236.984 577.093 227.789C573.721 225.092 570.717 222.64 570.472 222.395C570.166 222.15 570.534 218.778 571.269 214.977C578.687 176.296 578.441 128.053 570.656 90.3524C563.913 57.4951 551.653 31.3808 535.837 16.3008C523.209 4.28578 510.336 -0.863507 494.888 0.11731C459.456 2.20154 430.89 42.9667 419.61 107.21C417.771 117.57 416.178 129.708 416.178 133.018C416.178 134.305 415.932 135.347 415.626 135.347C415.319 135.347 412.929 134.121 410.354 132.589C383.014 116.405 352.608 107.762 323 107.762C293.392 107.762 262.986 116.405 235.646 132.589C233.071 134.121 230.681 135.347 230.374 135.347C230.068 135.347 229.822 134.305 229.822 133.018C229.822 129.585 228.167 117.08 226.39 107.21C216.152 49.5259 192.674 11.3354 161.472 1.71112C157.181 0.423799 144.982 -0.434382 140.629 0.239929ZM151.051 50.139C159.878 57.1273 169.686 77.1114 175.326 99.4863C176.368 103.532 177.471 108.191 177.778 109.907C178.023 111.563 178.697 115.302 179.249 118.183C181.64 131.179 182.743 145.217 182.866 162.32L182.927 179.178L178.697 185.43L174.468 191.744H164.598C153.074 191.744 141.61 193.216 130.637 196.158C126.714 197.139 122.913 198.12 122.178 198.304C121.013 198.549 120.829 198.181 120.155 193.154C116.538 165.875 116.722 135.654 120.707 110.52C125.12 82.5059 135.419 57.1273 145.472 49.6486C147.863 47.8708 148.292 47.9321 151.051 50.139ZM500.589 49.7098C506.658 54.1848 513.34 66.0772 518.305 81.2798C528.297 111.685 531.117 153.431 525.845 193.154C525.171 198.181 524.987 198.549 523.822 198.304C523.087 198.12 519.286 197.139 515.363 196.158C504.39 193.216 492.926 191.744 481.402 191.744H471.532L467.303 185.43L463.073 179.178L463.134 162.32C463.257 138.535 465.464 119.961 470.735 99.3024C476.314 77.1114 486.183 57.1273 494.949 50.139C497.708 47.9321 498.137 47.8708 500.589 49.7098Z"/><path d="M313.498 358.237C300.195 359.525 296.579 360.015 290.203 361.303C279.843 363.448 265.989 368.23 256.365 372.95C222.895 389.317 199.846 416.596 192.796 448.166C191.386 454.419 191.202 456.503 191.202 467.047C191.202 477.468 191.386 479.736 192.735 485.682C202.114 526.938 240.12 557.405 289.284 562.983C299.95 564.148 346.049 564.148 356.715 562.983C396.193 558.508 430.154 537.114 445.418 507.076C449.463 499.046 451.425 493.835 453.264 485.682C454.613 479.736 454.797 477.468 454.797 467.047C454.797 456.503 454.613 454.419 453.203 448.166C442.965 402.313 398.461 366.207 343.903 359.341C336.792 358.483 318.157 357.747 313.498 358.237ZM336.424 391.585C354.631 393.547 372.96 400.045 387.672 409.853C395.58 415.125 406.737 426.159 411.518 433.393C417.403 442.342 420.774 451.476 422.307 462.572C422.981 467.66 422.614 471.522 420.774 479.736C417.893 491.996 408.943 504.808 396.867 513.758C391.227 517.865 379.519 523.812 372.347 526.141C358.738 530.493 349.849 531.29 318.095 531.045C297.376 530.861 293.697 530.677 287.751 529.574C267.461 525.773 251.4 517.681 239.753 505.36C230.312 495.429 226.021 486.357 223.692 471.706C222.65 464.901 224.611 453.622 228.596 444.12C233.439 432.534 245.944 418.129 258.327 409.853C272.671 400.29 291.552 393.486 308.9 391.647C315.582 390.911 329.742 390.911 336.424 391.585Z"/><path d="M299.584 436.336C294.925 438.849 291.676 445.224 292.657 449.944C293.76 455.032 298.235 460.182 305.223 464.412C308.963 466.68 309.208 466.986 309.392 469.254C309.514 470.603 309.024 474.465 308.35 477.898C307.614 481.269 307.062 484.825 307.062 485.806C307.124 488.442 309.576 492.733 312.15 494.817C314.419 496.656 314.848 496.717 321.223 496.901C327.047 497.085 328.273 496.962 330.602 495.859C336.61 492.916 338.142 487.522 335.935 477.162C334.096 468.519 334.464 467.17 339.062 464.534C343.904 461.714 349.054 456.749 350.586 453.377C353.529 446.941 350.831 439.646 344.333 436.274C342.74 435.477 340.778 435.11 337.897 435.11C333.422 435.11 330.541 436.152 325.269 439.523L322.265 441.424L320.365 440.259C312.58 435.661 311.17 435.11 306.449 435.171C303.078 435.171 301.239 435.477 299.584 436.336Z"/><path d="M150.744 365.165C139.894 368.598 131.802 376.567 127.634 387.908C125.611 393.303 124.63 401.824 125.488 406.421C127.511 417.394 136.522 427.386 146.76 430.145C159.633 433.516 169.257 431.309 177.778 422.85C182.743 418.007 185.441 413.777 188.138 406.911C190.099 402.069 190.222 401.211 190.222 394.345L190.283 386.989L187.709 381.717C183.601 373.38 176.184 367.188 167.602 364.92C162.759 363.694 154.974 363.756 150.744 365.165Z"/><path d="M478.153 364.982C469.755 367.25 462.276 373.502 458.291 381.717L455.717 386.989L455.778 394.345C455.778 401.211 455.901 402.069 457.862 406.911C460.56 413.777 463.257 418.007 468.222 422.85C476.743 431.309 486.367 433.516 499.241 430.145C506.658 428.183 514.075 421.93 517.631 414.635C520.696 408.444 521.431 403.969 520.451 396.919C518.183 380.797 508.742 369.089 494.704 364.982C490.597 363.756 482.628 363.756 478.153 364.982Z"/></g></svg>
diff --git a/src/metabase/cmd/ai_provider_dox.clj b/src/metabase/cmd/ai_provider_dox.clj
index 04b8b45a3db..b19bcedcc1f 100644
--- a/src/metabase/cmd/ai_provider_dox.clj
+++ b/src/metabase/cmd/ai_provider_dox.clj
@@ -24,11 +24,12 @@
Google's covers connecting a Model Garden endpoint, which its fixed catalog cannot list."
{"metabase" "metabase/cmd/resources/ai-provider-metabase.md"
"bedrock" "metabase/cmd/resources/ai-provider-bedrock.md"
- "google" "metabase/cmd/resources/ai-provider-google.md"})
+ "google" "metabase/cmd/resources/ai-provider-google.md"
+ "ollama" "metabase/cmd/resources/ai-provider-ollama.md"})
(def ^:private dynamic-catalog-types
"Provider types that serve whatever models the operator loaded, so there is no list to publish."
- #{"vllm"})
+ #{"vllm" "ollama"})
(def ^:private max-enumerated-options
"Above this many `:options`, a field's choices are pointed at rather than listed. Bedrock's dozens of regions are
@@ -80,9 +81,14 @@
"The condition that reveals a field, or nil when it is always shown."
[{:keys [show-when]} {:keys [fields]}]
(when-let [{:keys [field value]} show-when]
- (let [controlling (field-at fields field)]
+ (let [controlling (field-at fields field)
+ value-label (option-label controlling value)]
+ ;; the stored value, in code, ties the label to the value [[field-options-sentence]] lists
(str "Only when " (md/bold (label controlling))
- " is " (md/bold (option-label controlling value)) "."))))
+ " is " (md/bold value-label)
+ (when (not= (str value-label) (str value))
+ (str " (" (md/code value) ")"))
+ "."))))
(defn- field-requires-sentence
"The siblings a field cannot be set without, or nil when it stands on its own. The registry keys `:requires` by
@@ -108,16 +114,17 @@
[{:keys [options]}]
(when (seq options)
(if (<= (count options) max-enumerated-options)
- (str "One of: " (str/join ", " (map #(md/code (label %)) options)) ".")
+ (str "One of: " (str/join ", " (map #(md/code (:value %)) options)) ".")
;; deliberately not a count: the long lists come from bundled SDKs, and would churn this page on every bump
(str "Pick one from the dropdown in " (md/bold "Admin > AI") "."))))
(defn- field-default-sentence
"The value a field starts at, or nil when it has no `:default`."
- [{:keys [default] :as field}]
+ [{:keys [default]}]
(when default
- ;; a field with `:options` stores one value but displays another, so show what the form shows
- (str "Defaults to " (md/code (option-label field default)) ".")))
+ ;; the stored value, not the form's label — an environment variable or an `MB_LLM_PROVIDERS`
+ ;; config map has to carry the value, and the form's own dropdown is where labels are read
+ (str "Defaults to " (md/code default) ".")))
(defn- field-env-var-sentence
"The environment variable that can stand in for filling the field in, or nil when none configures it. Phrased as an
@@ -189,8 +196,8 @@
"works out the model from " (field-labels fields model-fields) " instead.")
(contains? dynamic-catalog-types type)
- (str "Metabase lists whichever models your " provider-label " server is serving, so what you can "
- "pick depends on how you started it.")
+ (str "Metabase lists whichever models your " provider-label " server has available, so what you "
+ "can pick depends on how you set it up.")
:else
(throw (ex-info (str "No model source for provider type " (pr-str type)
diff --git a/src/metabase/cmd/resources/ai-provider-ollama.md b/src/metabase/cmd/resources/ai-provider-ollama.md
new file mode 100644
index 00000000000..18f23bade78
--- /dev/null
+++ b/src/metabase/cmd/resources/ai-provider-ollama.md
@@ -0,0 +1,3 @@
+### Context window
+
+Ollama can load a model with a smaller context window than the model supports. Metabot needs at least 16,384 tokens, and Metabase checks the window when you connect. See Ollama's documentation on [context length](https://docs.ollama.com/context-length).
diff --git a/src/metabase/llm/api/provider.clj b/src/metabase/llm/api/provider.clj
index f53dbe1d082..098b3e85e59 100644
--- a/src/metabase/llm/api/provider.clj
+++ b/src/metabase/llm/api/provider.clj
@@ -131,8 +131,7 @@
:source (name (or source :db))
:usable (llm.provider/config-complete? type config)
:env_vars (vec env-vars)
- ;; the config keys the environment owns; the form disables exactly these inputs
- :env_fields (mapv name env-fields)
+ :env_fields (mapv name (sort env-fields))
:config (or (:config (llm.provider/redact conn)) {})})
;;; ------------------------------------------------ Model listing -------------------------------------------------
@@ -201,9 +200,15 @@
{:models (or config-models (vec (:models listed)))}))
(catch clojure.lang.ExceptionInfo e
(if (provider-client-error? e)
- ;; Keep offering config-models, otherwise admin has no way to select a different model to fix "model not
- ;; served in given region" errors.
- {:models (or config-models []) :error (.getMessage e)}
+ (do
+ ;; the admin gets only the message; the parse error behind a model list that was not JSON is what
+ ;; tells an operator what actually came back. Only that error: any other carries the provider's
+ ;; response in its ex-data, which `rethrow-api-error!` keeps out of the log on purpose
+ (when (= :malformed-model-catalog (:error-code (ex-data e)))
+ (log/warn e "Listing models was refused for LLM provider connection" {:connection conn-key :type type}))
+ ;; Keep offering config-models, otherwise admin has no way to select a different model to fix "model
+ ;; not served in given region" errors.
+ {:models (or config-models []) :error (.getMessage e)})
(throw e))))))))
(def ^:private models-cache-ttl-ms
@@ -446,22 +451,32 @@
(not= llm.provider/managed-connection-key conn-key))
(tru "The {0} connection key is reserved for the Metabase AI service."
(pr-str llm.provider/managed-connection-key)))
- config (without-blank-values (update-keys config keyword))
- conn {:key conn-key
- :type type
- :name (or (not-empty name) (str (:label provider-type)))
- :config config}]
- (llm.provider/validate-config! type config)
- (let [{:keys [connection-info] :as listed} (verify-credentials! conn config model)
+ submitted (without-blank-values (update-keys config keyword))
+ conn {:key conn-key
+ :type type
+ :name (or (not-empty name) (str (:label provider-type)))
+ :config submitted}
+ env-config (llm.provider/env-overlay-config conn-key type)
+ ;; what this connection will run on the moment it exists, which a variable already naming one of
+ ;; its fields has a say in
+ effective (llm.provider/effective-config conn env-config)]
+ (llm.provider/assert-credentials-not-captured! conn submitted env-config)
+ ;; a secret the environment supplies may only go where the environment points, as on update
+ (llm.provider/assert-base-url-change-authorized! type env-config effective submitted (keys env-config))
+ (llm.provider/validate-config! type effective)
+ (let [{:keys [connection-info] :as listed} (verify-credentials! conn effective model)
conn (update conn :config with-connection-info connection-info)
had-usable-model? (metabot-has-a-usable-model?)]
(llm.provider/set-connections! (conj (llm.provider/stored-connections) conn))
- (when-not had-usable-model?
- ;; a type with no default model — vLLM, which serves whatever the operator loaded — starts on the model
- ;; the probe exercised, so connecting one leaves the instance working rather than model-less
- (select-model-for-new-connection! conn (or model (:probed-model connection-info))))
- (seed-models-cache! conn listed)
- (connection-response (assoc conn :source :db))))))
+ ;; read back rather than go on what was submitted: the environment has a say in this connection, and
+ ;; the model Metabot is pointed at, the cached listing and the answer all have to be about what it runs on
+ (let [live (llm.provider/connection conn-key)]
+ (when-not had-usable-model?
+ ;; a type with no default model — vLLM, which serves whatever the operator loaded — starts on the model
+ ;; the probe exercised, so connecting one leaves the instance working rather than model-less
+ (select-model-for-new-connection! live (or model (:probed-model connection-info))))
+ (seed-models-cache! live listed)
+ (connection-response live))))))
(api.macros/defendpoint :put "/providers/:key"
:- connection-response-schema
@@ -482,24 +497,27 @@
existing (nth stored idx)
live (llm.provider/connection conn-key)
_ (check-not-env-connection! live)
- ;; fields the environment owns are not the client's to edit — the form disables them, and what it echoes
- ;; back for them is the mask of the env value, which must not end up stored
- env-config (select-keys (:config live) (:env-fields live))
+ ;; fields the environment supplies are not the client's to edit — the form disables them, and what it
+ ;; echoes back for them is the mask of the env value, which must not end up stored
+ conn-type (:type existing)
+ env-config (llm.provider/env-overlay-config conn-key conn-type)
+ client-cfg (apply dissoc config (:env-fields live))
merged (cond-> existing
(some? config) (assoc :config (without-blank-values
(llm.provider/merge-config
- (:type existing)
- (:config existing)
- (apply dissoc config (:env-fields live)))))
+ conn-type (:config existing) client-cfg)))
(not-empty name) (assoc :name name))
- ;; what the connection will actually run on: the stored config with the environment layered back over it
- effective (merge (:config merged) env-config)]
+ ;; what the connection will actually run on, which is not everything it merged
+ effective (llm.provider/effective-config merged env-config)]
;; Before validation probes the new URL with the effective credentials, require proof that the caller holds every
;; secret that would travel there. Omitted and masked secrets were merged from storage; env-owned ones cannot be
;; re-supplied through this API at all.
- (llm.provider/assert-base-url-change-authorized! (:type merged) (:config live) effective config
+ (llm.provider/assert-base-url-change-authorized! conn-type (:config live) effective client-cfg
(:env-fields live))
- (llm.provider/validate-config! (:type merged) effective)
+ ;; and before the completeness check, which would otherwise report a credential the environment has moved this
+ ;; connection away from as one the admin never entered
+ (llm.provider/assert-credentials-not-captured! merged client-cfg env-config)
+ (llm.provider/validate-config! conn-type effective)
(let [{:keys [connection-info] :as listed}
(verify-credentials! merged effective (or model (selected-model conn-key)))
merged (update merged :config with-connection-info connection-info)
diff --git a/src/metabase/llm/provider.clj b/src/metabase/llm/provider.clj
index bf0496768f8..a547d692565 100644
--- a/src/metabase/llm/provider.clj
+++ b/src/metabase/llm/provider.clj
@@ -381,6 +381,28 @@
;; not required: a server started without --api-key takes no key, and a base URL on its own
;; is a complete configuration
:help (deferred-tru "Only needed if you started your server with --api-key.")}]}
+ {:type "ollama"
+ :label (deferred-tru "Ollama")
+ ;; serves whatever the operator pulled, so a new connection takes its model from the catalog
+ ;; that connecting fetches (see [[metabase.metabot.self.ollama/list-models]])
+ :default-model nil
+ :fields [{:key :base-url
+ :normalize strip-trailing-slashes
+ :validate llm.provider.settings/llm-url-problem
+ :label (deferred-tru "API base URL")
+ :type :text
+ :required? true
+ :placeholder "http://ollama.your.company:11434/v1"
+ :help (deferred-tru (str "Your Ollama server''s address, ending in /v1, or https://ollama.com/v1 "
+ "for Ollama Cloud. To reach a server on your private network, set "
+ "MB_LLM_ALLOWED_NETWORKS=allow-private; for one on this machine, "
+ "allow-all."))}
+ {:key :api-key
+ :label (deferred-tru "API key")
+ :type :password
+ ;; not required: a self-hosted server takes none, and a base URL on its own is a complete
+ ;; configuration. Ollama Cloud refuses a generation without one, which connecting finds out.
+ :help (deferred-tru "Required for Ollama Cloud. Leave blank if your server doesn''t require one.")}]}
{:type "metabase"
:label (deferred-tru "Metabase AI service")
:managed? true
@@ -517,6 +539,28 @@
(when-let [problem (and validate (some-> (u/trimmed-string (get config key)) validate))]
(throw (ex-info (str problem) {:status-code 400 :field key}))))
+(defn- field-descriptor
+ "`type-name`'s registry entry for `field-key`."
+ [type-name field-key]
+ (u/find-first-map (:fields (provider-type type-name)) [:key] field-key))
+
+(defn with-field-defaults
+ "Fill in each field's registry `:default` wherever `config` left it blank, and run each field's `:normalize` over
+ the value that results, so a resolved connection carries everything the adapter needs in the shape it needs it.
+ Normalizing here rather than on write covers every source of a value — the stored config, an environment
+ variable, and the default itself — so a base URL entered with a trailing slash cannot double up the `/` when a
+ path is joined onto it."
+ [type-name config]
+ (reduce (fn [config {:keys [key default normalize]}]
+ (let [config (cond-> config
+ (and default (not (u/trimmed-string (get config key))))
+ (assoc key default))]
+ (cond-> config
+ (and normalize (u/trimmed-string (get config key)))
+ (update key normalize))))
+ (or config {})
+ (:fields (provider-type type-name))))
+
(defn- validate-field!
[type-name {:keys [key label required? prefix default options] :as field} config]
(let [value (u/trimmed-string (get config key))]
@@ -534,9 +578,23 @@
(defn- validate-config-field!
"Run [[validate-field!]]'s checks for the single field `field-key` of `type-name` against `config`."
[type-name field-key config]
- (when-let [field (u/find-first-map (:fields (provider-type type-name)) [:key] field-key)]
+ (when-let [field (field-descriptor type-name field-key)]
(validate-field! type-name field config)))
+(defn- typed-value
+ "The value `submitted` carries for `field-key` when the caller really typed one, or nil.
+
+ A blank is the form clearing a field it hid, and a mask is the client echoing back what is already stored.
+ Both forms are checked for the mask: that of a newline-terminated secret (a JSON key file) matches only
+ untrimmed, while one that picked up padding in transit matches only trimmed."
+ [submitted field-key]
+ (let [raw (get submitted field-key)
+ value (u/trimmed-string raw)]
+ (when (and value
+ (not (setting/obfuscated-value? raw))
+ (not (setting/obfuscated-value? value)))
+ value)))
+
(defn- validate-required-any!
"Throw a 400 unless `config` satisfies one of `type-name`'s `:required-any` credential groups — for Google, a
service account key on its own or an OAuth token together with a project ID."
@@ -684,7 +742,12 @@
;; the base URL is the credential here, unlike Azure's: a server started without --api-key takes
;; no key, so the URL alone brings a usable connection into existence
:settings {:base-url {:setting :llm-vllm-api-base-url :credential? true}
- :api-key {:setting :llm-vllm-api-key}}}})
+ :api-key {:setting :llm-vllm-api-key}}}
+ "ollama" {:type "ollama"
+ ;; as for vLLM, the base URL is the credential: a self-hosted server takes no key, so the URL alone
+ ;; brings a usable connection into existence. A key alone, with no address to send it to, does not.
+ :settings {:base-url {:setting :llm-ollama-api-base-url :credential? true}
+ :api-key {:setting :llm-ollama-api-key}}}})
(defn connection-env-vars
"The environment variables that configure a connection of `type-name`, as `{config-field \"MB_LLM_...\"}`.
@@ -716,6 +779,13 @@
{:config {} :vars {}}
settings))
+(defn- applicable-overlay
+ "`overlay` when it describes a connection of `type-name`, and nil otherwise: the variables describe one
+ provider type's fields, so they mean nothing to a connection of another that happens to share the key."
+ [overlay type-name]
+ (when (= type-name (:type overlay))
+ overlay))
+
(defn- env-overlays
"The environment's contribution to each connection key, resolved on every read so editing a variable takes effect
on the next restart without anything having to be migrated or re-saved."
@@ -760,6 +830,15 @@
env-managed? (assoc :env-vars #{(setting/env-var-name :llm-providers)})))]
(into [] (map annotate) (stored-connections))))
+(def ^:private env-base-url-shadowing-types
+ "Types whose base-URL variable shadows the address alone, letting a key typed in the UI go along with it.
+
+ The exception to [[drop-captured-secrets]], held for compatibility: on these types that combination is how
+ an operator routes a connection through a gateway, and an upgrade must not drop the key it relies on.
+ Holding them to the rule needs an upgrade note of its own. A type added from now on starts with its
+ credentials and address tied together."
+ #{"anthropic" "azure" "deepseek" "google" "mistral" "moonshot" "openai" "openrouter" "vllm" "xai" "zai"})
+
(defonce ^:private warned-captured-base-urls
(atom #{}))
@@ -774,7 +853,7 @@
A connection the `MB_LLM_PROVIDERS` JSON supplies is exempt: it is `:source :env`, written by the operator
rather than through the API, so its base URL is as trusted as the variable holding the secret. A type whose base
- URL has no default — Azure, vLLM — is left incomplete, and so unusable, rather than pointed anywhere.
+ URL has no default — Azure, vLLM, Ollama — is left incomplete, and so unusable, rather than pointed anywhere.
Warned about once per value rather than on every read: it is the only trace an operator gets of a base URL their
instance is configured with but is not using."
@@ -792,6 +871,110 @@
conn-key (get (connection-env-vars type) :base-url "the matching base URL variable")))
(update conn :config dissoc :base-url))))
+(defn- base-url-choice
+ "Where `config` points a connection of `type-name`: its base URL, or the type's default when it stores none, as
+ [[with-field-defaults]] resolves it for the adapter.
+
+ A blank address still goes somewhere — the vendor's own — so an overlay replacing it moves the
+ connection just as replacing a typed one does. Normalized, so a trailing slash is not a move."
+ [type-name config]
+ (u/trimmed-string (:base-url (with-field-defaults type-name config))))
+
+(defn- env-moves-base-url?
+ "Whether an environment overlay's base URL sends a connection somewhere other than where it was pointing.
+
+ Never on a type in [[env-base-url-shadowing-types]], whose base-URL variable shadows the address alone."
+ [type-name config env-config]
+ (boolean
+ (and (not (contains? env-base-url-shadowing-types type-name))
+ (u/trimmed-string (:base-url env-config))
+ (not= (base-url-choice type-name config) (base-url-choice type-name env-config)))))
+
+(defn- captured-secret-fields
+ "The secret fields of `type-name` that `env-config` has moved away from, among those `present?` says the
+ connection carries — none unless `env-config` moves the connection at all. Sorted, so the fields are warned about
+ in a stable order and a type with several secrets always names the same one first."
+ [type-name config env-config present?]
+ (when (env-moves-base-url? type-name config env-config)
+ (filterv #(and (present? %) (not (contains? env-config %)))
+ (sort (secret-field-keys type-name)))))
+
+(defonce ^:private warned-captured-secrets
+ ;; hashed, not held: a credential this decided not to use has no business outliving the read
+ (atom #{}))
+
+(defn- drop-captured-secrets
+ "Keep a credential from following a connection the environment has moved, leaving it unusable rather
+ than sent somewhere it was never meant for.
+
+ A secret and the address it reaches have to come from the same place. [[drop-captured-base-url]] holds that
+ line when the environment brings the secret; this holds it when the environment brings the address — a
+ `MB_LLM_OLLAMA_API_BASE_URL` pointing somewhere other than the server an admin typed a key for. A key is
+ entered for the address it will be sent to, so an overlay pointing elsewhere moves the connection whether the
+ address was typed or left at the vendor's default. Only an overlay naming where the connection already points
+ moves nothing, and the base-URL variables of [[env-base-url-shadowing-types]] are exempt.
+
+ A connection the `MB_LLM_PROVIDERS` JSON supplies is the operator's own, so nothing is taken from it. Warned
+ about once per value, like [[drop-captured-base-url]]."
+ [{conn-key :key :keys [type source config] :as conn} env-config]
+ (let [captured (when (= :db source)
+ (captured-secret-fields type config env-config #(u/trimmed-string (get config %))))]
+ (doseq [field captured
+ :let [seen [conn-key field (hash (get config field))]]
+ :when (not (contains? @warned-captured-secrets seen))]
+ (swap! warned-captured-secrets conj seen)
+ (log/warnf (str "Ignoring the stored %s of the %s LLM connection: %s points it at another server, so "
+ "its credentials have to come from the environment as well. Set %s to keep using it.")
+ (name field) conn-key
+ (get (connection-env-vars type) :base-url "the environment")
+ (get (connection-env-vars type) field "the matching environment variable")))
+ (update conn :config #(apply dissoc % captured))))
+
+(defn assert-credentials-not-captured!
+ "Reject a credential the caller has just typed for a connection the environment points at another server.
+
+ The refusal side of [[drop-captured-secrets]]: that rule keeps the credential out of what the connection runs
+ on, which on its own leaves the admin reading a completeness error about the key they entered a moment ago.
+ Both variables are named, since between them they are what the operator has to act on — the one that moved the
+ connection, and the one the credential has to come from instead.
+
+ `submitted` is the caller's own input: a blank clears a field and a mask is the client echoing back what is
+ already stored, so neither is anyone asking for a credential to be sent anywhere. `conn` carries the config the
+ credential would land in, which is what decides whether the connection moved."
+ [{type-name :type :keys [config]} submitted env-config]
+ (when-let [field (first (captured-secret-fields type-name config env-config #(typed-value submitted %)))]
+ (throw (ex-info (tru "{0} points this connection at another server, so its credentials have to come from the environment as well. Set {1} to keep using it."
+ (get (connection-env-vars type-name) :base-url "The environment")
+ (get (connection-env-vars type-name) field "the matching environment variable"))
+ {:status-code 400
+ :api-error true
+ :error-code :llm-credentials-must-come-from-env
+ :field field}))))
+
+(defn env-overlay-config
+ "What the environment supplies for a connection of `type-name` stored under `conn-key`, or nil where it
+ supplies nothing for it.
+
+ For a writer deciding about a connection that does not exist yet, which cannot read the overlay off a
+ stored one — see [[applicable-overlay]] for which overlay reaches it."
+ [conn-key type-name]
+ (:config (applicable-overlay (get (env-overlays) conn-key) type-name)))
+
+(defn effective-config
+ "What the stored connection `conn` will run on once `env-config` is layered over it.
+
+ Not simply the two merged: a secret the environment has moved away from is no more this connection's
+ to send than it was to keep — see [[drop-captured-secrets]]. An edit has to be judged, probed and
+ answered for on this rather than on what it merged, or a write sends a credential where a read of the
+ same connection would not.
+
+ Only that half of what [[connections]] does. Dropping the stored base URL as well would take a base
+ URL the caller has just supplied out of the comparison that decides whether they moved the connection,
+ which is the one thing that must see it."
+ [conn env-config]
+ (merge (:config (drop-captured-secrets (assoc conn :source :db) env-config))
+ env-config))
+
(defn connections
"Every connection this instance can use, in admin-facing order.
@@ -802,8 +985,8 @@
config keys the environment owns, and `:env-vars` the variables supplying them, so the form can disable exactly
those inputs.
- The one field that does not simply stay editable is a stored base URL the environment's secret would travel to:
- see [[drop-captured-base-url]].
+ The exception is a secret and the address it reaches arriving from different places — see
+ [[drop-captured-base-url]] and [[drop-captured-secrets]].
A standalone `:env` connection is synthesized only when a variable marked `:credential?` is set — credentials are
what bring a connection into existence; a base URL alone shadows but does not create. The managed connection is
@@ -811,15 +994,16 @@
[]
(let [overlays (env-overlays)
overlaid (mapv (fn [{conn-key :key :keys [type] :as conn}]
- (let [{env-config :config vars :vars :as overlay} (get overlays conn-key)]
- ;; only a same-typed overlay applies: the fields describe this provider type's config
- (if (and overlay (= type (:type overlay)))
- (-> conn
- (drop-captured-base-url env-config)
- (update :config merge env-config)
- (update :env-vars (fnil into (sorted-set)) (vals vars))
- (assoc :env-fields (set (keys env-config))))
- conn)))
+ (if-let [{env-config :config vars :vars}
+ (applicable-overlay (get overlays conn-key) type)]
+ (-> conn
+ ;; first: it reads the stored base URL, which the next one removes
+ (drop-captured-secrets env-config)
+ (drop-captured-base-url env-config)
+ (update :config merge env-config)
+ (update :env-vars (fnil into (sorted-set)) (vals vars))
+ (assoc :env-fields (set (keys env-config))))
+ conn))
(annotated-stored-connections))
taken (into #{} (map :key) overlaid)
standalone (into []
@@ -858,7 +1042,7 @@
(defn validate-changed-connections!
"Run the per-field `:validate` hooks over the fields `conns` changes. This is the
- [[metabase.llm.settings/llm-providers]] setter, so trusted provisioning such as `config.yml` also validates base URLs.
+ [[metabase.llm.provider.settings/llm-providers]] setter, so trusted provisioning such as `config.yml` also validates base URLs.
Direct generic settings API writes are forbidden by that setter.
Only the `:validate` hooks: required fields, prefixes and options are the connection API's business, and demanding
@@ -962,23 +1146,6 @@
(str/replace-first model-ref (str managed-connection-key "/") "")
model-ref))
-(defn with-field-defaults
- "Fill in each field's registry `:default` wherever `config` left it blank, and run each field's `:normalize` over
- the value that results, so a resolved connection carries everything the adapter needs in the shape it needs it.
- Normalizing here rather than on write covers every source of a value — the stored config, an environment
- variable, and the default itself — so a base URL entered with a trailing slash cannot double up the `/` when a
- path is joined onto it."
- [type-name config]
- (reduce (fn [config {:keys [key default normalize]}]
- (let [config (cond-> config
- (and default (not (u/trimmed-string (get config key))))
- (assoc key default))]
- (cond-> config
- (and normalize (u/trimmed-string (get config key)))
- (update key normalize))))
- (or config {})
- (:fields (provider-type type-name))))
-
(defn resolve-model-ref
"Resolve a `connection-key/model` string against the configured connections.
@@ -1045,17 +1212,18 @@
carried-secrets (filter #(u/trimmed-string (get new-config %)) secret-keys)
env-fields (set env-fields)
fresh-secret? (fn [field]
- (let [value (u/trimmed-string (get submitted-config field))]
- (and (not (contains? env-fields field))
- value
- (not (setting/obfuscated-value? value)))))
+ (and (not (contains? env-fields field))
+ (typed-value submitted-config field)))
missing-secrets (remove fresh-secret? carried-secrets)]
(when (and (not= (:base-url old-config) (:base-url new-config))
(seq missing-secrets))
(let [env-secret? (some env-fields missing-secrets)]
(throw (ex-info (cond
env-secret?
- (tru "This connection''s credentials come from environment variables. Change its base URL there too.")
+ ;; the message names the variable because the operator can change only that. A create has
+ ;; no base URL of its own to change.
+ (tru "This connection''s credentials come from environment variables, so its base URL has to as well. Set {0}."
+ (get (connection-env-vars type-name) :base-url "the matching base URL variable"))
legacy-setting?
(tru "Use the provider connection settings to change the base URL and enter the credentials again.")
@@ -1076,7 +1244,9 @@
it will be sent. The connection settings submit the whole connection, base URL included, so they are where a
connection on a custom URL takes its credentials.
- A base URL the environment supplies needs no such treatment: the operator chose it."
+ A base URL the environment supplies is not this check's to judge: [[assert-credentials-not-captured!]]
+ refuses a credential typed for a connection the environment points elsewhere, and lets it through on a
+ type in [[env-base-url-shadowing-types]]."
[type-name field {:keys [config env-fields]}]
(when (contains? (secret-field-keys type-name) field)
(let [base-url (:base-url (with-field-defaults type-name config))]
@@ -1130,7 +1300,14 @@
(assert-base-url-change-authorized! group-type current-config new-config {field value}
(:env-fields live) {:legacy-setting? true})))
(when value
- (assert-credential-write-authorized! group-type field live))))
+ (assert-credential-write-authorized! group-type field live)
+ ;; with nothing stored yet, the connection this write creates, which runs on the default base URL
+ ;; until something moves it
+ (assert-credentials-not-captured! (if idx
+ (nth stored idx)
+ {:key conn-key :type group-type :config {}})
+ {field value}
+ (env-overlay-config conn-key group-type)))))
(when value
(validate-config-field! group-type field {field value}))
(cond
diff --git a/src/metabase/llm/settings.clj b/src/metabase/llm/settings.clj
index 2bbbd470817..b03f984b34a 100644
--- a/src/metabase/llm/settings.clj
+++ b/src/metabase/llm/settings.clj
@@ -411,6 +411,35 @@
:visibility :settings-manager
:export? false)
+;;; -------------------------------------------------- Ollama ---------------------------------------------------
+
+(defsetting llm-ollama-api-base-url
+ (deferred-tru "The base URL of your Ollama server''s OpenAI-compatible API, e.g. `http://localhost:11434/v1`, or `https://ollama.com/v1` for Ollama Cloud.")
+ :encryption :when-encryption-key-set
+ :visibility :settings-manager
+ :export? false
+ :getter (connection-field-getter :llm-ollama-api-base-url)
+ :setter (connection-field-setter :llm-ollama-api-base-url)
+ :doc "Backed by the ollama connection in the admin AI settings provider list: reads and writes go through the llm-providers connection list. A value set by this environment variable shadows that connection's base URL.")
+
+(defsetting llm-ollama-api-key
+ (deferred-tru (str "The API key for Ollama Cloud, with MB_LLM_OLLAMA_API_BASE_URL set to https://ollama.com/v1. "
+ "For self-hosted servers, only needed behind an authenticated proxy."))
+ :sensitive? true
+ :visibility :settings-manager
+ :export? false
+ :getter (connection-field-getter :llm-ollama-api-key)
+ :setter (connection-field-setter :llm-ollama-api-key)
+ :doc "Backed by the ollama connection in the admin AI settings provider list: reads and writes go through the llm-providers connection list, and a value set by this environment variable shadows this one field of that connection.")
+
+(defsetting llm-ollama-request-timeout-ms
+ (deferred-tru "Socket timeout in milliseconds for requests to your Ollama server.")
+ ;; Self-hosted TTFT is bounded by the operator's hardware, as it is for vLLM.
+ :type :integer
+ :default 300000
+ :visibility :settings-manager
+ :export? false)
+
;;; The per-provider credential settings above are read-only at runtime: they configure a connection only when set
;;; by an environment variable, which [[metabase.llm.provider/connections]] resolves on every read. Editing one in
;;; the app DB would not reach the connection serving requests, so a write is rejected rather than silently ignored.
diff --git a/src/metabase/metabot/self.clj b/src/metabase/metabot/self.clj
index 9737d7d034d..8c9e56bdc3d 100644
--- a/src/metabase/metabot/self.clj
+++ b/src/metabase/metabot/self.clj
@@ -45,7 +45,7 @@
This is the allow-list [[list-models]] intersects with the provider's live catalog, so a model listed here is
available only if the connection's credentials can actually reach it. Returns nil for the provider types that have
- no allow-list: `azure`, whose model is the deployment name the admin gives it, `vllm`, which serves whatever the
+ no allow-list: `azure`, whose model is the deployment name the admin gives it, `vllm` and `ollama`, which serve whatever the
operator loaded, and `google` and `metabase`, whose catalogs are fixed in [[metabase.llm.provider]] instead."
[provider]
(when-let [models (registry/optional provider :supported-models)]
diff --git a/src/metabase/metabot/self/adapter.clj b/src/metabase/metabot/self/adapter.clj
index a87f02bac84..a77b0d2ae64 100644
--- a/src/metabase/metabot/self/adapter.clj
+++ b/src/metabase/metabot/self/adapter.clj
@@ -17,7 +17,9 @@
[metabase.util.log :as log]
[metabase.util.malli :as mu]
[metabase.util.malli.registry :as mr]
- [metabase.util.o11y :refer [with-span]]))
+ [metabase.util.o11y :refer [with-span]])
+ (:import
+ (java.io IOException)))
(set! *warn-on-reflection* true)
@@ -299,8 +301,83 @@
:display_name (or (get entry catalog-name-key)
(get-in supported-models [id :display-name]))})))}))
+;;; ------------------------------------------- Preflight and its budget -----------------------------------------
+;;;
+;;; The self-hosted adapters (vLLM, Ollama) exercise the agent loop's contract at configuration time, because
+;;; the operator serves whatever they loaded and no allow-list can vouch for it. What they probe *with* is a
+;;; statement about Metabot, not about a provider, so it lives here rather than twice.
+
+(def probe-tool
+ "The tool a preflight probe offers. Trivial on purpose: a model that cannot call this cannot drive
+ Metabot, and one that can has told us the template wires tool calling up at all."
+ {:type "function"
+ :function {:name "record_table_name"
+ :description "Record the name of the table the user mentioned."
+ :parameters {:type "object"
+ :properties {:table_name {:type "string"
+ :description "The table name the user mentioned."}}
+ :required ["table_name"]
+ :additionalProperties false}}})
+
+(def probe-messages
+ "The prompt a preflight probe sends — one line, so nothing but the model's own verbosity can exhaust
+ [[probe-max-tokens]]."
+ [{:role "user" :content "Record the table name: orders"}])
+
+(def probe-max-tokens
+ "Generation ceiling for a preflight probe. High enough to clear a reasoning model's thinking, which is
+ billed against it: a probe that stops at `length` before the tool call looks identical to a server that
+ will not call tools at all."
+ 2048)
+
+(def probe-timeout-ceiling-ms
+ "Upper bound on a single preflight probe, which blocks the admin behind a spinner. An operator who sets
+ their provider's request timeout lower than this keeps their own value."
+ 120000)
+
+(def min-context-window-tokens
+ "Smallest context window a connection may be saved on. A product floor, not a measurement: Metabot's
+ tools and system prompt run to several thousand tokens before any history, and a self-hosted server is
+ the only place the window is small enough to matter — the hosted providers all exceed it.
+
+ Shared because it is a statement about Metabot, not about a provider: the adapters that can observe a
+ window (vLLM from its catalog, Ollama from the loaded model) must not drift apart on the number."
+ 16384)
+
+(def forced-tool-call-token-floor
+ "Smallest `max_tokens` a forced tool call is given, whatever the caller asked for — below it a reasoning
+ model spends the budget thinking and emits no call. Equal to [[probe-max-tokens]], which preflight
+ already proves the model can clear."
+ probe-max-tokens)
+
+(def reasoning-model-token-floor
+ "Smallest `max_tokens` a request on a reasoning model gets. Chat Completions bills thinking, answer and
+ tool call against one budget."
+ 16384)
+
+(def default-temperature
+ "Sampling temperature for a caller that supplies none, on a self-hosted server. The hosted providers pick
+ something sane server-side; a server the operator runs does not — vLLM defaults to 1.0 and an Ollama
+ model inherits whatever its Modelfile says — and both are wrong for tool calling and SQL generation."
+ 0.3)
+
;;; ------------------------------------------------- Streaming --------------------------------------------------
+(defn io-guarded
+ "Wrap a stream `reducible` so an `IOException` raised while *consuming* it surfaces as `(ex-fn e)` rather
+ than raw. An adapter's own `try` covers only establishing the request; the body is read long after that
+ returns.
+
+ Goes inside `core/reducible-with-api-errors`, never outside: an adapter's IO translation tags `:api-error
+ true`, which `core/rethrow-api-error!` rethrows unchanged, so this wins for IO."
+ [reducible ex-fn]
+ (reify clojure.lang.IReduceInit
+ (reduce [_ rf init]
+ (try
+ (.reduce ^clojure.lang.IReduceInit reducible rf init)
+ (catch IOException e
+ (throw (ex-fn e)))))))
+
(mu/defn stream!
"Open a provider's streaming request and return a reducible over its raw SSE events.
diff --git a/src/metabase/metabot/self/ollama.clj b/src/metabase/metabot/self/ollama.clj
new file mode 100644
index 00000000000..606160ef826
--- /dev/null
+++ b/src/metabase/metabot/self/ollama.clj
@@ -0,0 +1,647 @@
+(ns metabase.metabot.self.ollama
+ "Ollama adapter, for a self-hosted Ollama server or Ollama Cloud.
+
+ A sibling of [[metabase.metabot.self.vllm]] rather than a layer over it — same Chat Completions
+ transport, but Ollama has no server flags to point an admin at, so every diagnosis differs.
+
+ https://docs.ollama.com/api/openai-compatibility"
+ (:require
+ [clojure.set :as set]
+ [clojure.string :as str]
+ [metabase.llm.settings :as llm]
+ [metabase.metabot.self.adapter :as adapter]
+ [metabase.metabot.self.core :as core]
+ [metabase.metabot.self.ollama.capabilities :as caps]
+ [metabase.metabot.self.ollama.connection :as conn]
+ [metabase.metabot.self.ollama.forced-calls :as forced]
+ [metabase.metabot.self.openai.chat-completions :as chat-completions]
+ [metabase.util :as u]
+ [metabase.util.i18n :refer [tru]]
+ [metabase.util.json :as json]
+ [metabase.util.log :as log]
+ [metabase.util.malli :as mu])
+ (:import
+ (com.fasterxml.jackson.core JsonProcessingException)
+ (java.io IOException)
+ (java.net SocketTimeoutException)))
+
+(set! *warn-on-reflection* true)
+
+(defn- missing-model-ex []
+ (ex-info (tru "No Ollama model is set")
+ {:api-error true
+ :error-code :model-missing}))
+
+(def ^:private provider
+ "Ollama's descriptor. No `:supports-ai-proxy?`, so [[adapter/request!]] refuses a proxied request: the
+ proxy fronts hosted providers, and Ollama, self-hosted or Cloud, is reached directly."
+ (adapter/provider
+ {:slug "ollama"
+ :display-name "Ollama"
+ :auth conn/auth
+ ;; no `:error-fallback`: that key carries an adapter's own `HTTP {0}` msgid where one already has
+ ;; shipped translations. Ollama's has none, so it takes the shared msgid, which renders the same English.
+ :errors {400 #(tru "Ollama rejected the request — usually a model that cannot produce the requested tool call or JSON schema")
+ 401 #(tru "Ollama rejected the API key")
+ 404 #(tru "Ollama API endpoint was not found — the base URL should end in /v1, and the model must already be available")
+ 429 #(tru "Ollama is rate limiting this instance — wait and retry, or reduce concurrent Metabot use")
+ 500 #(tru "Ollama returned an internal server error")}}))
+
+(defn- inference-timeouts
+ "Timeouts for a generation request."
+ []
+ {:socket-timeout (llm/llm-ollama-request-timeout-ms)
+ :connection-timeout (llm/llm-connection-timeout-ms)})
+
+(defn- control-timeouts
+ "Timeouts for the non-generating `/models` call, on the shared (much shorter) request budget."
+ []
+ {:socket-timeout (llm/llm-request-timeout-ms)
+ :connection-timeout (llm/llm-connection-timeout-ms)})
+
+(defn- probe-timeouts
+ "Timeouts for a preflight probe, capped at [[adapter/probe-timeout-ceiling-ms]]."
+ []
+ {:socket-timeout (min (llm/llm-ollama-request-timeout-ms) adapter/probe-timeout-ceiling-ms)
+ :connection-timeout (llm/llm-connection-timeout-ms)})
+
+;;; ----------------------------------------------- Transport errors ---------------------------------------------
+
+(defn- unreachable-ex
+ "The Ollama error for a non-timeout transport failure. `extra` is the caller's own ex-data tags."
+ [^IOException e url extra]
+ (ex-info (tru "Could not reach Ollama at {0}. Check that it is reachable from the Metabase server."
+ (str url))
+ (merge {:api-error true :error-code :ollama-unreachable} extra)
+ e))
+
+(defn- list-models-io-ex
+ "Transport failure while fetching the catalog — the request behind the Connect button. Tagged 400
+ so a mistyped base URL surfaces this message rather than a 500."
+ [^IOException e url]
+ (if (instance? SocketTimeoutException e)
+ (ex-info (tru "The Ollama server at {0} did not respond within {1}ms. Check that it is running and not loading a model."
+ (str url) (str (llm/llm-request-timeout-ms)))
+ {:api-error true
+ :status-code 400
+ :error-code :ollama-timeout}
+ e)
+ (unreachable-ex e url {:status-code 400})))
+
+;;; ------------------------------------------------ Model listing -----------------------------------------------
+
+(defn- list-all-models
+ "Fetch the pulled model catalog. Doubles as the credential round-trip behind the Connect button,
+ and fails closed on a 2xx whose body is not a catalog — a mistyped base URL is the likeliest cause.
+
+ Not [[adapter/fetch-catalog]]: that renders a 2xx-but-not-a-catalog failure with the shared message,
+ and the address is the one thing an Ollama admin types, so naming it is the whole diagnosis."
+ [{:keys [credentials] :as req}]
+ ;; the URL off the request credentials, not the setting: a connect verifies them before they are saved
+ (let [url (conn/base-url credentials)
+ detail (tru "Check that {0} is an Ollama server''s OpenAI-compatible API — the base URL should end in /v1."
+ (str url))]
+ (try
+ (let [res (adapter/request! provider
+ (assoc req :method :get :path "/models" :as :json)
+ (control-timeouts))]
+ (chat-completions/models-catalog "Ollama" res {:detail detail}))
+ ;; Ordered ahead of the `IOException` catch, which Jackson's parse error is one of: a 2xx whose
+ ;; body is not JSON means the server answered, so the address is wrong rather than unreachable.
+ (catch JsonProcessingException e
+ (throw (chat-completions/malformed-catalog-ex "Ollama" detail e)))
+ ;; Ordered ahead of the generic catch, which a non-2xx still reaches as an `ExceptionInfo`.
+ (catch IOException e
+ (throw (list-models-io-ex e url)))
+ (catch Exception e
+ (adapter/rethrow! provider e)))))
+
+;;; -------------------------------------------------- Preflight -------------------------------------------------
+
+(defn- preflight-ex
+ "A preflight failure, tagged so `metabase.metabot.api` surfaces the message verbatim, not as a 500.
+
+ `data` may carry `::server-wide?`, for a failure every model on the server would hit as well, which
+ ends [[preflight!]]'s search rather than moving it on to the next model — see
+ [[model-specific-failure?]]."
+ ([msg] (preflight-ex msg nil))
+ ([msg data]
+ (ex-info msg (merge {:api-error true
+ :status-code 400
+ :error-code :ollama-preflight-failed}
+ data))))
+
+(defn- probe-chat!
+ "One non-streaming Chat Completions turn, returning the first choice. `body` carries what differs
+ between probes — the messages, and either a tool or a `response_format`. `finish_reason` comes back
+ with it because truncation and a model that cannot call tools both produce empty `tool_calls`."
+ [req model body]
+ (let [res (adapter/request! provider
+ (assoc req
+ :method :post
+ :path "/chat/completions"
+ :as :json
+ :body (json/encode (merge {:model model
+ :temperature 0
+ :max_tokens adapter/probe-max-tokens}
+ body)))
+ (probe-timeouts))]
+ (get-in res [:body :choices 0])))
+
+(defn- check-tool-calling!
+ "Check that the model can call the tool it is offered, with the arguments that tool declares. Ollama
+ drives tool calling from the model's own template, so the fix is always a different model.
+
+ Whether it thought along the way is not recorded — `/api/show` answers that per model, for every
+ model rather than only the probed one. Thinking is still read here, to tell \"spent the budget
+ thinking\" apart from \"generated too much\" when nothing came back."
+ [req model]
+ (let [{:keys [message finish_reason]} (probe-chat! req model {:messages adapter/probe-messages
+ :tools [adapter/probe-tool]})
+ content (str (:content message))
+ ;; `reasoning` is the OpenAI-compatible spelling; `reasoning_content` is the older one some
+ ;; builds still emit.
+ reasoning (str (or (:reasoning message) (:reasoning_content message)))
+ tool-calls (:tool_calls message)
+ truncated? (= "length" finish_reason)]
+ (cond
+ (str/includes? content "<think>")
+ (throw (preflight-ex
+ (tru "{0} streamed its reasoning as chat text, which would appear inside Metabot''s answers. Pull a build of this model whose template separates thinking from the answer."
+ (str model))))
+
+ (seq tool-calls)
+ (let [{:keys [function]} (first tool-calls)
+ {offered :name {required :required} :parameters} (:function adapter/probe-tool)
+ parsed (try (json/decode+kw (str (:arguments function)))
+ (catch Exception _ nil))]
+ (cond
+ (not= offered (:name function))
+ (throw (preflight-ex
+ (tru "{0} called ''{1}'' instead of the one tool it was offered. The agent loop runs only the tools it registers and drops anything else without a word, so a model that invents names cannot drive Metabot — pick a larger or more capable one."
+ (str model) (str (:name function)))))
+
+ (not (map? parsed))
+ (throw (preflight-ex
+ (if truncated?
+ (tru "{0} reached the {1} token connection-test ceiling before completing a tool call. A model that generates this much before calling a tool is too slow to drive Metabot."
+ (str model) (str adapter/probe-max-tokens))
+ (tru "{0} returned a tool call whose arguments are not valid JSON. Pull a larger or more capable model — Metabot needs reliable tool calling."
+ (str model)))))
+
+ :else
+ (when-let [missing (seq (remove #(u/trimmed-string (get parsed (keyword %))) required))]
+ (throw (preflight-ex
+ (tru "{0} called the tool without its required {1} argument. Metabot validates every tool call against that tool''s schema — pick a larger or more capable model."
+ (str model) (str/join ", " missing)))))))
+
+ (and truncated? (not (str/blank? reasoning)))
+ (throw (preflight-ex
+ (tru "{0} spent the entire {1} token connection-test budget reasoning without calling a tool. A model that thinks this long about a trivial prompt is too slow to drive Metabot."
+ (str model) (str adapter/probe-max-tokens))))
+
+ truncated?
+ (throw (preflight-ex
+ (tru "{0} reached the {1} token connection-test ceiling before completing a tool call. A model that generates this much before calling a tool is too slow to drive Metabot."
+ (str model) (str adapter/probe-max-tokens))))
+
+ :else
+ (throw (preflight-ex
+ (tru "{0} answered with text instead of calling a tool. Metabot needs a model that supports tool calling — pick a different one."
+ (str model)))))))
+
+(defn- check-structured-output!
+ "Check that the connection can actually deliver structured output, through the mechanism it will
+ really use — [[forced/probe-body]] decides which that is, and [[forced/probe-verdict]] reads the
+ answer.
+
+ The remedy is what differs between a self-hosted model and one Ollama Cloud serves, so only the
+ message branches here. Self-hosted can fail on the *server* rather than the model, because an Ollama
+ too old to read `response_format` ignores it and the model answers in prose. Cloud cannot fail that
+ way: nothing there was ever going to enforce it, so a model that will not take the instruction has
+ to be caught at connect — nothing downstream can repair it."
+ [{:keys [credentials] :as req} model]
+ (let [cloud? (conn/served-by-cloud? credentials model)
+ choice (probe-chat! req model (forced/probe-body cloud?))]
+ (when-let [verdict (forced/probe-verdict cloud? choice)]
+ (throw (preflight-ex
+ (case verdict
+ :truncated
+ (tru "{0} reached the {1} token connection-test ceiling before completing the structured answer it was asked for. Metabot needs structured output for conversation titles and SQL generation."
+ (str model) (str adapter/probe-max-tokens))
+
+ :not-honored
+ (if cloud?
+ (tru "{0} would not answer Ollama Cloud''s structured-output request with a tool call. Cloud cannot force one, so Metabot needs a model that does it when asked — pick a larger or more capable one."
+ (str model))
+ (tru "{0} did not answer with JSON matching the schema it was given. Metabot needs structured output for conversation titles and SQL generation — upgrade Ollama to a version that supports `response_format`, or pull a larger or more capable model."
+ (str model)))))))))
+
+(defn- no-models-ex []
+ (preflight-ex (tru "Ollama is reachable but is offering no models.")))
+
+(defn- probe-target
+ "The catalog entry [[preflight!]] will probe.
+
+ A requested model is the only acceptable target: falling back to another pulled model would pass
+ every check and then persist a provider string naming a model the server does not have. The
+ fallback is for the connect path, which supplies no model because the pulled name is knowable only
+ from this catalog.
+
+ That fallback takes the first *chat-capable* entry rather than the first entry. Ollama lists models
+ newest-first, so a newest pull that is an embedding model would otherwise fail a connect against a
+ server with a perfectly good chat model on it — with no way out, since the form hides the model
+ picker for a type whose catalog is not fixed. A server that rules every model out is told so,
+ rather than being handed one to probe and failing on whatever that model happens to do; a server
+ that reports nothing rules nothing out, so it still gets its first entry.
+
+ `entries` is the whole catalog as [[tag-chat-capable]] left it, not the subset [[list-models]]
+ offers, so that a model requested by name is answered about rather than reported missing."
+ [entries requested-model]
+ (if requested-model
+ (let [entry (or (u/seek #(= requested-model (:id %)) entries)
+ (throw (if (seq entries)
+ (preflight-ex (tru "{0} is not available. Models on offer: {1}."
+ (str requested-model) (str/join ", " (map :id entries))))
+ (no-models-ex))))]
+ ;; `::chat?` needs both `completion` and `tools`. Ollama gives tool calling only through the completion
+ ;; template, so a model that reports `tools` also reports `completion`. Thus, a model that fails this
+ ;; check cannot call tools: an embedding model has neither, and some chat models have no `tools`.
+ (when-not (::chat? entry)
+ (throw (preflight-ex (tru "{0} can''t call tools, which Metabot needs. Pick a model that supports tool calling."
+ (str requested-model)))))
+ entry)
+ (or (u/seek ::chat? entries)
+ (throw (if (seq entries)
+ (preflight-ex (tru "None of the models on this server can chat and call tools. Pull one that can, then connect again."))
+ (no-models-ex))))))
+
+(defn- run-probe!
+ "Run one contract probe, `check!`, against `model`, turning a timeout into its own diagnosis and any
+ other failure into the provider's."
+ [check! req model]
+ (try
+ (check! req model)
+ (catch SocketTimeoutException _
+ (throw (preflight-ex
+ (tru "Ollama did not answer the connection test within {0}ms. On a self-hosted server the first request also loads the model into memory — if it is large, retry once it is warm, otherwise it is too slow to drive Metabot."
+ (str (:socket-timeout (probe-timeouts))))
+ ;; the next model would load just as slowly
+ {::server-wide? true})))
+ (catch Exception e
+ (adapter/rethrow! provider e))))
+
+(defn- check-native-api!
+ "Warn when Ollama's own API is out of reach, typically behind a proxy that forwards only `/v1`.
+
+ Nothing on it is required, so connecting still succeeds, but the `/api/show` and `/api/ps` lookups
+ then fail quietly on every request: thinking models get the smaller token budget and the context-window
+ check is skipped. `/api/version` because it takes nothing and ollama.com serves it too. The message is
+ logged, not the exception, whose ex-data can carry the response."
+ [credentials]
+ (try
+ (adapter/request! conn/native-provider
+ {:credentials credentials :method :get :path "/api/version" :as :json}
+ (control-timeouts))
+ (catch Exception e
+ (log/warnf (str "Ollama at %s did not answer /api/version (%s). A proxy in front of it has to forward /api/ "
+ "as well as /v1/, or thinking models get the smaller token budget and the context window "
+ "check is skipped.")
+ (conn/base-url credentials) (ex-message e)))))
+
+(defn- loaded-context-length
+ "The context window Ollama loaded `model` with, from `/api/ps`, or nil when it would not say.
+
+ [[conn/native-provider]], because `/api/ps` is on Ollama's own API rather than the OpenAI-compatible
+ surface the adapter's own descriptor authenticates."
+ [credentials model]
+ (try
+ (let [res (adapter/request! conn/native-provider
+ {:credentials credentials :method :get :path "/api/ps" :as :json}
+ (control-timeouts))]
+ (some (fn [{:keys [name context_length] :as entry}]
+ ;; `/api/ps` names a loaded model under both keys, and they agree except where a
+ ;; Modelfile gave it another name
+ (when (and (or (= model (:model entry)) (= model name))
+ (pos-int? context_length))
+ context_length))
+ (get-in res [:body :models])))
+ (catch Exception e
+ (log/debug e "Ollama did not report a context window" {:model model})
+ nil)))
+
+(defn- check-context-budget!
+ "Fail a connection whose model runs with a window too small to hold Metabot's prompt.
+
+ No field in a request can make the window larger: Ollama's OpenAI-compatible surface has no `num_ctx`
+ field. A prompt that is too long does not cause an error. Ollama makes it shorter in two stages. First,
+ it removes the oldest messages until the prompt fits, but it keeps the system messages and the last
+ message. If the prompt is still too long, context shift (on by default) keeps the first few tokens and
+ the end, and removes the middle. The tools and the system prompt are in the middle, so Metabot would
+ run without its instructions and without tools it can name.
+
+ The probes cannot show this. They are one-line prompts that fit in any window, so the truncation
+ they would reveal is of the *answer*; this truncates the *question*.
+
+ The remedy differs by host. On a self-hosted server the window is server configuration. Cloud already
+ runs a model at its largest window, so a small window is the limit of the model, and the admin can only
+ pick a different one."
+ [credentials model]
+ (when-let [window (loaded-context-length credentials model)]
+ (when (< window adapter/min-context-window-tokens)
+ (throw (preflight-ex
+ (if (conn/served-by-cloud? credentials model)
+ (tru "{0} runs with a {1} token context window, which is too small for Metabot. Pick a model with a context window of at least {2} tokens."
+ (str model) (str window) (str adapter/min-context-window-tokens))
+ (tru "{0} runs with a {1} token context window, which is too small for Metabot. Configure Ollama to load the model with a context window of at least {2} tokens."
+ (str model) (str window) (str adapter/min-context-window-tokens))))))))
+
+(def ^:private max-fallback-candidates
+ "How many chat models the connect path tries before giving up. Each costs a model load and two
+ generations, so the bound is what keeps a server with a long catalog from stalling the connect."
+ 3)
+
+(defn- preflight-model!
+ "Run every check against `model`, returning it, and throw on the first failure.
+
+ Cheapest first, each where it can first be answered. Tool calling leads: structured output only means
+ anything once it works. The context window comes next — it is a lookup on `/api/ps`, but one that
+ can only answer once a probe has loaded the model — so a window too small costs one generation
+ rather than two. Which structured-output probe runs depends on whether Ollama Cloud serves the model,
+ because the mechanism does — see [[metabase.metabot.self.ollama.forced-calls/probe-body]].
+
+ Sequential: concurrency would not help — Ollama serializes generation per model unless
+ `OLLAMA_NUM_PARALLEL` is raised, and a losing probe cannot be called off: `future-cancel` interrupts,
+ and a blocking socket read ignores interrupts."
+ [{:keys [credentials] :as req} model]
+ (run-probe! check-tool-calling! req model)
+ (check-context-budget! credentials model)
+ (run-probe! check-structured-output! req model)
+ model)
+
+(defn- model-specific-failure?
+ "Whether `e`, thrown while preflighting one model, is about that model alone, so that
+ [[preflight!]]'s search moves on to the next one.
+
+ Two HTTP statuses are, on Ollama: 500 for a model too large to load (\"model requires more system
+ memory\"), and 400 for one whose template has no tool support, which only a probe finds out when
+ `/api/show` will not say."
+ [e]
+ (let [{:keys [error-code status] :as data} (ex-data e)]
+ (case error-code
+ :ollama-preflight-failed (not (::server-wide? data))
+ :provider-api-error (contains? #{400 500} status)
+ false)))
+
+(defn- preflight!
+ "Exercise the agent loop's contract against the model that will actually serve it, returning that
+ model's id. The connect path must adopt exactly this model rather than re-deriving it from the
+ listing, which agrees only while nothing reorders the catalog.
+
+ A model asked for by name is the only one probed. The connect path names none, and the form offers
+ no picker for Ollama, so a failing model there would leave the admin no way forward: it moves on to
+ the next chat model, newest first, up to [[max-fallback-candidates]] of them. A failure that is
+ not the model's own — see [[model-specific-failure?]] — ends the search."
+ [req entries requested-model]
+ (check-native-api! (:credentials req))
+ (if requested-model
+ (preflight-model! req (:id (probe-target entries requested-model)))
+ (let [candidates (->> entries (filter ::chat?) (map :id) (take max-fallback-candidates))
+ errors (volatile! [])]
+ ;; nothing to try: [[probe-target]] says why
+ (when (empty? candidates)
+ (probe-target entries nil))
+ (or (some (fn [model]
+ (try
+ (preflight-model! req model)
+ (catch clojure.lang.ExceptionInfo e
+ (when-not (model-specific-failure? e)
+ (throw e))
+ (vswap! errors conj e)
+ nil)))
+ candidates)
+ (let [[newest & others] @errors]
+ (throw (if others
+ (preflight-ex
+ (tru "None of the models tried can drive Metabot ({0}). {1}"
+ (str/join ", " candidates)
+ ;; the newest model's reason, the one an admin is likeliest to act on
+ (ex-message newest)))
+ newest)))))))
+
+(defn- tag-chat-capable
+ "`entries` with `::chat?` on each, saying whether Ollama offers that model for chat at all.
+
+ Tagged rather than filtered because the two readers want different views of the same answer: the
+ picker wants only the chat models, [[probe-target]] wants the whole catalog so it can say *why* a
+ model named by hand is no good. Asking once is what keeps this from being two passes."
+ [credentials entries]
+ (let [capable (caps/chat-capable-ids credentials (map :id entries))]
+ (mapv #(assoc % ::chat? (contains? capable (:id %))) entries)))
+
+(mu/defn list-models :- adapter/ModelListing
+ "The models the server has pulled that Metabot could run on, so the admin's picker only offers
+ models a connection can actually be saved against — see [[tag-chat-capable]].
+
+ `:probe?` also runs [[preflight!]] and returns what it learned as `:connection-info` for the connect
+ path to store. Reserved for connect and edit: probing on every listing would stall the model picker
+ behind a full model load. `:proposed-model` is ignored: an edit falls back over the chat models as a
+ connect does, so a stored model that no longer passes cannot block it."
+ ([] (list-models {}))
+ ([{:keys [credentials ai-proxy? model probe?]} :- adapter/ListOpts]
+ (let [req {:credentials credentials :ai-proxy? ai-proxy?}
+ catalog (tag-chat-capable credentials (list-all-models req))
+ entries (filterv ::chat? catalog)
+ probed (when probe?
+ (preflight! req catalog model))
+ models (mapv (fn [{:keys [id] :as entry}]
+ {:id id :display_name (or (:name entry) id)})
+ entries)]
+ (merge {:models models}
+ (when probed
+ {:connection-info {:probed-model probed}})))))
+
+;;; --------------------------------------------------- Requests -------------------------------------------------
+
+(defn- forced-plan
+ "How `opts`' forced tool call is expressed, decided by whether Ollama Cloud serves the requested model —
+ see [[conn/served-by-cloud?]]. Per request rather than per connection: Metabot and the mini model can be
+ on the same connection, and only one of them a Cloud model."
+ [{:keys [credentials model] :as opts}]
+ (forced/plan opts (conn/served-by-cloud? credentials model)))
+
+(defn- reasoning-message
+ "Builds the assistant message that sends one block of the model's thinking back to it.
+
+ When a model thinks and then calls a tool, we replay that thinking on the next request, so it
+ still knows why it made the call.
+
+ We put it under `:reasoning_content` because that is the key the shared Chat Completions code
+ knows how to merge. [[ollama-reasoning-spelling]] renames it before the request goes out.
+
+ This only ever replays thinking from the turn in progress. Thinking is thrown away before a turn
+ is saved, so it can never reach a later one. That is what makes this safe for any model: models
+ disagree about replaying *old* thinking, but not about the turn they are in the middle of."
+ [part]
+ {:role "assistant" :content "" :reasoning_content (:text part)})
+
+(defn- ollama-reasoning-spelling
+ "Renames replayed thinking to the field name Ollama actually reads.
+
+ Ollama sends thinking *to* us as either `reasoning` or `reasoning_content`, depending on its
+ version. But when we send thinking back, only `reasoning` works.
+
+ Get it wrong and Ollama quietly ignores the field: no error, replay just does nothing. Renaming
+ every message is safe, because [[reasoning-message]] is the only thing that sets this key."
+ [body]
+ (update body :messages #(mapv (fn [m] (set/rename-keys m {:reasoning_content :reasoning})) %)))
+
+(mu/defn ollama-request-body
+ "The Chat Completions body, as [[chat-completions/request-body]] builds it plus the adapter-local
+ adjustments: `max_tokens` is always sent (uncapped, a looping small model burns the whole context
+ window in one call), raised to the floors above where they apply; `temperature` falls back to
+ [[adapter/default-temperature]]; and the model's thinking is replayed (see [[reasoning-message]]). They
+ stay here rather than in the shared builder, which also serves Z.AI, Mistral and OpenRouter.
+
+ A forced tool call is [[metabase.metabot.self.ollama.forced-calls]]' subject, because `tool_choice`
+ does nothing on Ollama and what can be done instead depends on who serves the model — see [[forced-plan]].
+ `plan` may be supplied by a caller that already has one — the streaming path does — so that a request
+ derives it once.
+
+ The reasoning floor is looked up rather than read off the connection, because one connection serves
+ as many models as the operator has pulled — see
+ [[metabase.metabot.self.ollama.capabilities/reasoning-model?]]. It is answered from cache after the
+ first request on a model.
+
+ We never tell Ollama whether to think, just as vLLM does not. Both run whatever model the operator
+ installed, so we take that model's own default and leave room for it with the floors above.
+ Ollama does have a `reasoning_effort` switch, but turning thinking off would only save tokens on
+ the operator's own hardware, and it errors on models that cannot think at all."
+ ([opts :- core/LLMRequestOpts]
+ (ollama-request-body opts (forced-plan opts)))
+
+ ([{:keys [max-tokens model temperature credentials reasoning?] :as opts
+ :or {reasoning? true}} :- core/LLMRequestOpts
+ plan :- [:maybe ::forced/plan]]
+ (forced/body-for
+ plan
+ (assoc (ollama-reasoning-spelling
+ (chat-completions/request-body
+ (cond-> (forced/opts-for plan opts)
+ (nil? temperature) (assoc :temperature adapter/default-temperature))
+ (when reasoning? {:reasoning-part->message reasoning-message})))
+ :max_tokens (cond-> (or max-tokens core/chat-max-output-tokens)
+ (some? plan)
+ (max adapter/forced-tool-call-token-floor)
+
+ (caps/reasoning-model? credentials model)
+ (max adapter/reasoning-model-token-floor))))))
+
+(defn- stream-io-ex
+ "Transport failure while *consuming* a stream. `:retryable? false` is required, not decorative:
+ `retryable-error?` walks the cause chain and would otherwise match the `IOException`, replaying a
+ full cold prefill for what is really \"too slow\" or \"it died\"."
+ [^IOException e timeout-ms]
+ (if (instance? SocketTimeoutException e)
+ (ex-info (tru "The Ollama server stopped responding after {0}ms. Raise the Ollama request timeout, or pull a smaller model."
+ (str timeout-ms))
+ {:api-error true
+ :error-code :ollama-timeout
+ :retryable? false}
+ e)
+ (ex-info (tru "The connection to the Ollama server was interrupted before the response finished.")
+ {:api-error true
+ :error-code :ollama-stream-interrupted
+ :retryable? false}
+ e)))
+
+(defn- request-io-ex
+ "Transport failure while *establishing* a request. `core/rethrow-api-error!` would render these as
+ \"ollama API request failed: Read timed out\", naming neither the slowness nor the setting for it.
+ `:retryable? false` matters more here than in [[stream-io-ex]]: nothing has been emitted yet, so
+ `call-llm`'s own \"nothing emitted\" guard would not stop a replay."
+ [^IOException e url timeout-ms]
+ (if (instance? SocketTimeoutException e)
+ (ex-info (tru "The Ollama server did not respond within {0}ms. A cold model load happens on the first request — retry once it is warm, or raise the Ollama request timeout."
+ (str timeout-ms))
+ {:api-error true
+ :error-code :ollama-timeout
+ :retryable? false}
+ e)
+ (unreachable-ex e url {:retryable? false})))
+
+(mu/defn ollama-raw
+ "Stream a Chat Completions request. `:credentials` come from the connection serving it; `:ai-proxy?` is
+ unsupported and throws. `plan` may be supplied by a caller that already has one, as in
+ [[ollama-request-body]]."
+ ([opts :- core/LLMRequestOpts]
+ (ollama-raw opts (forced-plan opts)))
+
+ ([{:keys [model credentials] :as opts} :- core/LLMRequestOpts
+ plan :- [:maybe ::forced/plan]]
+ (when (str/blank? model) (throw (missing-model-ex)))
+ (let [timeout-ms (llm/llm-ollama-request-timeout-ms)]
+ (adapter/stream! provider opts
+ {:path "/chat/completions"
+ :body (ollama-request-body opts plan)
+ :request-options (inference-timeouts)
+ :span-attrs {:forced (some-> (:mechanism plan) name)}
+ :wrap-stream #(adapter/io-guarded % (fn [e] (stream-io-ex e timeout-ms)))
+ ;; clj-http raises an `IOException` only when there is no response at all, so the
+ ;; IO branch cannot swallow a failure the provider's own messages would translate.
+ :on-request-error (fn [e]
+ (if (instance? IOException e)
+ (throw (request-io-ex e (conn/base-url credentials) timeout-ms))
+ (adapter/rethrow! provider e)))}))))
+
+(mu/defn streams-reasoning? :- :boolean
+ "Registry capability. Ollama answers per *model*, from what the server reports about it — a connection
+ serves every model the operator pulled, and Metabot and the mini model need not be on the same one, so
+ no flag on the connection could describe both.
+
+ The cached reader, never the one that would call Ollama: this backs the public
+ `llm-metabot-supports-reasoning?` setting, so every client's page load reaches it and none of them may
+ wait on the operator's server."
+ [{:keys [credentials model]} :- adapter/ResolvedRef]
+ (caps/cached-reasoning-model? credentials model))
+
+(defn ollama->aisdk-chunks-xf
+ "Chat Completions chunks to AI SDK v5. Reasoning is forwarded when present, which is self-gating —
+ the field only appears for a reasoning model. Ollama adds no `finish_reason` beyond OpenAI's."
+ []
+ (chat-completions/chat-completions->aisdk-chunks-xf chat-completions/stop-reasons
+ {:forward-reasoning? true}))
+
+(defn- unfinished-stream-xf
+ "Fail a stream that ends without saying why it ended.
+
+ When generation fails after the first token — the runner crashing, or a model's parser rejecting a
+ malformed tool call — Ollama's OpenAI layer sends the error as an empty chunk and closes the stream with
+ no `finish_reason` and no `[DONE]`. Passed through, the partial text would read as a complete answer.
+ Its own error chunk, emitted before the stream completes, lets the shared translation close whatever
+ block is open and fail the turn the way a reported error does. A consumer that stopped reading early
+ ended the stream itself, so nothing is added then."
+ []
+ (fn [rf]
+ ;; set once the stream said why it ended, or the consumer ended it
+ (let [done? (volatile! false)]
+ (fn
+ ([] (rf))
+ ([result]
+ (rf (if @done?
+ result
+ (unreduced (rf result {:error {:message (tru "The Ollama server ended the response before finishing it.")}})))))
+ ([result {:keys [choices error] :as chunk}]
+ (let [result (rf result chunk)]
+ (when (or (reduced? result) (some? error) (some :finish_reason choices))
+ (vreset! done? true))
+ result))))))
+
+(defn ollama
+ "Call an Ollama server's Chat Completions API, return AISDK stream."
+ [opts]
+ (let [plan (forced-plan opts)]
+ (eduction (comp (or (forced/read-back-xf plan) identity)
+ (unfinished-stream-xf)
+ (ollama->aisdk-chunks-xf))
+ (ollama-raw opts plan))))
diff --git a/src/metabase/metabot/self/ollama/capabilities.clj b/src/metabase/metabot/self/ollama/capabilities.clj
new file mode 100644
index 00000000000..f898fb9de18
--- /dev/null
+++ b/src/metabase/metabot/self/ollama/capabilities.clj
@@ -0,0 +1,284 @@
+(ns metabase.metabot.self.ollama.capabilities
+ "What the models behind an Ollama connection can do, and remembering it.
+
+ Ollama reports this per model, from the model's own template, on `POST /api/show` — Cloud serves it
+ too. It is the only thing that tells a chat model from an embedding model, which its
+ OpenAI-compatible catalog lists side by side and marks neither of, and the only thing that says
+ whether a model thinks. Names cannot: an Ollama tag is local (`qwen3:8b`, or whatever an operator
+ called their own Modelfile), so nothing here can be settled the way it is for a hosted provider.
+
+ A connection serves as many models as the operator has pulled, so every answer here is about one
+ *model*, never about the connection. That is the whole point: Metabot and the mini model need not
+ be on the same one, and a flag recorded once against the connection could only ever describe one
+ of them.
+
+ Keeping the answers fresh is this namespace's own business — see [[cached-capabilities]]. Nothing
+ outside needs to know a cache exists, or to remember to fill it."
+ (:require
+ [clojure.core.cache :as cache]
+ [clojure.set :as set]
+ [clojure.string :as str]
+ [com.climate.claypoole :as cp]
+ [metabase.llm.settings :as llm]
+ [metabase.metabot.self.adapter :as adapter]
+ [metabase.metabot.self.ollama.connection :as conn]
+ [metabase.util :as u]
+ [metabase.util.json :as json]
+ [metabase.util.jvm :as u.jvm]
+ [metabase.util.log :as log]
+ [metabase.util.malli :as mu])
+ (:import
+ (java.util.concurrent Semaphore)))
+
+(set! *warn-on-reflection* true)
+
+;;; ---------------------------------------------- Asking the server ----------------------------------------------
+
+(def ^:private native-api-timeout-ms
+ "Upper bound on a `/api/show` lookup. It reads metadata Ollama already has in memory, so anything
+ slower than this is a server in trouble — and the callers are a model being listed and a request
+ about to be made, neither of which can wait on it."
+ 5000)
+
+(def ^:private thinking-capability
+ "The `capabilities` entry Ollama reports for a model whose template separates thinking from the
+ answer — the same template that decides whether a `reasoning` field ever comes back."
+ "thinking")
+
+(def ^:private chat-capabilities
+ "What a model has to report to be worth offering Metabot. Embedding models report neither."
+ #{"completion" "tools"})
+
+(def ^:private lookup-concurrency
+ "How many models to ask about at once. A catalog listing asks about every model it offers, and
+ `/api/show` opens a fresh connection per call — against Cloud that is a TLS handshake each, to one
+ host. Unbounded, a 40-model catalog is how an instance earns a 429, and a 429 is cached as \"would
+ not say\" until [[retry-after-ms]] passes, so one burst would cost reasoning detection for every
+ model on the connection meanwhile."
+ 8)
+
+(defonce ^:private lookup-permits
+ ;; Taken by every `/api/show` call, cold or background, so [[lookup-concurrency]] bounds all the
+ ;; calls open at once rather than the calls one fan-out makes.
+ (Semaphore. lookup-concurrency))
+
+(defn- with-permit
+ "Call `f` holding one of [[lookup-permits]]."
+ [f]
+ (.acquire ^Semaphore lookup-permits)
+ (try
+ (f)
+ (finally
+ (.release ^Semaphore lookup-permits))))
+
+(defn- fetch-capabilities
+ "Ask Ollama what `model` can do: `{:caps #{...}-or-nil, :answered? bool}`.
+
+ `:caps` is nil whenever the server would not say, and `:answered?` separates the two reasons it
+ might not. An Ollama too old to report `capabilities` answers, so asking again will not help; one
+ that errored or timed out did not, and probably will next time — see [[stale?]], which is the only
+ thing the distinction is for.
+
+ Never throws: this is metadata that sharpens a decision the adapter can still make without it, so a
+ server that will not answer must not take down the request or the page that asked."
+ [credentials model]
+ (try
+ (let [res (with-permit
+ #(adapter/request! conn/native-provider
+ {:credentials credentials
+ :method :post
+ :path "/api/show"
+ :as :json
+ :body (json/encode {:model model})}
+ {:socket-timeout native-api-timeout-ms
+ :connection-timeout (llm/llm-connection-timeout-ms)}))
+ caps (get-in res [:body :capabilities])]
+ {:answered? true
+ :caps (when (sequential? caps)
+ (into #{} (map str) caps))})
+ (catch Exception e
+ (log/debug e "Ollama did not report capabilities" {:model model})
+ {:answered? false})))
+
+;;; -------------------------------------------------- The cache --------------------------------------------------
+
+(def ^:private refresh-after-ms
+ "How old an answer may get before the next read re-asks behind the caller.
+
+ A real capability set is near-immutable for a given server and tag, but not quite — Ollama tags are
+ mutable, so a re-pull or a rebuilt Modelfile can change a template under the same name."
+ 600000)
+
+(def ^:private cache-threshold
+ "How many models the capabilities cache holds before it evicts the least recently used."
+ 256)
+
+(defonce ^:private capabilities-cache
+ ;; `{cache-key {:caps #{...}-or-nil :at ms}}`. LRU rather than TTL because age must not mean
+ ;; forgetting: an entry that has gone stale is still the best answer we have, and an expired one is
+ ;; not evidence that a model stopped thinking. `:at` says when to re-ask, `:caps` what to believe
+ ;; meanwhile, and the bound is on size instead.
+ (atom (cache/lru-cache-factory {} :threshold cache-threshold)))
+
+(defn- cache-key
+ "What a model's capabilities are filed under: the destination and credential that would be asked,
+ reduced to a hash, plus the model.
+
+ Hashed rather than held as-is for the reason `metabase.llm.api.provider`'s model cache does it —
+ the API key is in there, and a cache entry outlives the connection that produced it. Only the
+ fields that decide *which server answers* are in the key, so an entry survives the rest of a
+ connection being edited and, more to the point, survives the connect path storing what it learned
+ back onto the very config it probed with. Repointing at another address or rotating the key retires
+ it, since the same tag on another server is another model."
+ [credentials model]
+ [(hash (select-keys credentials [:base-url :api-key])) model])
+
+(def ^:private retry-after-ms
+ "How long an entry that holds no answer, from a server that did not give one, is left alone.
+
+ Much shorter than [[refresh-after-ms]] because there is nothing to serve meanwhile: until this
+ elapses, every model behind a server that was briefly unreachable reads as not reasoning and runs on
+ the smaller token budget. Not shorter still, so a server that is properly down is not asked once per
+ read."
+ 60000)
+
+(defn- stale?
+ "Whether `entry` is old enough to re-ask for.
+
+ An entry holding no answer from a server that never gave one is re-asked on [[retry-after-ms]]: it
+ records a failure rather than a fact, and unlike a real answer there is nothing to serve while it
+ stands. An Ollama that answered without `capabilities` is not that case — it will answer the same way
+ for as long as it runs that build — and neither is a failed re-ask over an answer we already hold."
+ [{:keys [caps answered? at]}]
+ (< (if (or caps answered?) refresh-after-ms retry-after-ms)
+ (u/since-ms at)))
+
+(defn- remember!
+ "Record what a lookup returned, and return the capability set now believed.
+
+ A lookup that came back with nothing keeps whatever was believed before — a server that would not
+ answer is not evidence that a model changed — but still stamps `:at`, so a broken endpoint is
+ re-asked on an interval rather than once per read."
+ [k {:keys [caps answered?]}]
+ (-> (swap! capabilities-cache
+ (fn [c]
+ (cache/miss c k {:caps (or caps (:caps (cache/lookup c k)))
+ :answered? answered?
+ :at (u/start-timer)})))
+ (cache/lookup k)
+ :caps))
+
+(defn- read-entry
+ "The cache entry for `k`, counted as a use so the LRU evicts by reads as well as writes."
+ [k]
+ (-> (swap! capabilities-cache #(cond-> % (cache/has? % k) (cache/hit k)))
+ (cache/lookup k)))
+
+;; Lookups in flight, by cache key. Callers after the same key share one, so a burst for one model —
+;; page loads against a stale entry, or requests for a model with none — asks once.
+(defonce ^:private in-flight
+ (atom {}))
+
+(defn- shared-lookup
+ "The lookup for `model`, joining the one in flight or starting one: `[started? lookup]`, where
+ `lookup` is a delay that asks the server and records the answer. Whoever gets `started?` is
+ responsible for it running; everyone else may deref it or leave it.
+
+ The lookup re-reads the cache first and asks only if the entry is still missing or stale. A caller
+ decides to start one from a read taken earlier, and a lookup that finished in between has already
+ answered."
+ [credentials model]
+ (let [k (cache-key credentials model)
+ lookup (delay
+ (try
+ (let [entry (cache/lookup @capabilities-cache k)]
+ (if (and entry (not (stale? entry)))
+ (:caps entry)
+ (remember! k (fetch-capabilities credentials model))))
+ (finally
+ (swap! in-flight dissoc k))))
+ ;; `swap-vals!` hands back the map as it was, so exactly one thread finds `k` missing from it
+ [before after] (swap-vals! in-flight #(cond-> % (not (contains? % k)) (assoc k lookup)))]
+ [(not (contains? before k)) (get after k)]))
+
+(defn- refresh-in-background!
+ "Re-ask for `model` on another thread, unless a lookup for it is already in flight."
+ [credentials model]
+ (let [[started? lookup] (shared-lookup credentials model)]
+ (when started?
+ (u.jvm/in-virtual-thread* @lookup))))
+
+(defn- capabilities
+ "`model`'s capability set, or nil when nothing is known about it.
+
+ Serves what is believed whenever there is an entry, re-asking behind the caller once it is stale.
+ Only a model with no entry at all is fetched in front of the caller."
+ [credentials model]
+ (when-not (str/blank? model)
+ (let [k (cache-key credentials model)]
+ (if-let [entry (read-entry k)]
+ (do (when (stale? entry)
+ (refresh-in-background! credentials model))
+ (:caps entry))
+ @(second (shared-lookup credentials model))))))
+
+(defn- cached-capabilities
+ "[[capabilities]] without the cold fetch: nil rather than a wait when nothing is known yet.
+
+ There is exactly one such reader, and it is the reason this exists: the public
+ `llm-metabot-supports-reasoning?` setting, read on page load by every client, which must never make
+ that client wait on the operator's Ollama."
+ [credentials model]
+ ;; the same guard [[capabilities]] has, and here it is load-bearing: a model reference with no model
+ ;; segment would never fill the cache, so every page load would hand a future the same nothing to do
+ (when-not (str/blank? model)
+ (let [k (cache-key credentials model)
+ entry (read-entry k)]
+ (when (or (nil? entry) (stale? entry))
+ (refresh-in-background! credentials model))
+ (:caps entry))))
+
+(defn clear-cache!
+ "Forget every lookup. For tests, which must not inherit each other's servers."
+ []
+ (reset! in-flight {})
+ (swap! capabilities-cache cache/seed {}))
+
+;;; --------------------------------------------- What callers ask for --------------------------------------------
+
+(mu/defn reasoning-model? :- :boolean
+ "Whether the connection's `model` streams its reasoning back to us, asking Ollama if we have not
+ asked recently. The ordinary accessor: a caller already making a generation request does not notice
+ one cached metadata lookup.
+
+ A server that will not say reads as not reasoning. Thinking still renders either way — the
+ `reasoning` field is forwarded whenever it appears — but the request keeps the smaller token
+ budget, which a thinking model can exhaust. That surfaces as truncation, not as a wrong answer."
+ [credentials :- conn/Credentials
+ model :- [:maybe :string]]
+ (contains? (capabilities credentials model) thinking-capability))
+
+(mu/defn cached-reasoning-model? :- :boolean
+ "[[reasoning-model?]] for the one caller that cannot make an HTTP call — see [[cached-capabilities]],
+ which also arranges for the next such read to be right."
+ [credentials :- conn/Credentials
+ model :- [:maybe :string]]
+ (contains? (cached-capabilities credentials model) thinking-capability))
+
+(mu/defn chat-capable? :- :boolean
+ "Whether `model` is one Metabot could run on. Unknown capabilities count as capable: only a server
+ that answers is allowed to rule a model out, so an older Ollama goes on offering what it always did."
+ [credentials :- conn/Credentials
+ model :- [:maybe :string]]
+ (let [caps (capabilities credentials model)]
+ (or (nil? caps) (set/subset? chat-capabilities caps))))
+
+(mu/defn chat-capable-ids :- [:set :string]
+ "Which of `model-ids` Ollama offers for chat at all, asked [[lookup-concurrency]] at a time."
+ [credentials :- conn/Credentials
+ model-ids :- [:sequential :string]]
+ (into #{}
+ (filter some?)
+ (cp/pmap lookup-concurrency
+ #(when (chat-capable? credentials %) %)
+ model-ids)))
diff --git a/src/metabase/metabot/self/ollama/connection.clj b/src/metabase/metabot/self/ollama/connection.clj
new file mode 100644
index 00000000000..2f23b56425e
--- /dev/null
+++ b/src/metabase/metabot/self/ollama/connection.clj
@@ -0,0 +1,120 @@
+(ns metabase.metabot.self.ollama.connection
+ "Which Ollama server a connection points at, how to reach it, and whether Ollama Cloud serves a model.
+
+ Ollama answers on two surfaces: the OpenAI-compatible API the adapter generates against, mounted
+ under the `/v1` an admin's base URL ends in, and Ollama's own API at the server root. Both live on
+ the same host and take the same key, so which one a caller wants is the only thing that differs.
+
+ Nothing here reaches the network. [[metabase.metabot.self.ollama.capabilities]] is the layer above
+ that does.
+
+ https://docs.ollama.com/api/openai-compatibility"
+ (:require
+ [clojure.string :as str]
+ [metabase.metabot.self.adapter :as adapter]
+ [metabase.metabot.self.core :as core]
+ [metabase.util :as u]
+ [metabase.util.http :as u.http]
+ [metabase.util.i18n :refer [tru]]
+ [metabase.util.malli :as mu]))
+
+(set! *warn-on-reflection* true)
+
+(def Credentials
+ "An Ollama connection's `:config` — the shape every adapter entry point receives as `:credentials`.
+
+ `:maybe`, and every key optional: a caller with no connection at all — a request body built outside
+ any connection — passes nothing."
+ [:maybe
+ [:map {:closed true}
+ [:api-key {:optional true} [:maybe :string]]
+ [:base-url {:optional true} [:maybe :string]]
+ ;; recorded by the connect-time probe, not entered by the admin
+ [:probed-model {:optional true} [:maybe :string]]]])
+
+(defn- missing-base-url-ex []
+ ;; `provider-client-error?` needs a numeric status to render this under the field; without one the
+ ;; admin gets a 500.
+ (ex-info (tru "No Ollama base URL is set. Give the address of your server, or https://ollama.com/v1 for Ollama Cloud.")
+ {:api-error true
+ :status-code 400
+ :field :base-url
+ :error-code :base-url-missing}))
+
+(mu/defn base-url :- :string
+ "The address to call.
+
+ Public because the adapter's transport errors name the address they failed to reach."
+ [credentials :- Credentials]
+ (or (not-empty (:base-url credentials)) (throw (missing-base-url-ex))))
+
+;;; ------------------------------------------------ Ollama Cloud ------------------------------------------------
+
+(defn- cloud-address?
+ "Whether `url` is Ollama Cloud's own API, `https://ollama.com/v1`, rather than a server of the operator's."
+ [url]
+ (let [host (some-> (u.http/->hostname url) u/lower-case-en)]
+ (boolean (and host (or (= "ollama.com" host) (str/ends-with? host ".ollama.com"))))))
+
+(defn- cloud-tag?
+ "Whether `model` names a Cloud model by Ollama's own rule (`parseSourceSuffix` in
+ `internal/modelref/modelref.go`): the part after the last `:` is `cloud`, or ends in `-cloud`, in any case.
+ A self-hosted server forwards such a model to ollama.com, request body and all."
+ [model]
+ (boolean
+ (when-let [idx (some-> model (str/last-index-of ":"))]
+ (let [suffix (u/lower-case-en (str/trim (subs model (inc idx))))]
+ (or (= "cloud" suffix)
+ (and (not (str/includes? suffix "/"))
+ (str/ends-with? suffix "-cloud")))))))
+
+(mu/defn served-by-cloud? :- :boolean
+ "Whether Ollama Cloud serves `model` on this connection — what decides how a forced tool call is
+ expressed, since Cloud discards both `tool_choice` and a `response_format` grammar.
+
+ Either signal is enough. Ollama Cloud's own address serves every model under its plain name
+ (`gemma4:31b`), so there the address is the only signal; a self-hosted server serves a Cloud model
+ under a name that says so (`gpt-oss:120b-cloud`) and forwards it.
+
+ Neither catches a Cloud model copied to a name without the suffix, nor a gateway in front of
+ ollama.com. Both are taken for self-hosted and get a grammar."
+ [credentials :- Credentials
+ model :- [:maybe :string]]
+ (or (cloud-address? (:base-url credentials))
+ (cloud-tag? model)))
+
+(mu/defn- resolve-auth :- adapter/Auth
+ [credentials :- Credentials]
+ (let [token (not-empty (:api-key credentials))]
+ (core/resolve-auth "ollama" "Ollama"
+ (cond-> {:url (base-url credentials)}
+ token (assoc :headers {"Authorization" (str "Bearer " token)}))
+ false)))
+
+(mu/defn auth :- adapter/Auth
+ "Ollama's `:auth`, for the OpenAI-compatible API the adapter generates against.
+
+ Never nil, so `core/resolve-auth`'s missing-key branch is unreachable: a keyless self-hosted server
+ is the normal configuration, not a broken one. The proxy is refused before this runs — the
+ descriptor declares no `:supports-ai-proxy?`, so [[adapter/request!]] rejects a proxied call."
+ [_provider :- adapter/Provider
+ {:keys [credentials]} :- adapter/Request]
+ (resolve-auth credentials))
+
+(mu/defn native-auth :- adapter/Auth
+ "Auth for Ollama's own API, which lives at the server root rather than under the `/v1` the
+ OpenAI-compatible surface is mounted at. Ollama Cloud serves it on the same host, and takes the same key."
+ [credentials :- Credentials]
+ (update (resolve-auth credentials) :url str/replace #"/v1/*$" ""))
+
+(def native-provider
+ "Descriptor for Ollama's own API, the surface that answers `/api/show` and `/api/ps`.
+
+ A second descriptor rather than a second door: the adapter's own descriptor authenticates the
+ OpenAI-compatible surface, and everything else [[adapter/request!]] does — the proxy refusal, the
+ `Content-Type` only where there is a body — applies here unchanged. It carries no `:errors`, because
+ every caller on this surface treats a failure as \"the server would not say\" and swallows it."
+ (adapter/provider
+ {:slug "ollama"
+ :display-name "Ollama"
+ :auth (fn [_provider {:keys [credentials]}] (native-auth credentials))}))
diff --git a/src/metabase/metabot/self/ollama/forced_calls.clj b/src/metabase/metabot/self/ollama/forced_calls.clj
new file mode 100644
index 00000000000..f78a22ca5ba
--- /dev/null
+++ b/src/metabase/metabot/self/ollama/forced_calls.clj
@@ -0,0 +1,349 @@
+(ns metabase.metabot.self.ollama.forced-calls
+ "How a forced tool call is expressed on Ollama, and read back.
+
+ Ollama accepts `tool_choice` and discards it — the field is not in its request struct at all
+ (https://docs.ollama.com/api/openai-compatibility lists it unsupported), so the forced tool call the
+ shared Chat Completions builder produces is ignored (without any error). That
+ makes forcing a call a subject in its own right here, rather than one flag on a request: what can be
+ enforced depends on who serves the model, the request goes out differently as a result, and the
+ answer comes back on a different channel and has to be moved.
+
+ What a self-hosted server does have is `response_format`, which compiles a JSON Schema into a
+ decoding grammar and the constraint holds
+ against a prompt arguing otherwise. Forcing a call among several tools uses the union schema
+ Ollama's own maintainers endorsed for this and mean to ship natively (ollama/ollama#6002) — an
+ `anyOf` over `{name, parameters}`, one arm per tool.
+
+ A grammar makes a tool call unavoidable, which is exactly `required` and exactly wrong for `auto`,
+ so [[plan]] produces one only where the caller asked for `required`.
+
+ None of it is available on Ollama Cloud, which serves no structured outputs (ollama/ollama#12362,
+ https://docs.ollama.com/capabilities/structured-outputs) and discards `format` as silently as
+ `tool_choice` — whether reached at ollama.com or through a self-hosted server that forwards a Cloud
+ model there. A forced call to a Cloud model is asked for in words and nothing guarantees it."
+ (:require
+ [clojure.string :as str]
+ [metabase.metabot.self.core :as core]
+ [metabase.metabot.self.openai.chat-completions :as chat-completions]
+ [metabase.metabot.self.schema :as schema]
+ [metabase.util :as u]
+ [metabase.util.json :as json]
+ [metabase.util.malli :as mu]
+ [metabase.util.malli.registry :as mr]))
+
+(set! *warn-on-reflection* true)
+
+;;; --------------------------------------------------- Contracts ------------------------------------------------
+
+(mr/def ::mode
+ "What kind of forced call a request asks for.
+
+ `:structured` is one known shape, from `:schema` — the caller wants data rather than prose.
+ `:tool-union` is a call among the caller's own tools, from `tool_choice: required`."
+ [:enum :structured :tool-union])
+
+(mr/def ::mechanism
+ "What the server serving the model can actually do about a [[::mode]].
+
+ `:grammar` is enforced by the server and cannot be talked out of. `:instruction` is a request in
+ words, which a model may ignore. `:none` means neither is available — a `required` turn carrying no
+ tools, which has nothing to name. It is a plan rather than a nil so that a forced request is always
+ planned for: the token budget has to cover what was asked for whether or not anything can compel it."
+ [:enum :grammar :instruction :none])
+
+(mr/def ::plan
+ "How a request's forced tool call will be expressed. `:schema` is present exactly when `:mechanism`
+ is `:grammar`: it is the thing the decoder gets constrained by, and the other mechanisms constrain
+ nothing. `:instruction` is present exactly when `:mechanism` is `:instruction`, and is the words
+ that stand in for a constraint."
+ [:and
+ [:map {:closed true}
+ [:mode ::mode]
+ [:mechanism ::mechanism]
+ [:schema {:optional true} [:map {::mr/deliberately-open true
+ :description "a JSON Schema the decoder is constrained by"}]]
+ [:instruction {:optional true} :string]]
+ [:fn {:error/message "a :schema belongs to a :grammar, and every :grammar needs one"}
+ (fn [{:keys [mechanism schema]}]
+ (= (= :grammar mechanism) (some? schema)))]
+ [:fn {:error/message "an :instruction belongs to the :instruction mechanism, and it needs one"}
+ (fn [{:keys [mechanism instruction]}]
+ (= (= :instruction mechanism) (some? instruction)))]])
+
+(mr/def ::probe-verdict
+ "What a structured-output probe showed: nil when the mechanism held, or why it did not."
+ [:maybe [:enum :truncated :not-honored]])
+
+(mr/def ::choice
+ "One `choices` entry of a non-streaming Chat Completions answer. Open, like every other shape a
+ provider sends rather than one we compose: a build that adds a field must not fail validation for it."
+ [:map {::mr/deliberately-open true
+ :description "a Chat Completions choice"}
+ [:finish_reason {:optional true} [:maybe :string]]
+ [:message {:optional true} [:maybe [:map {::mr/deliberately-open true
+ :description "a Chat Completions assistant message"}]]]])
+
+(mr/def ::chat-completions-body
+ "A Chat Completions request body. Open on purpose — this namespace reads and writes two of its keys
+ and must not care about the rest."
+ [:map {::mr/deliberately-open true
+ :description "a Chat Completions request body"}])
+
+(def ^:private tool-name
+ "A grammar-forced answer has no tool call of its own, so [[read-back-xf]] mints one — under the name
+ the shared builder would have used, so that downstream cannot tell the difference."
+ chat-completions/structured-output-tool-name)
+
+(defn cloud-instruction
+ "What a Cloud forced call asks for in place of the `tool_choice` Ollama discards: a call to one of
+ `tool-names`. Appended last, where an instruction carries furthest. It is not a guarantee and is not
+ treated as one: a model that answers in chat anyway produces the caller's ordinary missing-tool-call
+ failure.
+
+ The tools are named rather than left to `tools` and the discarded `tool_choice`, because naming them
+ is the whole of what the instruction can do — a model told only \"call a tool\" has been told less
+ than the request already said."
+ [tool-names]
+ (str "Answer by calling "
+ (if (next tool-names)
+ (str "one of these tools: " (str/join ", " (map #(str "`" % "`") tool-names)))
+ (str "the `" (first tool-names) "` tool"))
+ ". Do not reply in chat."))
+
+;;; ------------------------------------------------- The decision -----------------------------------------------
+
+(defn- response-format
+ "The `response_format` that puts Ollama's decoder under `schema`. Ollama reads only the inner
+ `:schema`; `:name` is sent to keep the body the shape every other OpenAI-compatible server expects."
+ [schema]
+ {:type "json_schema"
+ :json_schema {:name tool-name
+ :schema schema}})
+
+(defn- tool-union-schema
+ "A schema satisfied only by a call to one of `tools`, which are [[core/LLMRequestOpts]] tool entries.
+
+ One `anyOf` arm per tool, each pinning `name` to that tool with a single-value `enum` and taking its
+ own parameter schema. The model can satisfy the grammar only by naming a tool and supplying
+ arguments that fit it, which is the guarantee `tool_choice: required` was supposed to give."
+ [tools]
+ {:anyOf (mapv (fn [tool]
+ (let [{:keys [name parameters]} (schema/tool-function tool)]
+ {:type "object"
+ :properties {:name {:type "string" :enum [name]}
+ :parameters parameters}
+ :required ["name" "parameters"]}))
+ tools)})
+
+(mu/defn plan :- [:maybe ::plan]
+ "How `opts`' forced tool call will be expressed, or nil when nothing is forced — see [[::plan]] for the
+ shape. `cloud?` says whether Ollama Cloud serves the requested model.
+
+ `cloud?` is the only thing about the server this needs, and what it implies is decided here rather
+ than by the caller: the adapter should not have to know that Cloud's limitation is about grammars,
+ only who serves the model.
+
+ A plan is produced for every forced request, even where nothing can be done about it, so that
+ callers have one question to ask rather than two: `(some? (plan opts cloud?))` answers whether a
+ tool call was asked for, and `:mechanism` answers what came of it.
+
+ `auto` yields nil: a grammar would silently promote it to `required`."
+ [{:keys [schema tools tool_choice]} :- core/LLMRequestOpts
+ cloud? :- :boolean]
+ (when-let [mode (cond
+ (some? schema) :structured
+ (= "required" (some-> tool_choice name)) :tool-union)]
+ (let [union? (= :tool-union mode)]
+ (cond
+ ;; nothing to choose among: no mechanism can compel a call that has no tool to make, and a
+ ;; union grammar over no tools would be a constraint nothing could satisfy
+ (and union? (empty? tools)) {:mode mode :mechanism :none}
+ cloud? {:mode mode
+ :mechanism :instruction
+ :instruction (cloud-instruction
+ (if union?
+ (mapv #(:name (schema/tool-function %)) tools)
+ [tool-name]))}
+ :else {:mode mode
+ :mechanism :grammar
+ :schema (if union? (tool-union-schema tools) schema)}))))
+
+;;; --------------------------------------------- Shaping the request --------------------------------------------
+
+(mu/defn opts-for
+ "`opts` adjusted for `plan`, before the shared Chat Completions builder sees them. A nil plan changes
+ nothing, so an unforced request passes through untouched."
+ [plan :- [:maybe ::plan]
+ opts :- core/LLMRequestOpts]
+ (case (:mechanism plan)
+ ;; the shared builder mints a tool and a `tool_choice` from `:schema`; under a grammar that already
+ ;; carries the shape there is nothing for the model to call. `:tools` goes with it to preserve the
+ ;; builder's own rule — `chat-completions/request-body` drops real tools whenever `:schema` is set
+ ;; (its `all-tools` binding), so leaving them here would hand the model tools this request never had
+ :grammar (cond-> opts (= :structured (:mode plan)) (dissoc :schema :tools))
+ :instruction (update opts :input #(conj (vec %) {:role "user" :content (:instruction plan)}))
+ opts))
+
+(mu/defn body-for :- ::chat-completions-body
+ "The built Chat Completions `body` adjusted for `plan`. A nil plan changes nothing."
+ [plan :- [:maybe ::plan]
+ body :- ::chat-completions-body]
+ (if (= :grammar (:mechanism plan))
+ ;; the builder defaults `tool_choice` to "auto" whenever tools are present. Ollama discards it
+ ;; either way, but a body that says "auto" while being grammar-forced reads as the opposite of what
+ ;; it does — and this body is what lands in the debug capture. Cloud keeps its `tool_choice`, where
+ ;; it states the real intent and is the one thing that would make Cloud work on its own the day
+ ;; Ollama implements the field.
+ (-> body
+ (assoc :response_format (response-format (:schema plan)))
+ (dissoc :tool_choice))
+ body))
+
+;;; ---------------------------------------------- Reading it back -----------------------------------------------
+
+(defn- forced-call-chunk
+ "The Chat Completions chunk that carries `content` back as a tool call, or nil when it cannot be read
+ as one.
+
+ `:structured` knows the name up front and the whole answer is the arguments. `:tool-union` has to
+ read the name out of the answer, so a truncated one yields nil: emitting no call lets the `length`
+ finish reason stand as the diagnosis, which is the true one."
+ [mode content]
+ (let [[name arguments]
+ (case mode
+ :structured [tool-name content]
+ :tool-union (let [parsed (try (json/decode+kw content) (catch Exception _ nil))]
+ (when-let [called (and (map? parsed) (not-empty (str (:name parsed))))]
+ [called (json/encode (or (:parameters parsed) {}))])))]
+ (when name
+ {:choices [{:index 0
+ :delta {:tool_calls [{:id (core/mkid)
+ :type "function"
+ :function {:name name :arguments arguments}}]}}]})))
+
+(mu/defn read-back-xf :- [:maybe fn?]
+ "A transducer over Chat Completions chunks putting a grammar-forced answer back on the tool-call
+ channel, or nil when `plan` needs none.
+
+ Under a grammar the answer arrives in the content channel — `message.content`, not `tool_calls` —
+ because the grammar constrains decoding rather than driving the template's tool path. Both the
+ `call-llm-structured` consumers and the agent loop read tool calls, so the difference is undone
+ here, ahead of the shared translation: downstream sees exactly the stream a provider with a working
+ `tool_choice` produces.
+
+ Content is held rather than forwarded, because `:tool-union` cannot name the tool until it has read
+ the answer. Nothing is lost by waiting: a forced call is not a message being shown to anyone as it
+ arrives.
+
+ Reasoning deltas ride through untouched. The grammar binds the answer channel only, so a thinking
+ model still streams its thinking beside it."
+ [plan :- [:maybe ::plan]]
+ (when (= :grammar (:mechanism plan))
+ (let [mode (:mode plan)]
+ (fn [rf]
+ (let [buffer (StringBuilder.)
+ take! (fn [] (let [held (str buffer)] (.setLength buffer 0) held))
+ strip-content #(update-in % [:choices 0 :delta] dissoc :content)]
+ (fn
+ ;; a stream that ends without a finish chunk has no `stop` to vouch for the buffer, so it
+ ;; is dropped for the same reason a `length` finish drops it below
+ ([result]
+ (rf result))
+ ([result chunk]
+ (let [{:keys [delta finish_reason]} (get-in chunk [:choices 0])
+ content (not-empty (:content delta))]
+ ;; Taken before the chunk is judged, not instead of judging it: a build older than
+ ;; ollama/ollama#17485 puts the last content fragment and `finish_reason` on one chunk,
+ ;; so the two are not alternatives. Which build a server runs cannot be known here —
+ ;; the connection check probes with a non-streaming request.
+ (when content (.append buffer ^String content))
+ (cond
+ finish_reason
+ ;; Only a `stop` can become a call. On anything else the buffer holds a half-written
+ ;; answer, and `:structured` would mint a call from it regardless — its name is
+ ;; fixed, so nothing stops it — which reaches the caller as
+ ;; `structured-output-invalid` rather than the truncation that actually happened.
+ (let [held (take!)
+ call (when (= "stop" finish_reason) (forced-call-chunk mode held))]
+ (if call
+ ;; the answer was a call: emit it, then the finish chunk that closes it. The
+ ;; fragment left with the buffer, so leaving it on the chunk too would put the
+ ;; grammar's raw answer on the content channel, which is what this transducer
+ ;; exists to keep it off.
+ ;;
+ ;; the call carries the message `id` and `model` because it goes first: `:start` is
+ ;; built from the first chunk that has an id, and takes the message's model from it.
+ ;; Without them the call would open before the message it belongs to, and a one-chunk
+ ;; answer would report usage without a model. Two chunks from one, so the first `rf`
+ ;; may already have ended the reduction — a client that hung up mid-stream does
+ ;; exactly that.
+ (u/reduce-preserving-reduced
+ rf result
+ [(merge call (select-keys chunk [:id :model]))
+ (-> chunk
+ strip-content
+ (assoc-in [:choices 0 :finish_reason] "tool_calls"))])
+ ;; nothing became a call, so nothing may claim one: restating `stop` as
+ ;; `tool_calls` would promise the agent loop a call and hand it none. On `stop`
+ ;; the model answered outside the grammar, and that prose is the only answer
+ ;; there is, so the whole buffer goes back on the channel it arrived on rather
+ ;; than being swallowed. On `length` the buffer is a half-written call and
+ ;; `length` is already the diagnosis, so it is dropped — stripped first, or a
+ ;; build that shares the finish chunk would show the fragment that rode in on it.
+ (rf result (cond-> (strip-content chunk)
+ (and (= "stop" finish_reason) (seq held))
+ (assoc-in [:choices 0 :delta :content] held)))))
+
+ ;; the content is held, but the chunk it came on still goes on without it: Ollama opens
+ ;; with content rather than OpenAI's empty chunk (ollama/ollama#17485), so that chunk may
+ ;; be the one carrying the message `id` the `:start` is built from
+ :else (rf result (cond-> chunk content strip-content)))))))))))
+
+;;; ------------------------------------------------- Preflight --------------------------------------------------
+
+(def ^:private probe-schema
+ "A title-shaped schema for the structured-output probe. Deliberately the shape conversation titling
+ actually sends: a probe on a schema no caller uses could pass on a model that fails the real thing."
+ {:type "object"
+ :properties {:title {:type "string"
+ :description "A short title for the conversation."}}
+ :required ["title"]
+ :additionalProperties false})
+
+(def ^:private probe-messages
+ [{:role "user" :content "Give this conversation a short title. The user asked which orders shipped late."}])
+
+(mu/defn probe-body :- ::chat-completions-body
+ "The Chat Completions body fields for a structured-output probe of a model, `cloud?` saying whether
+ Ollama Cloud serves it.
+
+ Built by running a title-shaped request through the very pipeline a real one takes, rather than by
+ restating its shape: a probe that describes production by hand stops describing it the moment
+ production changes, which is exactly the failure this namespace exists to fix. Streaming is dropped
+ because the probe reads one whole answer, and the model is left to [[probe-chat!]]'s caller.
+
+ So self-hosted is held to a grammar and Cloud is asked in words and offered the tool — each probed
+ through the mechanism it will really use. Probing the other one proves nothing, being a path this
+ model never takes."
+ [cloud? :- :boolean]
+ (let [opts {:input probe-messages :schema probe-schema}
+ plan (plan opts cloud?)]
+ (-> (body-for plan (chat-completions/request-body (opts-for plan opts)))
+ (dissoc :model :stream :stream_options))))
+
+(mu/defn probe-verdict :- ::probe-verdict
+ "What a [[probe-body]] answer shows: nil when the mechanism held, `:truncated` when the answer was cut
+ off before it could, `:not-honored` when it simply was not.
+
+ Parsing is the check, not the presence of a reply: a schema honored in form but not in content is
+ still unusable by the callers this protects."
+ [cloud? :- :boolean
+ {:keys [message finish_reason]} :- ::choice]
+ (let [json-text (if cloud?
+ (get-in (first (:tool_calls message)) [:function :arguments])
+ (:content message))
+ parsed (try (json/decode+kw (str json-text)) (catch Exception _ nil))]
+ (cond
+ (and (map? parsed) (not-empty (str (:title parsed)))) nil
+ (= "length" finish_reason) :truncated
+ :else :not-honored)))
diff --git a/src/metabase/metabot/self/openai/chat_completions.clj b/src/metabase/metabot/self/openai/chat_completions.clj
index 3c42019d623..33524d3ce24 100644
--- a/src/metabase/metabot/self/openai/chat_completions.clj
+++ b/src/metabase/metabot/self/openai/chat_completions.clj
@@ -51,10 +51,11 @@
(= "assistant" (:role (first group))))
(let [tool-calls (into [] (mapcat :tool_calls) group)
;; :reasoning_content exists on a member only when the replay
- ;; hook of [[parts->cc-messages]] minted it — today only
- ;; Moonshot, whose dialect replays reasoning as a top-level
+ ;; hook of [[parts->cc-messages]] minted it — today Moonshot
+ ;; and Ollama, whose dialects replay reasoning as a top-level
;; sibling of :content (see
- ;; [[metabase.metabot.self.moonshot/reasoning-message]]).
+ ;; [[metabase.metabot.self.moonshot/reasoning-message]]; Ollama
+ ;; renames the field on the way out).
;; Joined in part order: the wire has a single field per
;; message, so order is the only fidelity available.
reasoning (apply str (keep :reasoning_content group))
@@ -100,9 +101,10 @@
;; channel passes :reasoning-part->message, a fn from a coalesced
;; :reasoning part to a replayed assistant message (or nil); today
;; Mistral (think chunks — see
- ;; [[metabase.metabot.self.mistral/think-message]]) and Moonshot
+ ;; [[metabase.metabot.self.mistral/think-message]]), Moonshot
;; (top-level reasoning_content — see
- ;; [[metabase.metabot.self.moonshot/reasoning-message]]) do. Z.AI and
+ ;; [[metabase.metabot.self.moonshot/reasoning-message]]) and Ollama
+ ;; (the same, renamed to reasoning on the way out) do. Z.AI and
;; vLLM define no such channel yet; when they grow one, each gets its
;; own hook fn here rather than more shared code.
:reasoning (when reasoning-part->message
@@ -128,6 +130,10 @@
:content (or (:content part) "")})))
merge-consecutive-assistant-messages)))
+(def structured-output-tool-name
+ "The tool a `:schema` request is expressed as."
+ "structured_output")
+
;;; Tool definition format
(defn- tool->cc-tool
@@ -401,7 +407,7 @@
all-tools (or (when schema
;; Structured output: force a tool call with the given JSON schema
[{:type "function"
- :function {:name "structured_output"
+ :function {:name structured-output-tool-name
:description "Output structured data"
:parameters schema}}])
(seq (mapv tool->cc-tool tools)))]
@@ -419,13 +425,8 @@
;;; Model catalog
-(defn models-catalog
- "Extract the model list from an OpenAI-compatible `GET /models` response, failing closed.
-
- `(get-in res [:body :data])` yields nil for any body shape we don't recognize — a base URL pointing at
- something that isn't a model endpoint, an HTML error page, a provider that renamed the key. Returning nil
- leaves the caller's whitelist intersection empty, so the admin Connect flow succeeds against a provider we
- never actually reached and leaves an empty model picker with no diagnostic. Throw instead.
+(defn malformed-catalog-ex
+ "The failure for a `GET /models` that answered with something other than a catalog.
`provider-name` is the display name, used in the message. The exception is tagged `:api-error` so the
adapter's surrounding [[metabase.metabot.self.core/rethrow-api-error!]] rethrows it unchanged, and
@@ -434,6 +435,27 @@
the admin still sees the sentence, but it bumps the unhandled-error counter and collapses to \"Something
went wrong\" under `MB_HIDE_STACKTRACES=true`, losing the diagnostic for the operators who enabled that.
+ `detail` is a sentence appended to the message, for a provider that has something more specific to say, and
+ `cause` the error that made the body unreadable, kept so a log shows whether it was HTML, empty or cut off."
+ ([provider-name] (malformed-catalog-ex provider-name nil))
+ ([provider-name detail] (malformed-catalog-ex provider-name detail nil))
+ ([provider-name detail cause]
+ (ex-info (cond-> (tru "{0} returned an unexpected model list response" provider-name)
+ detail (str ". " detail))
+ {:api-error true
+ :status-code 400
+ :error-code :malformed-model-catalog}
+ cause)))
+
+(defn models-catalog
+ "Extract the model list from an OpenAI-compatible `GET /models` response, failing closed.
+
+ `(get-in res [:body :data])` yields nil for any body shape we don't recognize — a base URL pointing at
+ something that isn't a model endpoint, an HTML error page, a provider that renamed the key. Returning nil
+ leaves the caller's whitelist intersection empty, so the admin Connect flow succeeds against a provider we
+ never actually reached and leaves an empty model picker with no diagnostic. Throw
+ [[malformed-catalog-ex]] instead.
+
A well-formed but empty `data` is a legitimate response — an account with no accessible models — and passes.
`:detail` is a sentence appended to the message, for a provider that has something more specific to say."
@@ -441,9 +463,5 @@
([provider-name res {:keys [detail]}]
(let [data (get-in res [:body :data])]
(when-not (sequential? data)
- (throw (ex-info (cond-> (tru "{0} returned an unexpected model list response" provider-name)
- detail (str ". " detail))
- {:api-error true
- :status-code 400
- :error-code :malformed-model-catalog})))
+ (throw (malformed-catalog-ex provider-name detail)))
data)))
diff --git a/src/metabase/metabot/self/registry.clj b/src/metabase/metabot/self/registry.clj
index 060af170803..f277f4b239a 100644
--- a/src/metabase/metabot/self/registry.clj
+++ b/src/metabase/metabot/self/registry.clj
@@ -14,6 +14,7 @@
[metabase.metabot.self.google :as google]
[metabase.metabot.self.mistral :as mistral]
[metabase.metabot.self.moonshot :as moonshot]
+ [metabase.metabot.self.ollama :as ollama]
[metabase.metabot.self.openai :as openai]
[metabase.metabot.self.openrouter :as openrouter]
[metabase.metabot.self.vllm :as vllm]
@@ -60,8 +61,9 @@
:stream - the adapter entry point, returning an AISDK stream.
:list-models - the provider's model listing, and the credential round trip behind Connect.
:supported-models - the allow-list a listing is intersected with. Absent for a provider that serves
- whatever the operator loaded (vLLM), names its own deployment (Azure), or has a
- catalog fixed in `metabase.llm.provider` (Google, and the managed connection).
+ whatever the operator loaded (vLLM, Ollama), names its own deployment (Azure), or
+ has a catalog fixed in `metabase.llm.provider` (Google, and the managed
+ connection).
:context-window - model -> its input context window in tokens.
:reasoning? - resolved model ref -> whether it streams its reasoning back to us.
:fast-mode? - resolved model ref -> whether it can be served in Anthropic fast mode.
@@ -71,8 +73,9 @@
every request. A second copy in this table would read as authoritative while changing nothing.
The two capability fns take the whole resolved ref rather than a model string because they do not all
- answer from the model: vLLM answers from what its connect-time probe recorded on the *connection*, and
- fast mode depends on whether the call is proxied. The adapters that do answer from the model name keep a
+ answer from the model: vLLM answers from what its connect-time probe recorded on the *connection*,
+ Ollama needs the credentials as well as the model because only its own server can say, and fast mode
+ depends on whether the call is proxied. The adapters that do answer from the model name keep a
small fn of their own so the table holds one shape rather than a per-provider argument list."
{"anthropic" {:stream #'claude/claude
:list-models #'claude/list-models
@@ -107,6 +110,9 @@
:supported-models #'moonshot/supported-models
:context-window #'moonshot/context-window-tokens
:reasoning? #'moonshot/streams-reasoning?}
+ "ollama" {:stream #'ollama/ollama
+ :list-models #'ollama/list-models
+ :reasoning? #'ollama/streams-reasoning?}
"openai" {:stream #'openai/openai
:list-models #'openai/list-models
:supported-models #'openai/supported-models
diff --git a/src/metabase/metabot/self/vllm.clj b/src/metabase/metabot/self/vllm.clj
index 7663dd38482..7c5ba338d28 100644
--- a/src/metabase/metabot/self/vllm.clj
+++ b/src/metabase/metabot/self/vllm.clj
@@ -18,6 +18,7 @@
[metabase.util.log :as log]
[metabase.util.malli :as mu])
(:import
+ (com.fasterxml.jackson.core JsonProcessingException)
(java.io IOException)
(java.net SocketTimeoutException)
(java.nio.charset StandardCharsets)
@@ -25,10 +26,6 @@
(set! *warn-on-reflection* true)
-(def ^:private min-context-length
- "Smallest `max_model_len` [[preflight!]] will accept. A product floor, not a measurement."
- 16384)
-
(defn- missing-base-url-ex []
(ex-info (tru "No vLLM base URL is set")
{:api-error true
@@ -90,15 +87,11 @@
{:socket-timeout (llm/llm-vllm-request-timeout-ms)
:connection-timeout (llm/llm-connection-timeout-ms)})
-(def ^:private probe-timeout-ceiling-ms
- "Upper bound on a single preflight probe, which blocks the admin behind a spinner."
- 120000)
-
(defn- probe-timeouts
"Timeouts for a preflight probe. An operator who lowers `llm-vllm-request-timeout-ms` below the
ceiling gets their own value."
[]
- {:socket-timeout (min (llm/llm-vllm-request-timeout-ms) probe-timeout-ceiling-ms)
+ {:socket-timeout (min (llm/llm-vllm-request-timeout-ms) adapter/probe-timeout-ceiling-ms)
:connection-timeout (llm/llm-connection-timeout-ms)})
;;; ----------------------------------------------- Transport errors ---------------------------------------------
@@ -138,55 +131,27 @@
No timeouts are passed: this is not a generation, so it rides [[core/request]]'s shared default
budget rather than vLLM's longer [[inference-timeouts]] one."
[{{:keys [base-url]} :credentials :as req}]
- (try
- (let [res (adapter/request! provider (assoc req
- :method :get
- :path "/models"
- :as :json))]
- ;; The URL off the request credentials, not the setting: a connect verifies them before they are saved.
- (chat-completions/models-catalog
- "vLLM" res
- {:detail (tru "Check that {0} is a vLLM server''s OpenAI-compatible API — the base URL should end in /v1."
- (str base-url))}))
- ;; Ordered ahead of the generic catch, which a non-2xx still reaches as an `ExceptionInfo`.
- ;; `:as :json` also lands a 2xx whose body is not JSON here, via Jackson's `JsonParseException`.
- (catch IOException e
- (throw (list-models-io-ex e base-url)))
- (catch Exception e
- (adapter/rethrow! provider e))))
+ ;; The URL off the request credentials, not the setting: a connect verifies them before they are saved.
+ (let [detail (tru "Check that {0} is a vLLM server''s OpenAI-compatible API — the base URL should end in /v1."
+ (str base-url))]
+ (try
+ (let [res (adapter/request! provider (assoc req
+ :method :get
+ :path "/models"
+ :as :json))]
+ (chat-completions/models-catalog "vLLM" res {:detail detail}))
+ ;; Ordered ahead of the `IOException` catch, which Jackson's parse error is one of: a 2xx whose
+ ;; body is not JSON means the server answered, so the address is wrong rather than unreachable.
+ (catch JsonProcessingException e
+ (throw (chat-completions/malformed-catalog-ex "vLLM" detail e)))
+ ;; Ordered ahead of the generic catch, which a non-2xx still reaches as an `ExceptionInfo`.
+ (catch IOException e
+ (throw (list-models-io-ex e base-url)))
+ (catch Exception e
+ (adapter/rethrow! provider e)))))
;;; -------------------------------------------------- Preflight -------------------------------------------------
-(def ^:private probe-tool
- {:type "function"
- :function {:name "record_table_name"
- :description "Record the name of the table the user mentioned."
- :parameters {:type "object"
- :properties {:table_name {:type "string"
- :description "The table name the user mentioned."}}
- :required ["table_name"]
- :additionalProperties false}}})
-
-(def ^:private probe-messages
- [{:role "user" :content "Record the table name: orders"}])
-
-(def ^:private probe-max-tokens
- "Generation ceiling for a preflight probe. High enough to clear a reasoning model's thinking, which
- is billed against it: a probe that stops at `length` before the tool call looks identical to a
- server that will not call tools at all."
- 2048)
-
-(def ^:private forced-tool-call-token-floor
- "Smallest `max_tokens` a cap on a forced tool call is raised to — below it a reasoning model spends
- the budget thinking and emits no tool call. Equal to [[probe-max-tokens]], which [[preflight!]]
- already proves the served model can clear."
- probe-max-tokens)
-
-(def ^:private reasoning-model-token-floor
- "Smallest `max_tokens` a cap is raised to once [[preflight!]] has observed the served model
- reasoning. Chat Completions bills thinking, answer, and tool call against one budget."
- 16384)
-
(def ^:private prompt-overhead-tokens
"Tokens [[estimated-prompt-tokens]] adds for what the chat template wraps around the messages and
tools: role markers, a tool-use preamble, the generation prompt."
@@ -215,12 +180,6 @@
catalog is cheap to serve, and the chat request waits for this lookup."
5000)
-(def ^:private default-temperature
- "Sampling temperature for a caller that supplies none. vLLM's own default is 1.0, which is wrong for
- the tool-calling and SQL-generation work the agent loop does. The hosted providers pick a sane
- default server-side; a self-hosted server does not, so the adapter supplies one."
- 0.3)
-
(defn- preflight-ex
"A preflight failure, tagged so `metabase.metabot.api` surfaces the message verbatim, not as a 500."
[msg]
@@ -231,7 +190,7 @@
(defn- probe-chat!
"Run one non-streaming Chat Completions turn against `model` and return the first choice. The
`finish_reason` is part of the return value because a generation truncated at
- [[probe-max-tokens]] and a server that will not call tools both produce empty `tool_calls`."
+ [[adapter/probe-max-tokens]] and a server that will not call tools both produce empty `tool_calls`."
[req model tool-choice]
(let [res (adapter/request! provider
(assoc req
@@ -239,20 +198,20 @@
:path "/chat/completions"
:as :json
:body (json/encode {:model model
- :messages probe-messages
- :tools [probe-tool]
+ :messages adapter/probe-messages
+ :tools [adapter/probe-tool]
:tool_choice tool-choice
:temperature 0
- :max_tokens probe-max-tokens}))
+ :max_tokens adapter/probe-max-tokens}))
(probe-timeouts))]
(get-in res [:body :choices 0])))
(defn- check-context-budget!
[{:keys [id max_model_len]}]
- (when (and max_model_len (< (long max_model_len) min-context-length))
+ (when (and max_model_len (< (long max_model_len) adapter/min-context-window-tokens))
(throw (preflight-ex
(tru "{0} is served with a {1} token context window, which is too small for Metabot — it needs at least {2}. Restart vLLM with a larger --max-model-len."
- (str id) (str max_model_len) (str min-context-length))))))
+ (str id) (str max_model_len) (str adapter/min-context-window-tokens))))))
(defn- check-tool-calling!
"Check that the server was started with `--enable-auto-tool-choice` and a `--tool-call-parser` whose
@@ -282,7 +241,7 @@
(throw (preflight-ex
(if truncated?
(tru "{0} reached the {1} token connection-test ceiling partway through a tool call. A model that generates this much before calling a tool is too slow to drive Metabot."
- (str model) (str probe-max-tokens))
+ (str model) (str adapter/probe-max-tokens))
(tru "The vLLM server returned a tool call whose arguments are not valid JSON. The --tool-call-parser most likely does not match {0}''s output format."
(str model))))))
(not (str/blank? reasoning)))
@@ -290,7 +249,7 @@
(and truncated? (not (str/blank? reasoning)))
(throw (preflight-ex
(tru "{0} spent the entire {1} token connection-test budget reasoning without calling a tool. A model that thinks this long about a trivial prompt is too slow to drive Metabot."
- (str model) (str probe-max-tokens))))
+ (str model) (str adapter/probe-max-tokens))))
;; A tool call cut off at the ceiling reaches here, not the `(seq tool-calls)` branch above: the
;; parsers extract from complete output, so a call missing its closing sentinel yields no
@@ -302,7 +261,7 @@
truncated?
(throw (preflight-ex
(tru "{0} reached the {1} token connection-test ceiling before completing a tool call. A model that generates this much before calling a tool is too slow to drive Metabot."
- (str model) (str probe-max-tokens))))
+ (str model) (str adapter/probe-max-tokens))))
:else
(throw (preflight-ex
@@ -318,7 +277,7 @@
(throw (preflight-ex
(if (= "length" finish_reason)
(tru "{0} reached the {1} token connection-test ceiling without producing a forced tool call. Metabot needs structured output support for conversation titles and SQL generation."
- (str model) (str probe-max-tokens))
+ (str model) (str adapter/probe-max-tokens))
(tru "The vLLM server did not honor a forced tool call. Metabot needs structured output support for conversation titles and SQL generation.")))))))
(defn- no-models-ex []
@@ -509,12 +468,12 @@
StandardCharsets/UTF_8))))
(defn- raise-to-floors
- "Raise `cap` to [[forced-tool-call-token-floor]] on a `forced?` call, and to
- [[reasoning-model-token-floor]] on a reasoning connection."
+ "Raise `cap` to [[adapter/forced-tool-call-token-floor]] on a `forced?` call, and to
+ [[adapter/reasoning-model-token-floor]] on a reasoning connection."
[cap forced? credentials]
(cond-> cap
- forced? (max forced-tool-call-token-floor)
- (reasoning-connection? credentials) (max reasoning-model-token-floor)))
+ forced? (max adapter/forced-tool-call-token-floor)
+ (reasoning-connection? credentials) (max adapter/reasoning-model-token-floor)))
(defn- output-cap
"The `max_tokens` to send with `body`, or nil to send none.
@@ -543,7 +502,7 @@
Matches what [[chat-completions/request-body]] emits, except that `max_tokens` is the one
[[output-cap]] picks for the context window `max-model-len` (nil when unknown), and `temperature`
- falls back to [[default-temperature]]. Both stay adapter-local rather than moving into the shared
+ falls back to [[adapter/default-temperature]]. Both stay adapter-local rather than moving into the shared
builder, which would also change Z.AI, Mistral, and OpenRouter.
Pure: [[vllm-raw]] looks the window up. The 1-arity, which Model Garden endpoints use, has none."
@@ -553,7 +512,7 @@
max-model-len :- [:maybe pos-int?]]
(let [forced? (or (some? schema) (= "required" (some-> tool_choice name)))
body (chat-completions/request-body (cond-> (dissoc opts :max-tokens)
- (nil? temperature) (assoc :temperature default-temperature)))
+ (nil? temperature) (assoc :temperature adapter/default-temperature)))
cap (output-cap body max-tokens forced? credentials max-model-len)]
(cond-> body
cap (assoc :max_tokens cap)))))
@@ -595,20 +554,6 @@
e)
(unreachable-ex e base-url {:retryable? false})))
-(defn- io-guarded
- "Wrap a stream reducible so a transport failure while consuming it surfaces as [[stream-io-ex]]
- rather than a raw `IOException`. The adapter's own `try` covers only establishing the request.
-
- Goes inside `core/reducible-with-api-errors`, never outside: [[stream-io-ex]] tags `:api-error
- true`, which `core/rethrow-api-error!` rethrows unchanged, so this translation wins for IO."
- [reducible timeout-ms]
- (reify clojure.lang.IReduceInit
- (reduce [_ rf init]
- (try
- (.reduce ^clojure.lang.IReduceInit reducible rf init)
- (catch IOException e
- (throw (stream-io-ex e timeout-ms)))))))
-
(mu/defn vllm-raw
"Perform a streaming request to a vLLM server's Chat Completions API.
Opts map takes `:credentials` (`{:base-url ... :api-key ...}`) from the connection serving this
@@ -623,7 +568,7 @@
{:path "/chat/completions"
:body (vllm-request-body opts (served-max-model-len opts))
:request-options (inference-timeouts)
- :wrap-stream #(io-guarded % timeout-ms)
+ :wrap-stream #(adapter/io-guarded % (fn [e] (stream-io-ex e timeout-ms)))
;; clj-http raises an `IOException` only when there is no response at all, so the
;; IO branch cannot swallow a failure the provider's own messages would have translated.
:on-request-error (fn [e]
diff --git a/test/metabase/cmd/ai_provider_dox_test.clj b/test/metabase/cmd/ai_provider_dox_test.clj
index fcd60555096..2d2a236659a 100644
--- a/test/metabase/cmd/ai_provider_dox_test.clj
+++ b/test/metabase/cmd/ai_provider_dox_test.clj
@@ -68,8 +68,9 @@
(is (nil? (#'ai-provider-dox/field-qualifier {})))))
(deftest ^:parallel field-options-sentence-test
- (testing "a short list of choices is spelled out"
- (is (= "One of: `OpenAI`, `Anthropic`."
+ (testing "a short list of choices is spelled out as the values that get stored — this sentence sits
+ next to the environment variable that can supply one, and that variable carries the value"
+ (is (= "One of: `openai`, `anthropic`."
(#'ai-provider-dox/field-options-sentence
{:options [{:value "openai" :label "OpenAI"} {:value "anthropic" :label "Anthropic"}]}))))
(testing "a long list is pointed at instead — but still says the field is a picker"
@@ -84,8 +85,9 @@
(deftest ^:parallel field-default-sentence-test
(testing "a plain default is shown as stored"
(is (= "Defaults to `https://example.com`." (#'ai-provider-dox/field-default-sentence base-url-field))))
- (testing "a default with `:options` is shown as the label the form displays, not the stored value"
- (is (= "Defaults to `OpenAI`."
+ (testing "a default with `:options` is shown as the stored value, which is what an environment
+ variable or a config file has to carry"
+ (is (= "Defaults to `openai`."
(#'ai-provider-dox/field-default-sentence
{:default "openai"
:options [{:value "openai" :label "OpenAI"} {:value "anthropic" :label "Anthropic"}]}))))
@@ -143,7 +145,7 @@
:label (deferred-tru "Service account key file")
:type :file
:show-when {:field :auth-method :value "key"}}]
- (is (= "**Service account key file**. Only when **Authentication method** is **Service account key**."
+ (is (= "**Service account key file**. Only when **Authentication method** is **Service account key** (`key`)."
(#'ai-provider-dox/field-entry keyfile (provider [method keyfile]))))))
(testing "a field optional here, required on Metabase Cloud, and useless without its partner says all three"
(let [fields [access-key-field secret-key-field]]
@@ -189,7 +191,7 @@
(is (str/includes? (#'ai-provider-dox/models-markdown (registry-entry "google"))
"| Model "))
(is (str/includes? (#'ai-provider-dox/models-markdown (registry-entry "vllm"))
- "whichever models your vLLM server is serving")))
+ "whichever models your vLLM server has available")))
(testing "Azure explains that the model comes from the deployment instead, naming the fields by their labels"
;; answer first: the block sits under a "Supported models:" label, so it leads with what you get rather than
;; with the catalog it hasn't got
@@ -233,7 +235,7 @@
(is (str/includes? markdown
(str "needs either **Service account key file**, or "
"**OAuth access token** and **Project ID**.")))
- (is (str/includes? markdown "Only when **Authentication method** is **Service account key**."))
+ (is (str/includes? markdown "Only when **Authentication method** is **Service account key** (`service-account-key`)."))
(testing "a field with no `:show-when` carries no condition of its own"
(let [project-id (->> (str/split-lines markdown)
(filter #(str/starts-with? % "- **Project ID**"))
diff --git a/test/metabase/llm/api/provider_test.clj b/test/metabase/llm/api/provider_test.clj
index 87c4ea0beac..890b21997ad 100644
--- a/test/metabase/llm/api/provider_test.clj
+++ b/test/metabase/llm/api/provider_test.clj
@@ -1,6 +1,7 @@
(ns metabase.llm.api.provider-test
(:require
[clj-http.client :as http]
+ [clojure.string :as str]
[clojure.test :refer [are deftest is testing use-fixtures]]
[medley.core :as m]
[metabase.llm.api.provider :as llm.api.provider]
@@ -13,7 +14,8 @@
[metabase.settings.models.setting.cache :as setting.cache]
[metabase.test :as mt]
[metabase.test.fixtures :as fixtures]
- [metabase.util.json :as json]))
+ [metabase.util.json :as json]
+ [metabase.util.log.capture :as log.capture]))
(use-fixtures :once (fixtures/initialize :db))
@@ -60,10 +62,10 @@
(testing "every provider type is listed with the credential fields a connection needs"
(let [types (mt/user-http-request :crowberto :get 200 "llm/provider-types")]
(is (= #{"anthropic" "openai" "openrouter" "mistral" "zai" "moonshot" "deepseek" "xai" "google" "azure"
- "bedrock" "vllm" "metabase"}
+ "bedrock" "vllm" "ollama" "metabase"}
(set (map :type types))))
(is (= ["anthropic" "openai" "openrouter" "mistral" "zai" "moonshot" "deepseek" "xai" "google" "azure"
- "bedrock" "vllm"]
+ "bedrock" "vllm" "ollama"]
(remove #{"metabase"} (map :type types)))
"the bring-your-own-key providers keep their registry order")
(is (=? {:type "anthropic"
@@ -412,6 +414,30 @@
{:type "anthropic" :config {:api-key "sk-ant-nope"}}))))
(is (= [] (llm.provider/connections)))))))
+(deftest a-refused-listing-logs-only-a-malformed-catalog-test
+ (testing "a model list that was not a catalog is logged with its cause, which the admin never sees"
+ (let [cause (Exception. "Unexpected character '<'")]
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models
+ (fn [& _]
+ (throw (ex-info "vLLM returned an unexpected model list response"
+ {:api-error true :status-code 400 :error-code :malformed-model-catalog}
+ cause)))]
+ (mt/with-temporary-setting-values [llm-providers []]
+ (log.capture/with-log-messages-for-level [messages [metabase.llm.api.provider :warn]]
+ (mt/user-http-request :crowberto :post 400 "llm/providers"
+ {:type "vllm" :config {:base-url "https://vllm.example.com/v1"}})
+ (is (some #(= cause (ex-cause (:e %))) (messages))))))))
+ (testing "any other refusal is not, since its ex-data carries the provider's response"
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models
+ (fn [& _]
+ (throw (ex-info "Invalid API key"
+ {:api-error true :status-code 401 :body {:secret "response"}})))]
+ (mt/with-temporary-setting-values [llm-providers []]
+ (log.capture/with-log-messages-for-level [messages [metabase.llm.api.provider :warn]]
+ (mt/user-http-request :crowberto :post 400 "llm/providers"
+ {:type "vllm" :config {:base-url "https://vllm.example.com/v1"}})
+ (is (empty? (messages))))))))
+
(deftest models-listing-does-not-probe-test
(testing "listing models is a page load; only a write may spend a generation on the operator's server"
(let [opts (atom nil)]
@@ -835,11 +861,98 @@
(mt/with-dynamic-fn-redefs [metabot.self/list-models (fn [& _]
(is false "the environment key must not leave")
{:models []})]
- (is (=? {:message "This connection's credentials come from environment variables. Change its base URL there too."}
+ (is (=? {:message (str "This connection's credentials come from environment variables, so its base URL "
+ "has to as well. Set MB_LLM_ANTHROPIC_API_BASE_URL.")}
(mt/user-http-request :crowberto :put 400 "llm/providers/anthropic"
{:config {:api-key "sk-ant-attempted-override"
:base-url "https://new.example.com"}})))))))
+(deftest update-keeps-a-location-the-environment-does-not-set-test
+ (testing "a location only picks among Google's own endpoints, so a key from the environment keeps it"
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models (fn [& _] {:models []})]
+ (mt/with-temporary-setting-values [llm-providers [(connection "google" "google"
+ {:project-id "my-project"
+ :location "europe-west4"})]]
+ (mt/with-temp-env-var-value! [mb-llm-google-oauth-access-token "ya29.env"]
+ (is (=? {:config {:location "europe-west4"}}
+ (mt/user-http-request :crowberto :put 200 "llm/providers/google" {:name "renamed"}))))))))
+
+(deftest update-does-not-verify-with-a-key-the-environment-moved-away-from-test
+ (testing (str "A connection the environment has moved no longer carries the key an admin entered for where "
+ "it used to point — a read of it drops that key. An edit merges its config from storage, so "
+ "without the same rule a rename would hand the key to the server it was moved to.")
+ (let [seen (atom [])]
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models (fn [& args] (swap! seen conj args) {:models []})]
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama"
+ {:base-url "http://ollama.internal:11434/v1"
+ :api-key "sk-own-proxy"})]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"]
+ (mt/user-http-request :crowberto :put "llm/providers/ollama" {:name "renamed"})
+ (is (not (str/includes? (pr-str @seen) "sk-own-proxy"))
+ "the key an admin entered for their own server is not sent to ollama.com")
+ (is (= "sk-own-proxy"
+ (get-in (first (llm.provider/stored-connections)) [:config :api-key]))
+ "while the stored key stays, so dropping the variable brings it back")))))))
+
+(deftest write-names-the-variable-a-captured-key-has-to-come-from-test
+ (testing (str "A credential entered for a connection the environment points elsewhere is dropped from what the "
+ "connection runs on, so without this the admin would be told the key is missing a moment after "
+ "entering it. The variables it has to come from, and the one that moved the connection, are named.")
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models (fn [& _]
+ (is false "a captured key must not leave")
+ {:models []})]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"]
+ (let [refused {:message (str "MB_LLM_OLLAMA_API_BASE_URL points this connection at another server, so "
+ "its credentials have to come from the environment as well. Set "
+ "MB_LLM_OLLAMA_API_KEY to keep using it.")
+ :error-code "llm-credentials-must-come-from-env"
+ :field "api-key"}]
+ (testing "editing a stored connection"
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama"
+ {:base-url "http://ollama.internal:11434/v1"})]]
+ (is (=? refused (mt/user-http-request :crowberto :put 400 "llm/providers/ollama"
+ {:config {:api-key "sk-new"}})))
+ (is (nil? (:api-key (stored-config "ollama")))
+ "and nothing was stored for the key the admin cannot use here")))))))
+ (testing "while a mask is the form echoing back a stored key rather than an admin entering one"
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models (fn [& _] {:models []})]
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama"
+ {:base-url "http://ollama.internal:11434/v1"
+ :api-key "sk-ollama"})]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "http://elsewhere.example.com:11434/v1"]
+ (is (=? {:name "renamed"}
+ (mt/user-http-request :crowberto :put 200 "llm/providers/ollama"
+ {:name "renamed"
+ :config {:api-key (setting/obfuscate-value "sk-ollama")}}))))))))
+
+(deftest env-gateway-base-url-with-a-key-typed-in-the-ui-keeps-working-test
+ (testing (str "A base-URL variable pointing a connection at a gateway, with the key typed in the UI: the setup "
+ "the docs give as the example. It keeps working on the types that always allowed it.")
+ (let [probed (atom [])]
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models (fn [_provider {:keys [credentials]}]
+ (swap! probed conj credentials)
+ {:models [{:id "m" :display_name "m"}]})]
+ (mt/with-temporary-setting-values [llm-providers [(connection "anthropic" "anthropic" {:api-key "sk-ant-ui"})]]
+ (mt/with-temp-env-var-value! [mb-llm-anthropic-api-base-url "https://gateway.example.com"]
+ (testing "the connection reads back usable, on the gateway, with its key"
+ (is (=? [{:key "anthropic"
+ :usable true
+ :env_fields ["base-url"]
+ :config {:api-key (setting/obfuscate-value "sk-ant-ui")
+ :base-url "https://gateway.example.com"}}]
+ (mt/user-http-request :crowberto :get 200 "llm/providers"))))
+ (testing "retyping the key in the connection form is accepted, and verified against the gateway"
+ (mt/user-http-request :crowberto :put 200 "llm/providers/anthropic" {:config {:api-key "sk-ant-new"}})
+ (is (=? {:api-key "sk-ant-new" :base-url "https://gateway.example.com"} (last @probed)))
+ (is (= "sk-ant-new" (:api-key (stored-config "anthropic")))))
+ (testing "and so is writing it through the per-provider setting"
+ (mt/user-http-request :crowberto :put 204 "setting/llm-anthropic-api-key" {:value "sk-ant-other"})
+ (is (= "sk-ant-other" (:api-key (stored-config "anthropic")))))))))
+ (testing "semantic search keeps the OpenAI key under an OpenAI base-URL variable"
+ (mt/with-temporary-setting-values [llm-providers [(connection "openai" "openai" {:api-key "sk-openai-ui"})]]
+ (mt/with-temp-env-var-value! [mb-llm-openai-api-base-url "https://gateway.example.com"]
+ (is (= "sk-openai-ui" (setting/get :llm-openai-api-key))))))))
+
(deftest generic-setting-api-cannot-write-provider-connections-test
(let [planted [(connection "anthropic" "anthropic" {:base-url "https://attacker.example.com"})]]
(mt/with-temporary-setting-values [llm-providers []]
@@ -862,7 +975,8 @@
(stored-config "anthropic")))
(testing "and explains when the credential must be moved through deployment configuration"
(mt/with-temp-env-var-value! [mb-llm-anthropic-api-key "sk-ant-env"]
- (is (=? {:message "This connection's credentials come from environment variables. Change its base URL there too."}
+ (is (=? {:message (str "This connection's credentials come from environment variables, so its base URL "
+ "has to as well. Set MB_LLM_ANTHROPIC_API_BASE_URL.")}
(mt/user-http-request :crowberto :put 400 "setting/llm-anthropic-api-base-url"
{:value "https://new.example.com"}))))))
(testing "and never plants a dormant value underneath an environment-owned base URL"
@@ -870,11 +984,24 @@
{:api-key "sk-ant-stored"
:base-url "https://api.anthropic.com"})]]
(mt/with-temp-env-var-value! [mb-llm-anthropic-api-base-url "https://env.example.com"]
- (is (=? {:message "This connection's base URL comes from an environment variable. Change it there."}
+ (is (=? {:message "This connection's base URL comes from an environment variable. Change it there."
+ :error-code "llm-base-url-is-env-managed"}
(mt/user-http-request :crowberto :put 400 "setting/llm-anthropic-api-base-url"
{:value "https://attacker.example.com"}))))
(is (= "https://api.anthropic.com" (:base-url (stored-config "anthropic")))
- "removing the environment overlay leaves the original stored URL, not the rejected one"))))
+ "removing the environment overlay leaves the original stored URL, not the rejected one")))
+ (testing (str "not even the value the variable already holds: storing it would change nothing today and "
+ "become the live address the day the operator drops the variable. This API can be that "
+ "strict because nothing echoes it — the connection settings cannot, since the form "
+ "resubmits every field it disables.")
+ (mt/with-temporary-setting-values [llm-providers [(connection "anthropic" "anthropic"
+ {:api-key "sk-ant-stored"})]]
+ (mt/with-temp-env-var-value! [mb-llm-anthropic-api-base-url "https://env.example.com"]
+ (is (=? {:error-code "llm-base-url-is-env-managed"}
+ (mt/user-http-request :crowberto :put 400 "setting/llm-anthropic-api-base-url"
+ {:value "https://env.example.com"}))))
+ (is (nil? (:base-url (stored-config "anthropic")))
+ "and with the overlay gone nothing was planted: the connection is back on the type's default"))))
(deftest legacy-credential-setting-refuses-a-connection-on-its-own-base-url-test
(testing (str "The per-provider settings write one field at a time, so a credential entered through them arrives "
@@ -890,11 +1017,19 @@
(mt/with-temporary-setting-values [llm-providers []]
(mt/user-http-request :crowberto :put 204 "setting/llm-anthropic-api-key" {:value "sk-ant-fresh"})
(is (= "sk-ant-fresh" (:api-key (stored-config "anthropic"))))))
- (testing "so is a base URL the environment supplies, which the operator chose"
+ (testing "so is a base URL the environment supplies on a type whose variable shadows only the address"
(mt/with-temporary-setting-values [llm-providers []]
(mt/with-temp-env-var-value! [mb-llm-anthropic-api-base-url "https://env.example.com"]
(mt/user-http-request :crowberto :put 204 "setting/llm-anthropic-api-key" {:value "sk-ant-fresh"})
- (is (= "sk-ant-fresh" (:api-key (stored-config "anthropic"))))))))
+ (is (= "sk-ant-fresh" (:api-key (stored-config "anthropic")))))))
+ (testing (str "but on Ollama, a key typed with nothing stored has no address recorded with it, so it cannot "
+ "follow the address the environment supplies. It has to come from the environment as well.")
+ (mt/with-temporary-setting-values [llm-providers []]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"]
+ (is (=? {:message #".*MB_LLM_OLLAMA_API_KEY.*"}
+ (mt/user-http-request :crowberto :put 400 "setting/llm-ollama-api-key" {:value "sk-fresh"})))
+ (is (nil? (:api-key (stored-config "ollama"))))))))
+
(deftest update-preserves-a-masked-service-account-key-test
(testing (str "re-saving a Google connection without touching the key file echoes back the mask of a JSON key "
"that ends in a newline — the stored key has to survive it rather than be replaced by the mask")
@@ -1580,3 +1715,104 @@
(setting/set! :llm-providers
[(assoc-in grandfathered [:config :base-url] "http://10.0.0.1/v1")])
(catch clojure.lang.ExceptionInfo e (ex-data e))))))))))))))
+
+(deftest update-requires-fresh-secrets-to-move-ollama-to-cloud-test
+ (testing (str "Moving a self-hosted connection to Ollama Cloud is a base URL change like any other, so it must "
+ "not carry a stored key the caller never supplied to ollama.com")
+ (mt/with-temp-env-var-value! [mb-llm-allowed-networks "allow-all"]
+ (mt/with-temporary-setting-values
+ [llm-providers [(connection "ollama" "ollama"
+ {:base-url "http://internal.example.com:11434/v1"
+ :api-key "ollama-stored"})]]
+ (let [probes (atom [])]
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models
+ (fn [_provider {:keys [credentials]}]
+ (swap! probes conj credentials)
+ {:models []})]
+ (testing "supplying nothing but the address is refused"
+ (is (=? {:message "Enter this connection's credentials again to point it at a different base URL."}
+ (mt/user-http-request :crowberto :put 400 "llm/providers/ollama"
+ {:config {:base-url "https://ollama.com/v1"}}))))
+ (testing "and so is echoing the stored key back masked"
+ (is (=? {:message "Enter this connection's credentials again to point it at a different base URL."}
+ (mt/user-http-request :crowberto :put 400 "llm/providers/ollama"
+ {:config {:base-url "https://ollama.com/v1" :api-key "**********ed"}}))))
+ (is (empty? @probes)
+ "the stored key was rejected before credential verification could send it to ollama.com")
+ (is (= {:base-url "http://internal.example.com:11434/v1"
+ :api-key "ollama-stored"}
+ (stored-config "ollama")))
+ (testing "a caller who holds a Cloud key may move it"
+ (mt/user-http-request :crowberto :put 200 "llm/providers/ollama"
+ {:config {:base-url "https://ollama.com/v1" :api-key "ollama-fresh"}})
+ (is (= ["ollama-fresh"] (map :api-key @probes))
+ "and only the freshly supplied key ever reaches Cloud"))))))))
+
+(deftest ollama-base-url-setting-cannot-move-a-connection-holding-a-stored-key-test
+ (testing "the per-provider settings write one field at a time, which is the other way to move a connection to Cloud"
+ (mt/with-temp-env-var-value! [mb-llm-allowed-networks "allow-all"]
+ (mt/with-temporary-setting-values
+ [llm-providers [(connection "ollama" "ollama"
+ {:base-url "http://internal.example.com:11434/v1"
+ :api-key "ollama-stored"})]]
+ (is (=? {:message "Use the provider connection settings to change the base URL and enter the credentials again."}
+ (mt/user-http-request :crowberto :put 400 "setting/llm-ollama-api-base-url"
+ {:value "https://ollama.com/v1"})))
+ (is (= "http://internal.example.com:11434/v1" (:base-url (stored-config "ollama"))))))))
+
+(deftest create-judges-the-connection-the-environment-will-leave-behind-test
+ (let [probed (atom [])]
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models
+ (fn [_provider {:keys [credentials]}]
+ (swap! probed conj credentials)
+ {:models [{:id "m" :display_name "m"}]})]
+ (testing (str "A key the environment supplies goes only where the environment points. Ollama has no default "
+ "address, so a create naming its own is refused before anything is probed.")
+ (mt/with-temporary-setting-values [llm-providers []]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-key "sk-operator"]
+ (is (=? {:message (str "This connection's credentials come from environment variables, so its base URL "
+ "has to as well. Set MB_LLM_OLLAMA_API_BASE_URL.")}
+ (mt/user-http-request :crowberto :post 400 "llm/providers"
+ {:type "ollama"
+ :config {:base-url "http://ollama.internal:11434/v1"}})))
+ (is (empty? @probed))
+ (is (empty? (llm.provider/stored-connections))))))
+ (testing (str "A base-URL variable brings the `ollama` connection into existence itself, so a create beside it "
+ "takes a key of its own, which no variable names, and is judged on its own config")
+ (mt/with-temporary-setting-values [llm-providers []]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"]
+ (let [created (mt/user-http-request :crowberto :post 200 "llm/providers"
+ {:type "ollama"
+ :config {:base-url "http://second.internal:11434/v1"
+ :api-key "sk-own-proxy"}})]
+ (is (=? {:key "ollama-2" :env_fields []} created))
+ (is (=? {:base-url "http://second.internal:11434/v1" :api-key "sk-own-proxy"} (last @probed))))))))))
+
+(deftest create-does-not-send-an-environment-key-to-an-address-the-admin-typed-test
+ (testing "a key the environment supplies goes only where the environment points, so a create naming its own address is refused before anything is probed"
+ (let [probed (atom [])]
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models (fn [_provider {:keys [credentials]}]
+ (swap! probed conj credentials)
+ {:models [{:id "m" :display_name "m"}]})]
+ (mt/with-temporary-setting-values [llm-providers []]
+ (mt/with-temp-env-var-value! [mb-llm-vllm-api-key "sk-operator"]
+ (is (=? {:message (str "This connection's credentials come from environment variables, so its base URL "
+ "has to as well. Set MB_LLM_VLLM_API_BASE_URL.")}
+ (mt/user-http-request :crowberto :post 400 "llm/providers"
+ {:type "vllm" :config {:base-url "https://elsewhere.example.com/v1"}}))
+ "naming the variable the operator has to set, since a create has no base URL of its own to change")
+ (is (not (str/includes? (pr-str @probed) "sk-operator")))
+ (is (empty? (llm.provider/stored-connections)))))))))
+
+(deftest create-selects-the-model-the-environment-leaves-test
+ (testing "Metabot is pointed at the deployment the connection runs on, not the one the form named"
+ (mt/with-dynamic-fn-redefs [metabot.self/list-models (fn [& _] {:models []})]
+ (mt/with-temporary-setting-values [llm-providers []]
+ (mt/with-temporary-raw-setting-values [llm-metabot-provider nil]
+ (mt/with-temp-env-var-value! [mb-llm-azure-deployment-name "prod"]
+ (mt/user-http-request :crowberto :post 200 "llm/providers"
+ {:type "azure"
+ :config {:api-key "azure-key"
+ :base-url "https://r.services.ai.azure.com/openai"
+ :deployment-name "dev"}})
+ (is (= "azure/openai/prod" (metabot.settings/llm-metabot-provider)))))))))
diff --git a/test/metabase/llm/provider_test.clj b/test/metabase/llm/provider_test.clj
index b48c349a880..64c9d5ceb4b 100644
--- a/test/metabase/llm/provider_test.clj
+++ b/test/metabase/llm/provider_test.clj
@@ -1,7 +1,7 @@
(ns metabase.llm.provider-test
(:require
[clojure.string :as str]
- [clojure.test :refer [deftest is testing use-fixtures]]
+ [clojure.test :refer [are deftest is testing use-fixtures]]
[medley.core :as m]
[metabase.llm.provider :as llm.provider]
[metabase.llm.settings :as llm.settings]
@@ -497,12 +497,15 @@
:env-vars #{"MB_LLM_GOOGLE_SERVICE_ACCOUNT_KEY"}
:env-fields #{:service-account-key}
:source :db}]
- (llm.provider/connections))))))
- (testing "a lone base-url variable reaches the stored connection's base URL"
- (mt/with-temporary-setting-values [llm-providers [(connection "anthropic" "anthropic" {:api-key "sk-ant-db"})]]
- (mt/with-temp-env-var-value! [mb-llm-anthropic-api-base-url "https://env.example.com"]
- (is (= {:api-key "sk-ant-db" :base-url "https://env.example.com"}
- (llm.provider/credentials "anthropic")))))))
+ (llm.provider/connections)))))))
+
+(deftest a-region-survives-keys-from-the-environment-test
+ (testing "the region only picks an AWS endpoint, so keys from the environment do not reset it to the default"
+ (mt/with-temporary-setting-values [llm-providers [(connection "bedrock" "bedrock" {:region "eu-central-1"})]]
+ (mt/with-temp-env-var-value! [mb-llm-bedrock-access-key-id "AKIAENV"
+ mb-llm-bedrock-secret-access-key "env-secret"]
+ (is (=? [{:key "bedrock" :config {:region "eu-central-1"}}]
+ (llm.provider/connections)))))))
(deftest connections-drop-a-stored-base-url-an-env-credential-would-reach-test
(testing (str "A base URL saved through the API is not where an environment-supplied credential gets sent: the "
@@ -538,6 +541,70 @@
(is (= {:api-key "vllm-env-key"} (llm.provider/credentials "vllm")))
(is (false? (llm.provider/connection-usable? "vllm")))))))
+(def ^:private self-hosted-ollama
+ "A self-hosted Ollama an admin gave a key to — a server of their own behind an authenticating proxy."
+ {:base-url "http://ollama.internal:11434/v1"
+ :api-key "sk-entered-for-our-own-server"})
+
+(deftest connections-drop-a-stored-secret-an-env-base-url-would-move-test
+ (testing (str "The other order: an address arriving from the environment over a credential already stored. A key "
+ "is entered for the server it will be sent to, so a variable pointing elsewhere does not take it along.")
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama" self-hosted-ollama)]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "http://elsewhere.example.com:11434/v1"]
+ (is (= {:base-url "http://elsewhere.example.com:11434/v1"} (llm.provider/credentials "ollama"))
+ "the key an admin entered for a server of their own does not go to the other one")
+ (testing "and the stored list still holds it, so removing the variable brings it back"
+ (is (= "sk-entered-for-our-own-server"
+ (get-in (first (llm.provider/stored-connections)) [:config :api-key])))))))
+ (testing "Ollama Cloud is one more address, so a variable naming it moves the connection the same way"
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama" self-hosted-ollama)]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"]
+ (is (nil? (:api-key (llm.provider/credentials "ollama")))
+ "a key typed for the operator's own server does not go to ollama.com"))))
+ (testing "an address the environment restates is not a move, so the key it was entered with stands"
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama" self-hosted-ollama)]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "http://ollama.internal:11434/v1"]
+ (is (= "sk-entered-for-our-own-server" (:api-key (llm.provider/credentials "ollama")))))))
+ (testing "an address the environment restates with a trailing slash is not a move either"
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama" self-hosted-ollama)]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "http://ollama.internal:11434/v1/"]
+ (is (= "sk-entered-for-our-own-server" (:api-key (llm.provider/credentials "ollama")))))))
+ (testing "the environment supplying both halves moves nothing it does not also credential"
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama" self-hosted-ollama)]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"
+ mb-llm-ollama-api-key "sk-operator-key"]
+ (is (= {:base-url "https://ollama.com/v1" :api-key "sk-operator-key"}
+ (llm.provider/credentials "ollama"))))))
+ (testing "a connection the llm-providers variable supplies is the operator's own, so its key stands"
+ (mt/with-temp-env-var-value! [mb-llm-providers (str "[{\"key\":\"ollama\",\"type\":\"ollama\","
+ "\"name\":\"Ollama\","
+ "\"config\":{\"base-url\":\"http://ollama.internal:11434/v1\","
+ "\"api-key\":\"sk-operator-key\"}}]")
+ mb-llm-ollama-api-base-url "http://elsewhere.example.com:11434/v1"]
+ (is (= "sk-operator-key" (:api-key (llm.provider/credentials "ollama")))))))
+
+(deftest env-base-url-moves-a-key-stored-without-an-address-test
+ (testing (str "A key stored with no address, on a type with no default, has no address recorded with it. Thus, "
+ "nothing shows that it is for the address the variable names.")
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama" {:api-key "sk-no-address"})]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"]
+ (is (= {:base-url "https://ollama.com/v1"} (llm.provider/credentials "ollama")))))))
+
+(deftest env-base-url-keeps-the-key-on-types-it-always-shadowed-test
+ (testing (str "on a type in the compatibility exemption, a base-URL variable shadows the address alone and a "
+ "key typed in the UI goes along with it, whatever the connection stored: the gateway setup an "
+ "upgrade must not break")
+ (doseq [[label stored-url] {"left blank" nil
+ "stored at the default" "https://api.anthropic.com"
+ "chosen by the admin" "https://chosen.example.com"}]
+ (testing label
+ (mt/with-temporary-setting-values [llm-providers [(connection "anthropic" "anthropic"
+ (cond-> {:api-key "sk-ant-db"}
+ stored-url (assoc :base-url stored-url)))]]
+ (mt/with-temp-env-var-value! [mb-llm-anthropic-api-base-url "https://gateway.example.com"]
+ (is (= {:api-key "sk-ant-db" :base-url "https://gateway.example.com"}
+ (llm.provider/credentials "anthropic")))))))))
+
(deftest stored-connections-keeps-a-connection-the-environment-shadows-test
(testing (str "The stored list keeps the credentials the environment shadows, so writes rebuild from here and "
"removing the env var brings the stored value back.")
@@ -690,7 +757,7 @@
"without updating them ships a provider that silently falls back to the generic icon. Update "
"both, then this list.")
(is (= #{"anthropic" "openai" "openrouter" "mistral" "zai" "moonshot" "deepseek" "xai" "google" "azure"
- "bedrock" "vllm" "metabase"}
+ "bedrock" "vllm" "ollama" "metabase"}
(into #{} (map :type) (llm.provider/provider-types))))))
(deftest ^:parallel provider-types-test
@@ -711,6 +778,11 @@
(testing "google's service account key is a file field, but it is the whole credential"
(is (= #{:service-account-key :oauth-access-token} (llm.provider/secret-field-keys "google"))))
(is (= #{} (llm.provider/secret-field-keys "metabase"))))
+ (testing "every type other than the managed one is always available"
+ (is (true? (llm.provider/type-available? "anthropic")))
+ (is (false? (llm.provider/type-available? "evilai")))))
+
+(deftest ^:parallel provider-type-models-test
(testing "every type's default model, which is what a first connection of that type gets selected for it"
(is (= {"anthropic" "claude-sonnet-4-6"
"openai" "gpt-5.4"
@@ -724,9 +796,10 @@
;; azure's models are deployment names the admin chooses, so there is nothing to default to
"azure" nil
"bedrock" "anthropic.claude-opus-4-8"
- ;; nor is there for vLLM, which serves whatever the operator loaded: connecting adopts the model
- ;; its probe exercised
+ ;; nor is there for the self-hosted types, which serve whatever the operator loaded or pulled:
+ ;; connecting adopts the model its probe exercised
"vllm" nil
+ "ollama" nil
"metabase" "anthropic/claude-sonnet-4-6"}
(into {} (map (juxt :type #(llm.provider/default-model (:type %)))) (llm.provider/provider-types))))
(is (nil? (llm.provider/default-model "evilai"))))
@@ -744,14 +817,40 @@
"google" nil
"azure" nil
"bedrock" "anthropic.claude-haiku-4-5"
- ;; a vLLM server serves the one model the operator loaded, so there is no cheaper tier to fall back to
+ ;; a self-hosted server serves the models the operator chose to host, so there is no cheaper tier
+ ;; to fall back to
"vllm" nil
+ "ollama" nil
"metabase" nil}
(into {} (map (juxt :type #(llm.provider/mini-model (:type %)))) (llm.provider/provider-types))))
- (is (nil? (llm.provider/mini-model "evilai"))))
- (testing "every type other than the managed one is always available"
- (is (true? (llm.provider/type-available? "anthropic")))
- (is (false? (llm.provider/type-available? "evilai")))))
+ (is (nil? (llm.provider/mini-model "evilai")))))
+
+(deftest ^:parallel env-base-url-shadowing-types-test
+ (testing (str "The types whose base-URL variable is exempt from tying credentials to their address. A type "
+ "added from now on starts outside it, so growing this set is a decision, not an accident.")
+ (is (= #{"anthropic" "azure" "deepseek" "google" "mistral" "moonshot" "openai" "openrouter" "vllm" "xai" "zai"}
+ @#'llm.provider/env-base-url-shadowing-types))))
+
+(deftest per-setting-write-refuses-a-key-the-environment-moved-away-from-test
+ (testing "a key written through the per-provider setting is refused, not stored for every read to drop"
+ (mt/with-temporary-setting-values [llm-providers [(connection "ollama" "ollama"
+ {:base-url "http://ollama.internal:11434/v1"})]]
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"]
+ (is (=? {:message #".*MB_LLM_OLLAMA_API_BASE_URL.*MB_LLM_OLLAMA_API_KEY.*"}
+ (mt/user-http-request :crowberto :put 400 "setting/llm-ollama-api-key" {:value "sk-new"})))
+ (is (nil? (get-in (first (llm.provider/stored-connections)) [:config :api-key])))))))
+
+(deftest typed-value-test
+ (testing "what counts as a value the caller really typed"
+ (are [submitted expected] (= expected (#'llm.provider/typed-value {:k submitted} :k))
+ "sk-new" "sk-new"
+ " sk-new " "sk-new"
+ nil nil
+ " " nil
+ "**********ew" nil
+ "**********ew " nil
+ ;; the mask of a newline-terminated JSON key file
+ "**********}\n" nil)))
(deftest ^:parallel served-mini-model-test
(testing "a listing that includes the type's mini model records it on the connection"
diff --git a/test/metabase/metabot/self/adapter_test.clj b/test/metabase/metabot/self/adapter_test.clj
index 621adee2922..3d7f3525360 100644
--- a/test/metabase/metabot/self/adapter_test.clj
+++ b/test/metabase/metabot/self/adapter_test.clj
@@ -15,6 +15,7 @@
[metabase.metabot.self.google :as google]
[metabase.metabot.self.mistral :as mistral]
[metabase.metabot.self.moonshot :as moonshot]
+ [metabase.metabot.self.ollama :as ollama]
[metabase.metabot.self.openai :as openai]
[metabase.metabot.self.openrouter :as openrouter]
[metabase.metabot.self.registry :as registry]
@@ -44,6 +45,7 @@
#'google/provider :metabot.google/request
#'mistral/provider :metabot.mistral/request
#'moonshot/provider :metabot.moonshot/request
+ #'ollama/provider :metabot.ollama/request
#'openai/provider :metabot.openai/request
#'openrouter/provider :metabot.openrouter/request
#'vllm/provider :metabot.vllm/request
@@ -59,7 +61,7 @@
(def ^:private expected-fallback-messages
"What each adapter renders for an HTTP status it has no specific message for.
- English only. Each adapter declares its own `:error-fallback` rather than sharing one parameterised
+ English only. The pre-split adapters declare their own `:error-fallback` rather than sharing one parameterised
msgid, so these strings keep the translations already shipped for them in `locales/*.po`; the msgids
themselves are guarded by review and by the extractor (`clojure -X:build i18n.enumerate/enumerate`),
not by this test, which only pins the rendered English."
@@ -70,6 +72,9 @@
#'google/provider "Google API error (HTTP 418)"
#'mistral/provider "Mistral API error (HTTP 418)"
#'moonshot/provider "Moonshot API error (HTTP 418)"
+ ;; the one adapter with no `:error-fallback` of its own: no shipped translation to keep, so it takes
+ ;; the shared msgid, which renders identically
+ #'ollama/provider "Ollama API error (HTTP 418)"
#'openai/provider "OpenAI API error (HTTP 418)"
#'openrouter/provider "OpenRouter API error (HTTP 418)"
#'vllm/provider "vLLM API error (HTTP 418)"
@@ -89,12 +94,25 @@
(is (= "Anthropic API error (HTTP 0)"
((:error-msg @#'claude/provider) {})))))
+(def ^:private legacy-error-fallbacks
+ "The adapters that declare their own `... API error (HTTP {0})` msgid as an `:error-fallback`. Most have
+ translations of it in `locales/*.po` and keep it until the shared parameterised template is translated too —
+ see [[metabase.metabot.self.adapter/provider]], which says the key exists for exactly that. xAI's has none
+ yet; it is here because it already declares one.
+
+ Enumerated rather than derived as \"everything except the new ones\": this set may only shrink, and an
+ adapter added from here on uses the shared template, so it must not be enrolled by default."
+ #{#'azure/provider #'bedrock/provider #'claude/provider #'deepseek/provider #'google/provider
+ #'mistral/provider #'moonshot/provider #'openai/provider #'openrouter/provider #'vllm/provider
+ #'xai/provider #'zai/provider})
+
(deftest every-descriptor-brings-its-own-translated-messages-test
- (testing "each adapter declares its own `:error-fallback` rather than inheriting the shared
- parameterised template, which ships no translations. The rendered English is identical either
- way, so `error-message-test` cannot tell the two apart — this can"
- (doseq [provider-var (keys expected-spans)]
- (is (fn? (:error-fallback @provider-var)) (str provider-var))))
+ (testing "exactly the adapters in `legacy-error-fallbacks` declare their own msgid. The
+ rendered English is identical either way, so `error-message-test` cannot tell the two apart —
+ this can, in both directions: a legacy adapter may not silently lose its msgid, and a new one
+ may not add another for translators"
+ (is (= legacy-error-fallbacks
+ (set (filter #(fn? (:error-fallback @%)) (keys expected-spans))))))
(testing "the proxy refusal deliberately does not get the same treatment: it stays one shared msgid,
because `:ai-proxy?` is only ever set for the managed connection, whose catalog names only
Anthropic models — the one provider the proxy can serve — so the refusal is unreachable"
diff --git a/test/metabase/metabot/self/chat_max_tokens_test.clj b/test/metabase/metabot/self/chat_max_tokens_test.clj
index 61262a5cba5..b2d358a4962 100644
--- a/test/metabase/metabot/self/chat_max_tokens_test.clj
+++ b/test/metabase/metabot/self/chat_max_tokens_test.clj
@@ -16,6 +16,7 @@
[metabase.metabot.self.google.stream-generate-content :as stream-generate-content]
[metabase.metabot.self.mistral :as mistral]
[metabase.metabot.self.moonshot :as moonshot]
+ [metabase.metabot.self.ollama :as ollama]
[metabase.metabot.self.openai :as openai]
[metabase.metabot.self.openrouter :as openrouter]
[metabase.metabot.self.vllm :as vllm]
@@ -70,6 +71,7 @@
:moonshot (get (moonshot/moonshot-request-body opts) :max_tokens omitted)
:mistral (get (mistral/mistral-request-body opts) :max_tokens omitted)
:vllm (get (vllm/vllm-request-body opts) :max_tokens omitted)
+ :ollama (get (ollama/ollama-request-body opts) :max_tokens omitted)
;; xAI sends the cap as max_completion_tokens
:xai (get (xai/xai-request-body opts) :max_completion_tokens omitted)
:gemini (get-in (stream-generate-content/request-body opts)
@@ -219,7 +221,11 @@
"grok-4.3" 32000}
;; no default cap without a known context window, which the pure builder is not given
;; (see vllm/vllm-request-body)
- :vllm {"Qwen/Qwen3-32B" omitted}})
+ :vllm {"Qwen/Qwen3-32B" omitted}
+ ;; the default with no known window, unlike vLLM: Ollama shortens a prompt that does not fit rather than
+ ;; refusing it. Self-hosted, and a Cloud model forwarded under its `-cloud` name
+ :ollama {"qwen3:4b" 32000
+ "gpt-oss:120b-cloud" 32000}})
(deftest ^:parallel chat-cap-is-the-flat-default-test
(testing "each surface sends the same default cap for every model, or deliberately none"
@@ -243,6 +249,7 @@
:openrouter "z-ai/glm-5.3" 32000
:zai "glm-5.3" 32000
:moonshot "kimi-k3" 32000
+ :ollama "qwen3:4b" 32000
:gemini "google/gemini-3.7-flash" 32000))
(testing "vLLM with no known window still sends no cap: a floor raises a caller's cap, it never adds one"
(is (= omitted (cap-for :vllm "Qwen/Qwen3-32B" {:schema {:type "object"}})))))
diff --git a/test/metabase/metabot/self/ollama/capabilities_test.clj b/test/metabase/metabot/self/ollama/capabilities_test.clj
new file mode 100644
index 00000000000..75b495f49b2
--- /dev/null
+++ b/test/metabase/metabot/self/ollama/capabilities_test.clj
@@ -0,0 +1,350 @@
+(ns metabase.metabot.self.ollama.capabilities-test
+ "The lookup that replaced a single reasoning flag on the connection. What matters here is that the
+ answer is per *model* — a connection serves as many as the operator has pulled, and Metabot and the
+ mini model need not be on the same one."
+ (:require
+ [clj-http.client :as http]
+ [clojure.core.cache :as cache]
+ [clojure.test :refer :all]
+ [metabase.metabot.self.ollama.capabilities :as ollama.capabilities]
+ [metabase.test :as mt]
+ [metabase.test.util :as tu]
+ [metabase.util :as u]
+ [metabase.util.json :as json]))
+
+(set! *warn-on-reflection* true)
+
+(def ^:private credentials
+ {:base-url "http://ollama.internal:11434/v1"})
+
+(def ^:private cloud-credentials
+ {:base-url "https://ollama.com/v1" :api-key "sk-cloud-key"})
+
+(defn- showing
+ "Stub `http/request` for `/api/show`, answering from `capabilities-by-model` — a model absent from it
+ gets a body with no `capabilities` key, which is what an Ollama too old to report them returns.
+ Records every request it saw, and which thread made it."
+ ([capabilities-by-model] (showing capabilities-by-model (atom [])))
+ ([capabilities-by-model seen]
+ (fn [{:keys [body] :as req}]
+ ;; the thread, so a test can tell a lookup made on the caller's thread from one handed to a future
+ (swap! seen conj (assoc req ::thread (Thread/currentThread)))
+ (let [model (:model (json/decode+kw (str body)))]
+ {:status 200
+ :body (cond-> {:model model}
+ (contains? capabilities-by-model model)
+ (assoc :capabilities (get capabilities-by-model model)))}))))
+
+(defn- with-stub!
+ "Call `f` with `handler` standing in for every HTTP request, the capability cache emptied on both
+ sides so that neither the lookups under test nor the next test's inherit an answer."
+ [handler f]
+ (ollama.capabilities/clear-cache!)
+ (try
+ (mt/with-dynamic-fn-redefs [http/request handler]
+ (f))
+ (finally
+ (ollama.capabilities/clear-cache!))))
+
+(defn- with-server!
+ "[[with-stub!]] against a `/api/show` stub, calling `f` with the atom of requests it received."
+ [capabilities-by-model f]
+ (let [seen (atom [])]
+ (with-stub! (showing capabilities-by-model seen) #(f seen))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Where the lookup goes
+;;; ──────────────────────────────────────────────────────────────────
+
+(deftest asks-ollamas-own-api-at-the-server-root-test
+ (testing "`capabilities` is not on the OpenAI-compatible surface, so the lookup drops the `/v1` the
+ connection's base URL ends in rather than posting to a path that does not exist"
+ (with-server! {}
+ (fn [seen]
+ (ollama.capabilities/reasoning-model? credentials "qwen3:8b")
+ (is (= [{:method :post
+ :url "http://ollama.internal:11434/api/show"
+ :body "{\"model\":\"qwen3:8b\"}"}]
+ (mapv #(select-keys % [:method :url :body]) @seen)))))))
+
+(deftest asks-cloud-at-its-own-address-test
+ (testing "Cloud serves the same endpoint, on the host that is not configurable, and takes the key"
+ (with-server! {}
+ (fn [seen]
+ (ollama.capabilities/reasoning-model? cloud-credentials "gpt-oss:20b")
+ (is (= "https://ollama.com/api/show" (:url (first @seen))))
+ (is (= "Bearer sk-cloud-key" (get-in (first @seen) [:headers "Authorization"])))))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Reasoning, per model
+;;; ──────────────────────────────────────────────────────────────────
+
+(deftest reasoning-is-answered-per-model-not-per-connection-test
+ (testing "one connection, two models: a single flag on the connection could only ever
+ describe one of them"
+ (with-server! {"gpt-oss:20b" ["completion" "tools" "thinking"]
+ "mistral-large-3:675b" ["completion" "tools" "vision"]}
+ (fn [_]
+ (is (true? (ollama.capabilities/reasoning-model? credentials "gpt-oss:20b")))
+ (is (false? (ollama.capabilities/reasoning-model? credentials "mistral-large-3:675b")))))))
+
+(deftest a-server-that-reports-nothing-reads-as-not-reasoning-test
+ (testing "an Ollama too old to report `capabilities` reads as not reasoning — a smaller token budget
+ rather than a wrong answer, since `reasoning` is forwarded whenever it appears"
+ (with-server! {}
+ (fn [_]
+ (is (false? (ollama.capabilities/reasoning-model? credentials "qwen3:8b")))))))
+
+(deftest an-unreachable-server-is-not-an-error-test
+ (testing "capabilities sharpen a decision the adapter can still make without them, so a server that
+ will not answer must not take down the request that asked"
+ (with-stub! (fn [_] (throw (ex-info "boom" {})))
+ (fn []
+ (is (false? (ollama.capabilities/reasoning-model? credentials "qwen3:8b")))
+ (testing "and rules no model out, rather than emptying the picker because Ollama was down"
+ (is (true? (ollama.capabilities/chat-capable? credentials "qwen3:8b"))))))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Caching
+;;; ──────────────────────────────────────────────────────────────────
+
+(deftest a-model-is-looked-up-once-test
+ (testing "the lookup is on the request path, so repeating it per request would be a round-trip per
+ message"
+ (with-server! {"gpt-oss:20b" ["completion" "tools" "thinking"]}
+ (fn [seen]
+ (dotimes [_ 3] (ollama.capabilities/reasoning-model? credentials "gpt-oss:20b"))
+ (is (= 1 (count @seen)))))))
+
+(deftest a-server-that-will-not-answer-is-asked-once-too-test
+ (testing "an unknown answer is cached like any other, so an old server is not asked once per request"
+ (with-server! {}
+ (fn [seen]
+ (dotimes [_ 3] (ollama.capabilities/reasoning-model? credentials "qwen3:8b"))
+ (is (= 1 (count @seen)))))))
+
+(deftest repointing-the-connection-retires-what-was-cached-test
+ (testing "the same tag on another server is another model"
+ (with-server! {"qwen3:8b" ["completion" "tools" "thinking"]}
+ (fn [seen]
+ (ollama.capabilities/reasoning-model? credentials "qwen3:8b")
+ (ollama.capabilities/reasoning-model? (assoc credentials :base-url "http://other:11434/v1") "qwen3:8b")
+ (is (= 2 (count @seen))))))
+ (testing "but storing what the connect path learned back onto the config it probed with does not —
+ that would empty the cache exactly when it was just filled"
+ (with-server! {"qwen3:8b" ["completion" "tools" "thinking"]}
+ (fn [seen]
+ (ollama.capabilities/reasoning-model? credentials "qwen3:8b")
+ (ollama.capabilities/reasoning-model? (assoc credentials :probed-model "qwen3:8b") "qwen3:8b")
+ (is (= 1 (count @seen)))))))
+
+(deftest a-full-cache-evicts-the-model-read-least-recently-test
+ (testing "reads count as use, so the models being asked about are the ones kept"
+ (let [threshold @#'ollama.capabilities/cache-threshold
+ model #(str "model-" %)
+ cached? #(cache/has? @@#'ollama.capabilities/capabilities-cache
+ (#'ollama.capabilities/cache-key credentials (model %)))]
+ (with-server! {}
+ (fn [_]
+ (doseq [i (range threshold)]
+ (ollama.capabilities/reasoning-model? credentials (model i)))
+ (ollama.capabilities/reasoning-model? credentials (model 0))
+ (ollama.capabilities/cached-reasoning-model? credentials (model 1))
+ ;; two past the threshold, so without reads counting both 0 and 1 would be evicted
+ (ollama.capabilities/reasoning-model? credentials (model threshold))
+ (ollama.capabilities/reasoning-model? credentials (model (inc threshold)))
+ (testing "the blocking reader's model survives"
+ (is (cached? 0)))
+ (testing "the cache-only reader's model survives"
+ (is (cached? 1)))
+ (testing "the oldest models nobody read since are the ones evicted"
+ (is (not (cached? 2)))
+ (is (not (cached? 3)))))))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; The cache-only read
+;;; ──────────────────────────────────────────────────────────────────
+
+(deftest the-public-setting-never-calls-ollama-on-the-callers-thread-test
+ (testing "`llm-metabot-supports-reasoning?` is public, so every client's page load reaches this — it
+ has to answer without the caller waiting on the operator's Ollama"
+ (with-server! {"gpt-oss:20b" ["completion" "tools" "thinking"]}
+ (fn [seen]
+ (let [answer (ollama.capabilities/cached-reasoning-model? credentials "gpt-oss:20b")]
+ (testing "a cold read answers immediately, from what it has"
+ (is (false? answer)))
+ (testing "and heals itself, so the next read is right without anyone asking it to"
+ (is (true? (tu/poll-until 5000 (ollama.capabilities/cached-reasoning-model? credentials "gpt-oss:20b")))))
+ (testing "with the lookup on some other thread — never the one that asked"
+ (is (seq @seen))
+ (is (not-any? #(= (Thread/currentThread) (::thread %)) @seen))))))))
+
+(deftest a-model-less-reference-asks-for-nothing-test
+ (testing "`llm-metabot-provider` can name a connection and no model. There is nothing to look up, and
+ nothing would ever be cached for it, so scheduling a lookup would hand a future the same
+ nothing to do on every page load."
+ (with-server! {"gpt-oss:20b" ["completion" "tools" "thinking"]}
+ (fn [seen]
+ (dotimes [_ 5] (is (false? (ollama.capabilities/cached-reasoning-model? credentials ""))))
+ (Thread/sleep 100)
+ (is (empty? @seen))))))
+
+(defn- lookup-in-flight?
+ "Whether this model's lookup is still running.
+
+ Per model: the in-flight map is process-wide, so any other server's refresh would answer for it."
+ [credentials model]
+ (contains? @@#'ollama.capabilities/in-flight
+ (#'ollama.capabilities/cache-key credentials model)))
+
+(defn- age-lookups-by!
+ "Backdate every entry by `ms`, so a test can put it either side of an interval.
+
+ Always relative: `:at` is a `u/start-timer`, a monotonic reading in nanoseconds counted from whenever
+ the machine decided to start counting. Nothing can be said about a fixed value of it."
+ [ms]
+ (swap! @#'ollama.capabilities/capabilities-cache
+ (fn [c]
+ (reduce (fn [acc [k v]] (cache/miss acc k (update v :at - (* ms 1000000))))
+ c
+ (into {} c)))))
+
+(defn- age-out-lookups!
+ "Backdate every entry past the refresh interval, so the next read finds it stale. Emptying the cache
+ would not do: that is forgetting, which is the thing under test."
+ []
+ (age-lookups-by! (inc @#'ollama.capabilities/refresh-after-ms)))
+
+(deftest a-stale-answer-is-served-not-waited-on-test
+ (testing (str "an entry's age says when to re-ask, not what to believe. The listing asks about "
+ "every model in the catalog, so with an answer already in hand there is nothing to "
+ "wait for — serve it and re-ask behind.")
+ (ollama.capabilities/clear-cache!)
+ (try
+ (mt/with-dynamic-fn-redefs [http/request (showing {"gpt-oss:20b" ["completion" "tools" "thinking"]})]
+ (is (true? (ollama.capabilities/reasoning-model? credentials "gpt-oss:20b"))))
+ (age-out-lookups!)
+ (is (true? (#'ollama.capabilities/stale?
+ (cache/lookup @@#'ollama.capabilities/capabilities-cache
+ (#'ollama.capabilities/cache-key credentials "gpt-oss:20b"))))
+ "the entry this test serves from has to be stale, or nothing below is exercised")
+ (let [attempts (atom 0)]
+ (mt/with-dynamic-fn-redefs [http/request (fn [_]
+ (swap! attempts inc)
+ (Thread/sleep 300)
+ (throw (ex-info "hang" {})))]
+ (testing "the blocking reader answers from what it has, without waiting on the server"
+ (let [timer (u/start-timer)
+ answer (ollama.capabilities/reasoning-model? credentials "gpt-oss:20b")]
+ (is (true? answer))
+ (is (> 200 (u/since-ms timer)))))
+ (testing "and the public setting does too"
+ (is (true? (ollama.capabilities/cached-reasoning-model? credentials "gpt-oss:20b"))))
+ (testing "once the re-ask has actually failed, the answer still stands — a server that
+ would not answer is not evidence that the model changed"
+ (is (tu/poll-until 5000 (and (pos? @attempts)
+ (not (lookup-in-flight? credentials "gpt-oss:20b")))))
+ (is (true? (ollama.capabilities/reasoning-model? credentials "gpt-oss:20b"))))))
+ (finally
+ (ollama.capabilities/clear-cache!)))))
+
+(deftest a-server-that-never-answered-is-re-asked-sooner-test
+ (testing (str "serving the last answer needs there to be one. A first lookup that fails records no "
+ "answer at all, so until it is re-asked every model behind that server reads as not "
+ "reasoning and runs on the smaller token budget — the shorter interval is how long "
+ "that lasts.")
+ ;; not written with `with-server!`: the failed entry has to survive into the second stub, and that
+ ;; helper empties the cache on the way in
+ (ollama.capabilities/clear-cache!)
+ (try
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (throw (ex-info "down" {})))]
+ (is (false? (ollama.capabilities/reasoning-model? credentials "gpt-oss:20b"))))
+ (testing "just short of the retry interval the failure still stands, and nothing is re-asked"
+ ;; ten seconds either side of the interval rather than a millisecond: the wall clock this test
+ ;; spends between backdating an entry and reading it counts towards that entry's age
+ (age-lookups-by! (- @#'ollama.capabilities/retry-after-ms 10000))
+ (let [seen (atom [])]
+ (mt/with-dynamic-fn-redefs [http/request (showing {"gpt-oss:20b" ["completion" "tools" "thinking"]} seen)]
+ (is (false? (ollama.capabilities/reasoning-model? credentials "gpt-oss:20b")))
+ (Thread/sleep 100)
+ (is (empty? @seen)))))
+ (testing "past it, the recovered server is asked and answers"
+ (age-lookups-by! 20000)
+ (mt/with-dynamic-fn-redefs [http/request (showing {"gpt-oss:20b" ["completion" "tools" "thinking"]})]
+ (is (true? (tu/poll-until 5000 (ollama.capabilities/reasoning-model? credentials "gpt-oss:20b"))))))
+ (finally
+ (ollama.capabilities/clear-cache!)))))
+
+(deftest an-ollama-that-answers-without-capabilities-is-not-re-asked-sooner-test
+ (testing (str "a build too old to report `capabilities` will answer the same way for as long as it "
+ "runs, so it keeps the long interval rather than being polled every minute")
+ (with-server! {}
+ (fn [seen]
+ (is (false? (ollama.capabilities/reasoning-model? credentials "qwen3:8b")))
+ (is (= 1 (count @seen)))
+ (age-lookups-by! (inc @#'ollama.capabilities/retry-after-ms))
+ (is (false? (ollama.capabilities/reasoning-model? credentials "qwen3:8b")))
+ (Thread/sleep 100)
+ (is (= 1 (count @seen)) "still the one lookup")
+ (testing "and the long interval does re-ask it"
+ (age-lookups-by! @#'ollama.capabilities/refresh-after-ms)
+ (is (false? (ollama.capabilities/reasoning-model? credentials "qwen3:8b")))
+ (is (tu/poll-until 5000 (= 2 (count @seen)))))))))
+
+(deftest a-cold-read-starts-one-lookup-however-many-ask-test
+ (testing "a burst of page loads against a cold cache must not each start their own lookup"
+ (with-server! {"gpt-oss:20b" ["completion" "tools" "thinking"]}
+ (fn [seen]
+ (dotimes [_ 20] (ollama.capabilities/cached-reasoning-model? credentials "gpt-oss:20b"))
+ (is (true? (tu/poll-until 5000 (ollama.capabilities/cached-reasoning-model? credentials "gpt-oss:20b"))))
+ (is (= 1 (count @seen)))))))
+
+(deftest a-cold-lookup-is-shared-by-every-caller-waiting-on-it-test
+ (testing "requests for a model with no entry yet share one lookup rather than each asking the server"
+ (let [seen (atom [])
+ handler (showing {"gpt-oss:20b" ["completion" "tools" "thinking"]} seen)]
+ (with-stub! (fn [req] (Thread/sleep 200) (handler req))
+ (fn []
+ (let [answers (tu/repeat-concurrently 10 #(ollama.capabilities/reasoning-model?
+ credentials "gpt-oss:20b"))]
+ (is (every? true? answers))
+ (is (= 1 (count @seen)))))))))
+
+(deftest lookups-in-front-of-the-caller-share-the-concurrency-bound-test
+ (testing "two catalog listings at once stay within one bound, not one bound each"
+ (let [open (atom 0)
+ peak (atom 0)
+ handler (showing {})
+ models (mapv #(str "model-" %) (range 24))]
+ (with-stub! (fn [req]
+ (swap! peak max (swap! open inc))
+ (try
+ (Thread/sleep 50)
+ (handler req)
+ (finally
+ (swap! open dec))))
+ (fn []
+ (run! deref [(future (ollama.capabilities/chat-capable-ids credentials models))
+ (future (ollama.capabilities/chat-capable-ids credentials (rseq models)))])
+ (is (<= @peak @#'ollama.capabilities/lookup-concurrency)))))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Chat-capable
+;;; ──────────────────────────────────────────────────────────────────
+
+(deftest embedding-models-are-not-chat-models-test
+ (with-server! {"nomic-embed-text" ["embedding"]
+ "llama2-uncensored" ["completion"]
+ "qwen3:8b" ["completion" "tools" "thinking"]}
+ (fn [_]
+ (testing "Ollama's OpenAI-compatible catalog lists embedding models alongside chat models and
+ marks neither; this is the only thing that tells them apart"
+ (is (false? (ollama.capabilities/chat-capable? credentials "nomic-embed-text")))
+ (is (true? (ollama.capabilities/chat-capable? credentials "qwen3:8b"))))
+ (testing "a completion model that cannot call tools is no use to the agent loop either"
+ (is (false? (ollama.capabilities/chat-capable? credentials "llama2-uncensored"))))))
+ (testing "and a server that reports nothing rules nothing out — an older Ollama goes on offering
+ what it always did"
+ (with-server! {}
+ (fn [_]
+ (is (true? (ollama.capabilities/chat-capable? credentials "nomic-embed-text")))))))
diff --git a/test/metabase/metabot/self/ollama/forced_calls_test.clj b/test/metabase/metabot/self/ollama/forced_calls_test.clj
new file mode 100644
index 00000000000..43721dcdda6
--- /dev/null
+++ b/test/metabase/metabot/self/ollama/forced_calls_test.clj
@@ -0,0 +1,300 @@
+(ns metabase.metabot.self.ollama.forced-calls-test
+ "The decision table for forcing a tool call on Ollama.
+
+ Covered through the adapter too, but pinned here as well: this is the seam the adapter hands its one
+ deployment fact across, and every wrong answer in the table is a silent failure rather than an
+ error — a missing grammar means structured output that works until it does not, and a grammar where
+ none was asked for means a model that can no longer answer in prose."
+ (:require
+ [clojure.test :refer :all]
+ [metabase.metabot.self.ollama.forced-calls :as forced]
+ [metabase.metabot.self.openai.chat-completions :as chat-completions]))
+
+(set! *warn-on-reflection* true)
+
+(def ^:private tools
+ [{:tool-name "search_facts"
+ :doc "Search for facts."
+ :schema [:=> [:cat [:map [:question :string]]] :any]
+ :fn identity}
+ {:tool-name "record_table_name"
+ :doc "Record a table name."
+ :schema [:=> [:cat [:map [:table_name :string]]] :any]
+ :fn identity}])
+
+(def ^:private schema {:type "object" :properties {:title {:type "string"}} :required ["title"]})
+
+(defn- decision [opts cloud?]
+ (select-keys (forced/plan opts cloud?) [:mode :mechanism]))
+
+(deftest ^:parallel plan-decision-table-test
+ (testing "self-hosted structured output is held to a grammar"
+ (is (= {:mode :structured :mechanism :grammar}
+ (decision {:schema schema} false))))
+ (testing "and so is a forced call over the caller's own tools"
+ (is (= {:mode :tool-union :mechanism :grammar}
+ (decision {:tools tools :tool_choice "required"} false))))
+ (testing "Cloud can only ask, for structured output"
+ (is (= {:mode :structured :mechanism :instruction}
+ (decision {:schema schema} true))))
+ (testing "and for a call over the caller's own tools too — `tool_choice` is discarded on the way
+ out, so without the asking a profile that renders nothing but tool results (`:sql`) would
+ take a prose answer it cannot show"
+ (is (= {:mode :tool-union :mechanism :instruction}
+ (decision {:tools tools :tool_choice "required"} true))))
+ (testing "an `auto` turn is never planned for: a grammar would silently promote it to `required`,
+ and the model could then never answer in prose again"
+ (is (nil? (forced/plan {:tools tools} false)))
+ (is (nil? (forced/plan {:tools tools :tool_choice "auto"} false)))
+ (is (nil? (forced/plan {:tools tools :tool_choice "auto"} true))))
+ (testing "`required` with no tools has nothing to choose among, so no mechanism can compel a call —
+ but it is still a plan, because the token budget has to cover what was asked for"
+ (is (= {:mode :tool-union :mechanism :none}
+ (decision {:tool_choice "required"} false)))
+ (is (nil? (:schema (forced/plan {:tool_choice "required"} false)))
+ "and emphatically no grammar: a union over no tools could not be satisfied")))
+
+(deftest ^:parallel every-forced-request-is-planned-for-test
+ (testing "the token floor reads `(some? plan)`, so a plan has to exist wherever a call was asked for
+ — including the cases nothing can enforce, which would otherwise lose their budget"
+ (are [opts cloud?] (some? (forced/plan opts cloud?))
+ {:schema schema} false
+ {:schema schema} true
+ {:tools tools :tool_choice "required"} false
+ {:tools tools :tool_choice "required"} true
+ {:tool_choice "required"} false))
+ (testing "and nowhere else"
+ (is (nil? (forced/plan {:tools tools} false)))
+ (is (nil? (forced/plan {} false)))))
+
+(deftest ^:parallel grammar-shapes-test
+ (testing "structured output constrains the decoder with the caller's schema, untouched"
+ (is (= schema (:schema (forced/plan {:schema schema} false)))))
+ (testing "a forced call becomes one `anyOf` arm per tool, each pinning the name to that tool so the
+ only text satisfying the grammar is a call to one of them"
+ (let [arms (get-in (forced/plan {:tools tools :tool_choice "required"} false) [:schema :anyOf])]
+ (is (= [["search_facts"] ["record_table_name"]]
+ (mapv #(get-in % [:properties :name :enum]) arms)))
+ (is (every? #(= ["name" "parameters"] (:required %)) arms))
+ (is (= [:question :table_name]
+ (mapv #(first (keys (get-in % [:properties :parameters :properties]))) arms))
+ "each arm carries that tool's own parameters, so arguments are constrained too")))
+ (testing "Cloud is given no schema to constrain with, because it would discard it"
+ (is (nil? (:schema (forced/plan {:schema schema} true))))))
+
+(deftest ^:parallel opts-and-body-follow-the-plan-test
+ (testing "under a grammar the shared builder's synthetic tool and `tool_choice` are redundant, and
+ its real-tool handling would hand the model tools this request never had"
+ (let [plan (forced/plan {:schema schema} false)
+ opts (forced/opts-for plan {:schema schema :tools tools :input []})]
+ (is (nil? (:schema opts)))
+ (is (nil? (:tools opts)))
+ (is (= "json_schema" (get-in (forced/body-for plan {}) [:response_format :type])))
+ (is (nil? (:tool_choice (forced/body-for plan {:tool_choice "auto"})))
+ "a body saying \"auto\" while being grammar-forced reads as the opposite of what it does")))
+ (testing "a forced call over real tools keeps them — the template still renders their definitions"
+ (let [plan (forced/plan {:tools tools :tool_choice "required"} false)]
+ (is (= tools (:tools (forced/opts-for plan {:tools tools :tool_choice "required"}))))))
+ (testing "Cloud asks in words, last, where an instruction carries furthest"
+ (let [plan (forced/plan {:schema schema} true)
+ opts (forced/opts-for plan {:schema schema :input [{:role "user" :content "hi"}]})]
+ (is (= [{:role "user" :content "hi"}
+ {:role "user" :content (forced/cloud-instruction [chat-completions/structured-output-tool-name])}]
+ (:input opts)))
+ (is (= schema (:schema opts)) "the tool the shared builder mints from it is all Cloud has")
+ (is (nil? (:response_format (forced/body-for plan {})))
+ "and a grammar Cloud discards is not sent")))
+ (testing "a Cloud forced call names the caller's own tools, since naming them is all the
+ instruction can do that the discarded `tool_choice` did not already say"
+ (let [plan (forced/plan {:tools tools :tool_choice "required"} true)
+ opts (forced/opts-for plan {:tools tools :tool_choice "required" :input []})]
+ (is (= [{:role "user"
+ :content "Answer by calling one of these tools: `search_facts`, `record_table_name`. Do not reply in chat."}]
+ (:input opts)))
+ (is (= tools (:tools opts)) "and keeps them, because they are what it just named")))
+ (testing "an unplanned request is passed through untouched, both ways"
+ (is (= {:tools tools :input []} (forced/opts-for nil {:tools tools :input []})))
+ (is (= {:tool_choice "auto"} (forced/body-for nil {:tool_choice "auto"})))))
+
+(deftest ^:parallel read-back-only-where-something-was-constrained-test
+ (testing "only a grammar puts its answer on the wrong channel, so only a grammar needs moving back"
+ (is (some? (forced/read-back-xf (forced/plan {:schema schema} false))))
+ (is (some? (forced/read-back-xf (forced/plan {:tools tools :tool_choice "required"} false))))
+ (is (nil? (forced/read-back-xf (forced/plan {:schema schema} true)))
+ "Cloud's tool call arrives the ordinary way and must not be touched")
+ (is (nil? (forced/read-back-xf (forced/plan {:tools tools :tool_choice "required"} true))))
+ (is (nil? (forced/read-back-xf nil)))))
+
+(deftest ^:parallel probe-verdict-reads-the-mechanism-that-will-run-test
+ (testing "self-hosted reads the content channel, where a grammar puts its answer"
+ (is (nil? (forced/probe-verdict false {:message {:content "{\"title\": \"Late orders\"}"}
+ :finish_reason "stop"})))
+ (is (= :not-honored (forced/probe-verdict false {:message {:content "How about \"Late orders\"?"}
+ :finish_reason "stop"})))
+ (testing "JSON alone is not the contract — the caller reads named fields out of it"
+ (is (= :not-honored (forced/probe-verdict false {:message {:content "{\"summary\": \"x\"}"}
+ :finish_reason "stop"}))))
+ (testing "and being cut off is a different diagnosis from not complying"
+ (is (= :truncated (forced/probe-verdict false {:message {:content "{\"title\": \"Late or"}
+ :finish_reason "length"})))))
+ (testing "Cloud reads the tool-call channel, because that is where its answer arrives"
+ (is (nil? (forced/probe-verdict true {:message {:tool_calls [{:function {:name "structured_output"
+ :arguments "{\"title\": \"Late orders\"}"}}]}
+ :finish_reason "tool_calls"})))
+ (is (= :not-honored (forced/probe-verdict true {:message {:content "Late orders"}
+ :finish_reason "stop"}))))
+ (testing "each deployment reads only its own channel: an answer on the other one is not compliance,
+ because it is not what the runtime will look at"
+ (is (= :not-honored (forced/probe-verdict false {:message {:tool_calls [{:function {:arguments "{\"title\": \"x\"}"}}]}
+ :finish_reason "tool_calls"})))
+ (is (= :not-honored (forced/probe-verdict true {:message {:content "{\"title\": \"x\"}"}
+ :finish_reason "stop"})))))
+
+;;; ------------------------------------------- Reading the answer back --------------------------------------------
+
+(defn- content-chunk
+ ([content] (content-chunk content nil))
+ ([content finish-reason]
+ {:choices [(cond-> {:index 0 :delta {:content content}}
+ finish-reason (assoc :finish_reason finish-reason))]}))
+
+(defn- finish-chunk [finish-reason]
+ {:choices [{:index 0 :delta {} :finish_reason finish-reason}]})
+
+(defn- read-back
+ "`chunks` put through the transducer `plan` calls for, as a vector."
+ [plan chunks]
+ (into [] (forced/read-back-xf plan) chunks))
+
+(defn- tool-calls [chunks]
+ (mapcat #(get-in % [:choices 0 :delta :tool_calls]) chunks))
+
+(defn- finish-reasons [chunks]
+ (keep #(get-in % [:choices 0 :finish_reason]) chunks))
+
+(def ^:private structured-plan (forced/plan {:schema schema} false))
+
+(def ^:private tool-union-plan (forced/plan {:tools tools :tool_choice "required"} false))
+
+(deftest ^:parallel read-back-moves-the-answer-onto-the-tool-channel-test
+ (testing "content is held back and re-emitted as one tool call, so the raw answer is never shown"
+ (let [out (read-back structured-plan
+ [(content-chunk "{\"title\": ")
+ (content-chunk "\"Late orders\"}")
+ (finish-chunk "stop")])]
+ (is (= [{:name "structured_output" :arguments "{\"title\": \"Late orders\"}"}]
+ (map :function (tool-calls out))))
+ (is (empty? (keep #(get-in % [:choices 0 :delta :content]) out))
+ "no content reaches the channel this transducer exists to keep it off")
+ (testing "and `stop` is restated as the tool call it actually was"
+ (is (= ["tool_calls"] (finish-reasons out)))))))
+
+(deftest ^:parallel read-back-reads-a-chunk-carrying-both-content-and-finish-reason-test
+ (testing (str "an Ollama build older than ollama/ollama#17485 puts the last content fragment and "
+ "`finish_reason` in one chunk, so taking the content must not end the chunk's "
+ "handling — the finish chunk still has to be emitted.")
+ (testing "the trailing fragment still lands in the call, and the finish chunk still closes it"
+ (let [out (read-back structured-plan
+ [(content-chunk "{\"title\": ")
+ (content-chunk "\"Late orders\"}" "stop")])]
+ (is (= [{:name "structured_output" :arguments "{\"title\": \"Late orders\"}"}]
+ (map :function (tool-calls out)))
+ "the fragment that shared the finish chunk is not lost")
+ (is (= ["tool_calls"] (finish-reasons out)))
+ (is (empty? (keep #(get-in % [:choices 0 :delta :content]) out))
+ "and the fragment does not also ride out on the content channel")))
+ (testing "a truncated answer stays diagnosable as truncation rather than as a broken call"
+ (let [out (read-back structured-plan
+ [(content-chunk "{\"title\": \"Late or" "length")])]
+ (is (= ["length"] (finish-reasons out))
+ "`length` reaches the caller as the diagnosis, not as a parse error over a half-written answer")
+ (is (empty? (tool-calls out))
+ (str "and no call is minted from the half-written buffer: `:structured`'s name is fixed, so it "
+ "would mint one regardless, and the caller would see `structured-output-invalid` instead "
+ "of the truncation"))
+ (is (empty? (keep #(get-in % [:choices 0 :delta :content]) out))
+ (str "nor does the fragment ride out as text. Prose goes back on the content channel only "
+ "for a `stop` the grammar failed to shape — on `length` the buffer is a half-written "
+ "call, and half a JSON object is not an answer to show anyone"))))))
+
+(deftest ^:parallel read-back-call-carries-the-message-id-test
+ (testing (str "The call goes out before the chunk it was read from, and `:start` is built from the first "
+ "chunk carrying a message id — so the call has to carry it, or it opens before its message.")
+ (let [out (read-back structured-plan
+ [(assoc (content-chunk "{\"title\": \"x\"}" "stop") :id "chatcmpl-1")])]
+ (is (= "chatcmpl-1" (:id (first out)))))))
+
+(deftest ^:parallel read-back-call-carries-the-model-test
+ (testing (str "`:start` takes the message's model from the chunk it is built from, which in a one-chunk "
+ "answer is the call — so the call has to carry the model, or usage reports none.")
+ (let [out (read-back structured-plan
+ [(assoc (content-chunk "{\"title\": \"x\"}" "stop") :id "chatcmpl-1" :model "gpt-oss:20b")])]
+ (is (= "gpt-oss:20b" (:model (first out)))))))
+
+(deftest ^:parallel read-back-stops-when-the-consumer-has-had-enough-test
+ (testing (str "The call and the finish chunk that closes it come from one input chunk, so the first is "
+ "handed downstream while the reduction may already be over — the writer ends it as soon "
+ "as the client hangs up.")
+ (let [seen (atom [])
+ rf (fn ([acc] acc)
+ ([acc chunk]
+ (swap! seen conj chunk)
+ (reduced acc)))]
+ (transduce (forced/read-back-xf structured-plan) rf nil
+ [(content-chunk "{\"title\": \"Late orders\"}" "stop")])
+ (is (= 1 (count @seen)) "the finish chunk is not emitted past the end of the reduction")
+ (is (= [{:name "structured_output" :arguments "{\"title\": \"Late orders\"}"}]
+ (map :function (tool-calls @seen)))
+ "and the one chunk that did go is the call"))))
+
+(deftest ^:parallel read-back-mints-no-call-from-a-stream-without-a-finish-chunk-test
+ (testing (str "a stream that ends without a finish chunk leaves no `stop` to vouch for the buffer, so "
+ "no call is minted from it — `:structured`'s name is fixed, so it would mint one from "
+ "half-written JSON and the caller would see `structured-output-invalid`")
+ (doseq [[plan fragment] [[structured-plan "{\"title\": \"Late or"]
+ [tool-union-plan "{\"name\": \"search_facts\", \"parame"]]]
+ (testing (:mode plan)
+ (let [out (read-back plan [(content-chunk fragment)])]
+ (is (empty? (tool-calls out)))
+ (is (empty? (keep #(get-in % [:choices 0 :delta :content]) out))))))))
+
+(deftest ^:parallel read-back-passes-on-the-chunks-it-holds-content-from-test
+ (testing (str "Ollama opens a stream with content rather than OpenAI's empty chunk "
+ "(ollama/ollama#17485), so a held content chunk may be the first one carrying the "
+ "message `id` — it has to go on, or the minted call precedes `:start`.")
+ (doseq [[plan answer] [[structured-plan "{\"title\": \"Late orders\"}"]
+ [tool-union-plan "{\"name\": \"search_facts\", \"parameters\": {\"question\": \"q\"}}"]]]
+ (testing (:mode plan)
+ (let [chunk (fn [content & [finish-reason]]
+ (assoc (content-chunk content finish-reason) :id "chatcmpl-1" :model "qwen2.5"))
+ types (into []
+ (comp (chat-completions/chat-completions->aisdk-chunks-xf) (map :type))
+ (read-back plan [(chunk (subs answer 0 5)) (chunk (subs answer 5))
+ (chunk "" "stop")]))]
+ (is (= :start (first types)))
+ (is (= [:tool-input-start :tool-input-delta :tool-input-available]
+ (filterv #{:tool-input-start :tool-input-delta :tool-input-available} types)))
+ (is (not-any? #{:text-start :text-delta} types)
+ "the held content still stays off the text channel"))))))
+
+(deftest ^:parallel a-union-answer-that-is-not-a-call-is-not-reported-as-one-test
+ (testing (str "`:tool-union` reads the tool's name out of the answer, so a model that answers in "
+ "prose yields no call. Restating `stop` as `tool_calls` would promise the agent loop "
+ "a call and hand it none, and dropping the content would swallow the only answer.")
+ (let [out (read-back tool-union-plan
+ [(content-chunk "No tool is needed ")
+ (content-chunk "for that." "stop")])]
+ (is (empty? (tool-calls out)))
+ (is (= ["stop"] (finish-reasons out))
+ "the finish reason stands: nothing was turned into a call")
+ (is (= "No tool is needed for that."
+ (apply str (keep #(get-in % [:choices 0 :delta :content]) out)))
+ "and the answer reaches the caller"))))
+
+(deftest ^:parallel read-back-leaves-an-unconstrained-stream-alone-test
+ (testing "without a grammar there is no transducer, so nothing is buffered or rewritten"
+ (is (nil? (forced/read-back-xf (forced/plan {:schema schema} true))))
+ (is (nil? (forced/read-back-xf nil))))
+ (testing "chunks carrying neither content nor a finish reason pass straight through"
+ (let [reasoning {:choices [{:index 0 :delta {:reasoning "thinking"}}]}]
+ (is (= [reasoning] (read-back structured-plan [reasoning]))))))
diff --git a/test/metabase/metabot/self/ollama_test.clj b/test/metabase/metabot/self/ollama_test.clj
new file mode 100644
index 00000000000..42c59f806e2
--- /dev/null
+++ b/test/metabase/metabot/self/ollama_test.clj
@@ -0,0 +1,1235 @@
+(ns metabase.metabot.self.ollama-test
+ "Deliberately narrower than [[metabase.metabot.self.vllm-test]] — the two adapters share a
+ transport, so what is covered here is what differs: telling Ollama Cloud's models apart, the catalog
+ shape Ollama returns, the context-window check, and the diagnoses."
+ (:require
+ [clj-http.client :as http]
+ [clojure.test :refer :all]
+ [medley.core :as m]
+ [metabase.llm.provider :as llm.provider]
+ [metabase.llm.settings :as llm.settings]
+ [metabase.metabot.self :as self]
+ [metabase.metabot.self.core :as self.core]
+ [metabase.metabot.self.debug :as debug]
+ [metabase.metabot.self.ollama :as ollama]
+ [metabase.metabot.self.ollama.capabilities :as ollama.capabilities]
+ [metabase.metabot.self.ollama.connection :as ollama.connection]
+ [metabase.test :as mt]
+ [metabase.util :as u]
+ [metabase.util.json :as json]
+ [metabase.util.log.capture :as log.capture])
+ (:import
+ (java.net SocketTimeoutException)))
+
+(set! *warn-on-reflection* true)
+
+(defn- no-unstubbed-http
+ "Refuse any HTTP request a test did not stub. Building a request body looks a model's capabilities
+ up, so without this every `ollama-request-body` test below would resolve `ollama.internal` for real
+ — on a resolver that wildcards, waiting out the socket timeout to do it. A refused lookup reads as
+ a server that would not answer, which is what the adapter degrades to anyway."
+ [thunk]
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (throw (java.net.UnknownHostException. "unstubbed")))]
+ (thunk)))
+
+(use-fixtures :each no-unstubbed-http)
+
+(defn- with-clean-capabilities!
+ "Call `f` against an empty capability cache, emptied again afterwards so a failed assertion cannot
+ leave one test's models behind for the next.
+
+ Deliberately not a fixture: the cache is process-wide, and emptying shared state from an `:each`
+ fixture would run against this namespace's `^:parallel` tests. `metabase/validate-deftest` rejects
+ that, and is right to — only the few tests that assert on what Ollama said about a model need it,
+ and none of those are parallel."
+ [f]
+ (ollama.capabilities/clear-cache!)
+ (try
+ (f)
+ (finally
+ (ollama.capabilities/clear-cache!))))
+
+;; not localhost: Metabase is a server, so a realistic self-hosted address is another host
+(def ^:private base-url "http://ollama.internal:11434/v1")
+
+(def ^:private credentials
+ "A self-hosted connection. No API key: a self-hosted Ollama is unauthenticated, so this is the
+ ordinary case rather than a degenerate one."
+ {:base-url base-url})
+
+(def ^:private cloud-credentials
+ "An Ollama Cloud connection: Cloud's own address, and the key it requires."
+ {:base-url "https://ollama.com/v1" :api-key "sk-cloud-key"})
+
+(def ^:private keyed-credentials
+ "A self-hosted Ollama behind a proxy that requires a key."
+ (assoc credentials :api-key "proxy-key"))
+
+(defn- captured-request
+ "Drive `list-models` against a stub and return the request it issued. Credentials go through
+ `with-field-defaults` first, as every real caller does."
+ [raw-config]
+ (let [seen (atom nil)]
+ (mt/with-dynamic-fn-redefs [http/request (fn [req]
+ (reset! seen req)
+ {:status 200 :body {:data []}})]
+ (ollama/list-models {:credentials (llm.provider/with-field-defaults "ollama" raw-config)}))
+ @seen))
+
+;;; The shape Ollama's OpenAI-compatible `/v1/models` actually returns: no `max_model_len`, no
+;;; `parent`, no `name`. Every field vLLM's adapter reads beyond `id` is absent here, which is why
+;;; the context-window check reads the window off `/api/ps` instead.
+(def ^:private ollama-catalog
+ [{:id "good-model" :object "model" :created 1786676106 :owned_by "library"}
+ {:id "chatty-model" :object "model" :created 1786676106 :owned_by "library"}])
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; ollama-request-body
+;;; ──────────────────────────────────────────────────────────────────
+
+(def ^:private fake-tool
+ {:tool-name "fake_tool"
+ :doc "Do a fake thing."
+ :schema [:=> [:cat [:map [:a :string]]] :any]
+ :fn identity})
+
+(def ^:private other-fake-tool
+ {:tool-name "other_fake_tool"
+ :doc "Do another fake thing."
+ :schema [:=> [:cat [:map [:b :int]]] :any]
+ :fn identity})
+
+(deftest ^:parallel request-body-applies-default-max-tokens-test
+ (testing "an explicit max_tokens is always sent — without one a looping small model consumes the
+ whole context window in a single call"
+ (is (= self.core/chat-max-output-tokens
+ (:max_tokens (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]}))))))
+
+(deftest ^:parallel request-body-caller-max-tokens-wins-test
+ (testing "a caller-supplied max-tokens is not overridden by the default"
+ (is (= 128
+ (:max_tokens (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :max-tokens 128}))))))
+
+(deftest ^:parallel request-body-raises-max-tokens-for-a-forced-tool-call-test
+ (testing "a forced tool call gets the token floor even when the caller asked for less — below it a
+ reasoning model spends the budget thinking and emits no call"
+ (testing "schema"
+ (is (= 2048
+ (:max_tokens (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :schema {:type "object"}
+ :max-tokens 128})))))
+ (testing "tool_choice required, even with no tools to choose among"
+ (is (= 2048
+ (:max_tokens (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :tool_choice "required"
+ :max-tokens 128})))))))
+
+(deftest request-body-raises-max-tokens-for-a-reasoning-model-test
+ (testing "a reasoning model gets the larger floor — thinking, answer and tool call are billed
+ against one budget"
+ (with-clean-capabilities!
+ (fn []
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] {:status 200
+ :body {:capabilities ["completion" "tools" "thinking"]}})]
+ (is (= 16384
+ (:max_tokens (ollama/ollama-request-body {:model "thinking-model"
+ :input [{:role :user :content "hi"}]
+ :credentials credentials
+ :max-tokens 128}))))))))
+ (testing "and a model Ollama does not call a thinking one keeps the caller's value"
+ (with-clean-capabilities!
+ (fn []
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] {:status 200
+ :body {:capabilities ["completion" "tools"]}})]
+ (is (= 128
+ (:max_tokens (ollama/ollama-request-body {:model "chat-model"
+ :input [{:role :user :content "hi"}]
+ :credentials credentials
+ :max-tokens 128})))))))))
+
+(deftest ^:parallel request-body-constrains-the-decoder-on-a-self-hosted-schema-test
+ (testing "Ollama discards `tool_choice`, so the shared builder's forced-tool framing is only a
+ suggestion. Self-hosted has a real substitute: compile the schema into a decoding grammar."
+ (let [body (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :schema {:type "object"}
+ :credentials credentials})]
+ (is (= {:type "json_schema"
+ :json_schema {:name "structured_output" :schema {:type "object"}}}
+ (:response_format body)))
+ (testing "and nothing is left for the model to call — a tool here would be the suggestion again"
+ (is (nil? (:tools body)))
+ (is (nil? (:tool_choice body)))))))
+
+(deftest ^:parallel request-body-forces-a-real-tool-call-with-a-union-grammar-test
+ (testing "a `:required-tool-call?` profile needs a call among its own tools, which no single schema
+ describes. The union Ollama's maintainers endorsed for this does: one arm per tool, each
+ pinning `name` to that tool, so the only way to satisfy the grammar is to call one."
+ (let [body (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :tools [fake-tool other-fake-tool]
+ :tool_choice "required"
+ :credentials credentials})
+ arms (get-in body [:response_format :json_schema :schema :anyOf])]
+ (is (= [["fake_tool"] ["other_fake_tool"]]
+ (mapv #(get-in % [:properties :name :enum]) arms)))
+ (is (every? #(= ["name" "parameters"] (:required %)) arms))
+ (testing "each arm carries that tool's own parameter schema, so arguments are constrained too"
+ (is (= [:a :b] (mapv #(-> % :properties :parameters :properties keys first) arms))))
+ (testing "the tools stay on the request — the template still has to render their definitions"
+ (is (= ["fake_tool" "other_fake_tool"] (mapv #(get-in % [:function :name]) (:tools body)))))
+ (testing "and the discarded `tool_choice` is dropped rather than sent as decoration"
+ (is (nil? (:tool_choice body)))))))
+
+(deftest ^:parallel request-body-leaves-an-auto-tool-call-unconstrained-test
+ (testing "a grammar makes a tool call unavoidable, which is `required` and emphatically not `auto` —
+ applying one to an ordinary turn would stop the model ever answering in text"
+ (let [body (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :tools [fake-tool]
+ :credentials credentials})]
+ (is (nil? (:response_format body)))
+ (is (= "auto" (:tool_choice body))))))
+
+(deftest ^:parallel request-body-asks-cloud-for-the-call-in-words-test
+ (testing "Ollama Cloud serves no structured outputs and discards `format` as silently as
+ `tool_choice`, so there is no grammar to reach for — the tool stays and the request says
+ what it wants. Nothing here pretends that is a guarantee."
+ (let [body (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :schema {:type "object"}
+ :credentials cloud-credentials})]
+ (is (nil? (:response_format body)))
+ (is (= ["structured_output"] (mapv #(get-in % [:function :name]) (:tools body))))
+ (is (= "Answer by calling the `structured_output` tool. Do not reply in chat."
+ (:content (last (:messages body))))
+ "last, where an instruction carries furthest"))))
+
+(deftest ^:parallel request-body-asks-a-cloud-model-on-a-self-hosted-server-in-words-test
+ (testing (str "a self-hosted server forwards a Cloud-tagged model to ollama.com, body and all, so the grammar "
+ "would be discarded there just the same — the model, not the connection, decides")
+ (let [body (ollama/ollama-request-body {:model "gpt-oss:120b-cloud"
+ :input [{:role :user :content "hi"}]
+ :schema {:type "object"}
+ :credentials credentials})]
+ (is (nil? (:response_format body)))
+ (is (= "Answer by calling the `structured_output` tool. Do not reply in chat."
+ (:content (last (:messages body)))))))
+ (testing "while a model the same server runs itself still gets the grammar"
+ (is (some? (:response_format (ollama/ollama-request-body {:model "qwen3:8b"
+ :input [{:role :user :content "hi"}]
+ :schema {:type "object"}
+ :credentials credentials}))))))
+
+(deftest ^:parallel served-by-cloud-test
+ (testing "Ollama Cloud's own address serves every model under its plain name"
+ (are [url] (true? (ollama.connection/served-by-cloud? {:base-url url} "gemma4:31b"))
+ "https://ollama.com/v1"
+ "https://OLLAMA.com/v1"
+ "https://api.ollama.com/v1"))
+ (testing "an address that merely contains the name is not it"
+ (are [url] (false? (ollama.connection/served-by-cloud? {:base-url url} "gemma4:31b"))
+ "https://ollama.com.example.org/v1"
+ "https://notollama.com/v1"
+ "http://ollama.internal:11434/v1"))
+ (testing "on any other server, a Cloud model says so in its tag, by Ollama's own rule"
+ (are [model expected] (= expected (ollama.connection/served-by-cloud? credentials model))
+ "gpt-oss:120b-cloud" true
+ "gemma4:cloud" true
+ "GPT-OSS:120B-CLOUD" true
+ "qwen3:8b" false
+ ;; the rule reads the tag, and a name with no tag has none
+ "my-model-cloud" false
+ ;; a registry port is not a tag
+ "registry.example.com:5000/foo-cloud" false
+ nil false)))
+
+(deftest ^:parallel request-body-leaves-an-unstructured-request-alone-test
+ (testing "the instruction is for structured requests only — appending it to ordinary chat would put
+ a tool the model does not have in front of it"
+ (let [body (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :credentials cloud-credentials})]
+ (is (= ["hi"] (mapv :content (:messages body))))
+ (is (nil? (:response_format body))))))
+
+(deftest ^:parallel request-body-supplies-a-default-temperature-test
+ (testing "a self-hosted server picks no sane default server-side, so the adapter supplies one"
+ (is (= 0.3
+ (:temperature (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]})))))
+ (testing "and a caller-supplied temperature wins"
+ (is (= 0
+ (:temperature (ollama/ollama-request-body {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :temperature 0}))))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Reasoning replay
+;;; ──────────────────────────────────────────────────────────────────
+
+(def ^:private reasoning-turn
+ "One agent-loop turn: reasoning, the tool call it led to, the result, the answer."
+ [{:role :user :content "how many orders?"}
+ {:type :reasoning :id "r1" :text "I should "}
+ {:type :reasoning :id "r1" :text "count them"}
+ {:type :tool-input :id "c1" :function "run_query" :arguments {:sql "select 1"}}
+ {:type :tool-output :id "c1" :result {:output "42"}}
+ {:type :text :text "42 orders."}])
+
+(defn- replayed-assistant
+ "The assistant message from a `reasoning-turn` request — the one thinking should land on."
+ [opts]
+ (second (:messages (ollama/ollama-request-body (merge {:model "good-model"
+ :input reasoning-turn}
+ opts)))))
+
+(deftest ^:parallel request-body-replays-in-turn-reasoning-test
+ (testing "reasoning lands on the assistant message carrying the tool call it led to"
+ (is (= [{:role "user" :content "how many orders?"}
+ {:role "assistant"
+ :content ""
+ :reasoning "I should count them"
+ :tool_calls [{:id "c1"
+ :type "function"
+ :function {:name "run_query" :arguments "{\"sql\":\"select 1\"}"}}]}
+ {:role "tool" :tool_call_id "c1" :content "42"}
+ {:role "assistant" :content "42 orders."}]
+ (:messages (ollama/ollama-request-body {:model "good-model" :input reasoning-turn})))))
+ (testing "renamed to `reasoning`, since Ollama ignores `reasoning_content` on the way in"
+ (is (not (contains? (replayed-assistant nil) :reasoning_content)))))
+
+(deftest ^:parallel request-body-replay-is-not-capability-gated-test
+ (testing "replay does not wait on knowing whether the model reasons"
+ (doseq [[label creds] {"a connection" credentials
+ "none at all" nil}]
+ (testing label
+ (is (= "I should count them"
+ (:reasoning (replayed-assistant {:credentials creds}))))))))
+
+(deftest ^:parallel request-body-strips-reasoning-when-the-caller-wants-none-test
+ (testing "when the caller wants no thinking, thinking already in the input is dropped too"
+ (is (not (contains? (replayed-assistant {:reasoning? false}) :reasoning)))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Credentials
+;;; ──────────────────────────────────────────────────────────────────
+
+(deftest a-connection-without-a-base-url-fails-rather-than-borrowing-the-setting-test
+ (testing "the connection's credentials are the only source — a setting configuring another
+ connection is not a fallback"
+ (mt/with-temporary-setting-values [llm.settings/llm-ollama-api-base-url "http://saved:11434/v1"]
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"No Ollama base URL is set"
+ (ollama/list-models {:credentials {}}))))))
+
+(deftest ai-proxy-is-unsupported-test
+ (testing "the managed proxy fronts hosted providers; a self-hosted server is reached directly"
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (throw (ex-info "should never be called" {})))]
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"AI proxy is not supported for Ollama"
+ (ollama/list-models {:credentials credentials :ai-proxy? true}))))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Model listing
+;;; ──────────────────────────────────────────────────────────────────
+
+(deftest list-models-passes-the-catalog-through-test
+ (testing "every pulled model is offered — there is no whitelist — in the order the server lists them"
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] {:status 200 :body {:data ollama-catalog}})]
+ (is (= {:models [{:id "good-model" :display_name "good-model"}
+ {:id "chatty-model" :display_name "chatty-model"}]}
+ (ollama/list-models {:credentials credentials}))))))
+
+(deftest list-models-keeps-a-tagged-model-id-intact-test
+ (testing "Ollama names models `family:tag`, and `llm-metabot-provider` stores the result as
+ `ollama/{id}` — the tag separator must survive the listing"
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] {:status 200 :body {:data [{:id "qwen3:14b"}]}})]
+ (is (= {:models [{:id "qwen3:14b" :display_name "qwen3:14b"}]}
+ (ollama/list-models {:credentials credentials}))))))
+
+(deftest list-models-fails-closed-on-a-body-that-is-not-a-catalog-test
+ (testing "a 2xx whose body carries no model list throws, naming the base URL — the likeliest cause
+ being a base URL that omits /v1"
+ (doseq [body [{:status "ok"} {:object "list"} 404]]
+ (testing (str "body " (pr-str body))
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] {:status 200 :body body})]
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"Ollama returned an unexpected model list response.*http://ollama\.internal:11434/v1"
+ (ollama/list-models {:credentials credentials}))))))))
+
+(deftest list-models-fails-closed-on-a-2xx-that-is-not-json-test
+ (testing "a 2xx whose body is not JSON — a proxy's HTML, or a base URL missing /v1 — is a server
+ that answered, so it reads as a bad address rather than an unreachable one. The stub throws
+ what `:as :json` throws: clj-http parses a 2xx whatever its content type."
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (json/decode "<html>404 Not Found</html>"))]
+ (is (=? {:error-code :malformed-model-catalog
+ :status-code 400}
+ (try
+ (ollama/list-models {:credentials credentials})
+ (catch clojure.lang.ExceptionInfo e (ex-data e)))))
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"Ollama returned an unexpected model list response.*http://ollama\.internal:11434/v1"
+ (ollama/list-models {:credentials credentials}))))))
+
+(deftest list-models-keeps-the-parse-error-as-the-cause-test
+ (testing "the parse error travels as the cause, so a log shows whether the body was HTML, empty or cut off"
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (json/decode "<html>404 Not Found</html>"))]
+ (is (instance? com.fasterxml.jackson.core.JsonProcessingException
+ (try (ollama/list-models {:credentials credentials})
+ (catch clojure.lang.ExceptionInfo e (ex-cause e))))))))
+
+(deftest unreachable-server-names-the-address-it-tried-test
+ (testing "a transport failure names the address actually called, so a Cloud connection — which
+ carries no base URL of its own — does not report a blank one"
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (throw (java.net.ConnectException. "refused")))]
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"Could not reach Ollama at http://ollama\.internal:11434/v1"
+ (ollama/list-models {:credentials credentials})))
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"Could not reach Ollama at https://ollama\.com/v1"
+ (ollama/list-models {:credentials cloud-credentials}))))))
+;;; ──────────────────────────────────────────────────────────────────
+;;; Preflight
+;;; ──────────────────────────────────────────────────────────────────
+
+(defn- probe-kind
+ "Which probe a stubbed `POST /chat/completions` is serving, read off the body.
+
+ Never off `tool_choice`: Ollama discards that field, so a stub that answered differently for it
+ would let a check that proves nothing look like one that proves something."
+ [req]
+ (if (or (:response_format req)
+ (some #(= "structured_output" (get-in % [:function :name])) (:tools req)))
+ :structured
+ :tools))
+
+(defn- probing-server
+ "Stub `http/request` for the preflight path: `GET /models` returns `models`, `POST /api/show` reports
+ each model's `:capabilities` from the catalog entry (absent where the entry has none, as an Ollama
+ too old to report them answers), `GET /api/ps` reports the window each model was loaded with from
+ its `:context-length` (so a catalog entry without one is a server that will not say), and each
+ `POST /chat/completions` returns whatever `choice-by-probe` holds for the probe it was sent, so the
+ two probes can disagree — or, where that is a function, what it returns for the model probed."
+ [models choice-by-probe]
+ (fn [{:keys [url body]}]
+ (cond
+ (re-find #"/models$" (str url))
+ {:status 200 :body {:data models}}
+
+ (re-find #"/api/show$" (str url))
+ (let [model (:model (json/decode+kw (str body)))
+ entry (m/find-first #(= model (:id %)) models)]
+ {:status 200 :body (cond-> {:model model}
+ (:capabilities entry) (assoc :capabilities (:capabilities entry)))})
+
+ (re-find #"/api/version$" (str url))
+ {:status 200 :body {:version "0.12.0"}}
+
+ (re-find #"/api/ps$" (str url))
+ {:status 200 :body {:models (for [{:keys [id context-length]} models
+ :when context-length]
+ {:model id :name id :context_length context-length})}}
+
+ :else
+ (let [decoded (json/decode+kw (str body))
+ choice (get choice-by-probe (probe-kind decoded))]
+ {:status 200 :body {:choices [(if (fn? choice) (choice (:model decoded)) choice)]}}))))
+
+(def ^:private tool-calling-message
+ {:content ""
+ :tool_calls [{:id "call-1"
+ :type "function"
+ :function {:name "record_table_name" :arguments "{\"table_name\": \"orders\"}"}}]})
+
+(def ^:private constrained-message
+ "What a self-hosted server honoring `response_format` returns: the JSON as ordinary content, with no
+ tool call anywhere in sight."
+ {:content "{\"title\": \"Late orders\"}"})
+
+(def ^:private structured-tool-message
+ "What Cloud returns when the model takes the instruction it was given in place of a forced call."
+ {:content ""
+ :tool_calls [{:id "call-2"
+ :type "function"
+ :function {:name "structured_output" :arguments "{\"title\": \"Late orders\"}"}}]})
+
+(def ^:private structured-success
+ {:message constrained-message :finish_reason "stop"})
+
+(defn- probe-choice!
+ "Drive a probing connect against stubbed probe answers. `creds` decides which structured-output probe
+ preflight runs, so it decides which of the two answers is reached."
+ ([models tool-calling-choice structured-choice]
+ (probe-choice! models tool-calling-choice structured-choice credentials))
+ ([models tool-calling-choice structured-choice creds]
+ (mt/with-dynamic-fn-redefs [http/request (probing-server models {:tools tool-calling-choice
+ :structured structured-choice})]
+ (ollama/list-models {:credentials creds :probe? true}))))
+
+(defn- probe!
+ [models chat-message]
+ (probe-choice! models {:message chat-message :finish_reason "tool_calls"} structured-success))
+
+(deftest preflight-passes-on-a-model-that-calls-tools-test
+ (testing "a model that returns a well-formed tool call passes and is adopted as the one to run on"
+ (is (= {:models [{:id "good-model" :display_name "good-model"}]
+ :connection-info {:probed-model "good-model"}}
+ (probe! [{:id "good-model"}] tool-calling-message)))))
+
+(deftest preflight-skips-models-that-cannot-chat-test
+ (testing "Ollama lists models newest-first and its OpenAI-compatible listing does not filter out
+ embedding models, so a newest pull that is one would fail the connect outright — with no
+ way out, since the form hides the model picker for a type whose catalog is not fixed"
+ (with-clean-capabilities!
+ (fn []
+ (is (= "thinking-model"
+ (get-in (probe! [{:id "embedding-model" :capabilities ["embedding"]}
+ {:id "thinking-model" :capabilities ["completion" "tools" "thinking"]}]
+ tool-calling-message)
+ [:connection-info :probed-model]))))))
+ (testing "a server that reports no capabilities rules nothing out, and keeps taking the first entry"
+ (with-clean-capabilities!
+ (fn []
+ (is (= "first-model"
+ (get-in (probe! [{:id "first-model"} {:id "second-model"}] tool-calling-message)
+ [:connection-info :probed-model])))))))
+
+(defn- probe-by-model!
+ "Drive a probing connect where the tool-calling probe answers per model from `tool-message-by-model`."
+ [models tool-message-by-model opts]
+ (with-clean-capabilities!
+ (fn []
+ (mt/with-dynamic-fn-redefs [http/request (probing-server
+ models
+ {:tools (fn [model]
+ {:message (get tool-message-by-model model)
+ :finish_reason "tool_calls"})
+ :structured structured-success})]
+ (ollama/list-models (merge {:credentials credentials :probe? true} opts))))))
+
+(deftest preflight-moves-on-from-a-model-that-fails-the-probes-test
+ (let [models [{:id "qwen3:0.6b"} {:id "qwen3:14b"}]
+ by-model {"qwen3:0.6b" {:content "Sure! The table is orders."}
+ "qwen3:14b" tool-calling-message}]
+ (testing "the connect path names no model and the form offers no picker, so a newest model that cannot
+ drive Metabot hands over to the next chat model rather than leaving the admin stuck"
+ (is (= "qwen3:14b"
+ (get-in (probe-by-model! models by-model {}) [:connection-info :probed-model]))))
+ (testing "a model asked for by name is the only one probed"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"qwen3:0\.6b answered with text instead of calling a tool"
+ (probe-by-model! models by-model {:model "qwen3:0.6b"}))))
+ (testing "an edit's `:proposed-model` is a guess, not a request: a stored model that no longer passes
+ hands over to the next chat model as on connect, since the form offers no picker"
+ (is (= "qwen3:14b"
+ (get-in (probe-by-model! models by-model {:proposed-model "qwen3:0.6b"})
+ [:connection-info :probed-model])))))
+ (testing "when every model tried fails, the error names them and gives the newest one's reason"
+ (let [ids (map #(str "m" %) (range 5))]
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"None of the models tried can drive Metabot \(m0, m1, m2\)\. m0 answered with text"
+ (probe-by-model! (map #(hash-map :id %) ids) (zipmap ids (repeat {:content "Sure!"})) {})))))
+ (testing "a timeout ends the search at the first model: the next one would load just as slowly"
+ (let [probed (atom [])]
+ (with-clean-capabilities!
+ (fn []
+ (mt/with-dynamic-fn-redefs [http/request (let [serve (probing-server [{:id "m0"} {:id "m1"}] {})]
+ (fn [{:keys [url body] :as req}]
+ (if (re-find #"/chat/completions$" (str url))
+ (do (swap! probed conj (:model (json/decode+kw (str body))))
+ (throw (SocketTimeoutException. "Read timed out")))
+ (serve req))))]
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"^Ollama did not answer the connection test"
+ (ollama/list-models {:credentials credentials :probe? true})))
+ (is (= ["m0"] @probed)))))))
+ (let [connect (fn [failing-status]
+ (let [probed (atom [])
+ tools (fn [model]
+ (swap! probed conj model)
+ (if (= "m0" model)
+ (throw (ex-info "clj-http: status" {:status failing-status
+ :body "{\"error\":\"m0\"}"}))
+ {:message tool-calling-message :finish_reason "tool_calls"}))]
+ (with-clean-capabilities!
+ (fn []
+ {:result (try (get-in (probe-choice! [{:id "m0"} {:id "m1"}] tools structured-success)
+ [:connection-info :probed-model])
+ (catch clojure.lang.ExceptionInfo e (ex-message e)))
+ :probed @probed}))))]
+ (testing "a 500 is how Ollama answers for a model too large to load, so the search moves on"
+ (is (= {:result "m1" :probed ["m0" "m1"]} (connect 500))))
+ (testing "a 400 is how Ollama answers for a model with no tool support when capabilities are unknown"
+ (is (= {:result "m1" :probed ["m0" "m1"]} (connect 400))))
+ (testing "a rejected key would be rejected for every model, so it ends the search"
+ (is (=? {:result #"^Ollama rejected the API key" :probed ["m0"]} (connect 401)))))
+ (testing "a window too small is the model's own as often as the server's — a Modelfile can set one — so
+ the search moves on"
+ (is (= "m1"
+ (get-in (probe! [{:id "m0" :context-length 4096} {:id "m1" :context-length 32768}]
+ tool-calling-message)
+ [:connection-info :probed-model])))))
+
+(deftest connect-warns-when-ollamas-own-api-is-out-of-reach-test
+ (let [serve (probing-server [{:id "good-model"}]
+ {:tools {:message tool-calling-message :finish_reason "tool_calls"}
+ :structured structured-success})
+ connect (fn [handler]
+ (with-clean-capabilities!
+ (fn []
+ (mt/with-dynamic-fn-redefs [http/request handler]
+ (log.capture/with-log-messages-for-level [messages [metabase.metabot.self.ollama :warn]]
+ (ollama/list-models {:credentials credentials :probe? true})
+ (messages))))))]
+ (testing (str "a proxy that forwards only /v1 connects fine and degrades quietly, so connecting says so "
+ "where an operator will see it, without the response the error carries")
+ (let [messages (connect (fn [{:keys [url] :as req}]
+ (if (re-find #"/api/" (str url))
+ (throw (ex-info "404 Not Found" {:status 404 :body "secret response"}))
+ (serve req))))]
+ (is (=? [{:message #"Ollama at http://ollama\.internal:11434/v1 did not answer /api/version \(404 Not Found\)\..*"}]
+ messages))
+ (is (not-any? #(re-find #"secret response" (str (:message %) (:e %))) messages))))
+ (testing "a server whose own API answers is not warned about"
+ (is (empty? (connect serve))))))
+
+(deftest preflight-says-so-when-no-model-can-chat-test
+ (testing "a server that rules every model out is told so, rather than handed one to probe"
+ (with-clean-capabilities!
+ (fn []
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"None of the models on this server can chat and call tools"
+ (probe! [{:id "embed-a" :capabilities ["embedding"]}
+ {:id "embed-b" :capabilities ["embedding"]}]
+ tool-calling-message)))))))
+
+(deftest preflight-diagnoses-a-model-that-was-asked-for-by-name-test
+ (testing "an API client can name a model the picker no longer offers. Saying why beats Ollama's own 400,
+ and beats reporting a model the server plainly has as missing."
+ (doseq [[label capabilities] {"an embedding model" ["embedding"]
+ "a chat model that cannot call tools" ["completion"]}]
+ (testing label
+ (with-clean-capabilities!
+ (fn []
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"named-model can't call tools, which Metabot needs"
+ (mt/with-dynamic-fn-redefs [http/request (probing-server
+ [{:id "named-model" :capabilities capabilities}]
+ {:tools {:message tool-calling-message :finish_reason "tool_calls"}
+ :structured structured-success})]
+ (ollama/list-models {:credentials credentials :model "named-model" :probe? true}))))))))))
+
+(deftest listing-offers-only-models-metabot-could-run-on-test
+ (testing "the admin's picker is the catalog minus what Ollama says cannot chat"
+ (with-clean-capabilities!
+ (fn []
+ (is (= [{:id "thinking-model" :display_name "thinking-model"}]
+ (:models (mt/with-dynamic-fn-redefs
+ [http/request (probing-server [{:id "embedding-model" :capabilities ["embedding"]}
+ {:id "thinking-model" :capabilities ["completion" "tools" "thinking"]}]
+ {})]
+ (ollama/list-models {:credentials credentials})))))))))
+
+(deftest preflight-rejects-a-context-window-too-small-for-metabots-prompt-test
+ (testing (str "Ollama defaults the window to 4096 below 23GiB of VRAM, and a request cannot widen it: "
+ "the OpenAI-compatible surface has no `num_ctx`. Exceeding it is not an error either — "
+ "context shift keeps the first few tokens and the tail and discards the middle, which "
+ "is where the tools and the system prompt are, so Metabot would run with neither.")
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"4096 token context window, which is too small for Metabot\. Configure Ollama to load the model with a context window of at least 16384 tokens\.$"
+ (probe! [{:id "good-model" :context-length 4096}] tool-calling-message))))
+ (testing "a window that clears the floor passes"
+ (is (= "good-model"
+ (get-in (probe! [{:id "good-model" :context-length 16384}] tool-calling-message)
+ [:connection-info :probed-model])))
+ (testing "and the rejected case above passes every probe, which is why the probes cannot stand in
+ for this check: they are one-line prompts that fit in any window"
+ (is (= "good-model"
+ (get-in (probe! [{:id "good-model"}] tool-calling-message)
+ [:connection-info :probed-model]))))))
+
+(deftest preflight-checks-the-window-before-spending-a-second-generation-test
+ (testing "the window is a lookup once the tool probe has loaded the model, so a model whose window is
+ too small is turned away before the structured-output probe runs"
+ (let [probes (atom [])]
+ (mt/with-dynamic-fn-redefs [http/request (let [serve (probing-server
+ [{:id "good-model" :context-length 4096}]
+ {:tools {:message tool-calling-message :finish_reason "tool_calls"}
+ :structured structured-success})]
+ (fn [{:keys [url body] :as req}]
+ (when (re-find #"/chat/completions$" (str url))
+ (swap! probes conj (probe-kind (json/decode+kw (str body)))))
+ (serve req)))]
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"4096 token context window"
+ (ollama/list-models {:credentials credentials :probe? true})))
+ (is (= [:tools] @probes))))))
+
+(deftest preflight-does-not-read-the-window-off-the-catalog-test
+ (testing (str "the window comes from `/api/ps`, not from the listing — Ollama's catalog carries no "
+ "`max_model_len`, and a hand-written one must not be mistaken for the loaded window")
+ (is (= "good-model"
+ (get-in (probe! [{:id "good-model" :max_model_len 4096}] tool-calling-message)
+ [:connection-info :probed-model])))))
+
+(deftest preflight-passes-when-the-server-will-not-say-what-window-it-loaded-test
+ (testing (str "an Ollama that answers `/api/ps` without the model — already unloaded under "
+ "`OLLAMA_KEEP_ALIVE=0` — or will not answer it at all is no grounds to fail a "
+ "connection whose probes passed")
+ (is (= "good-model"
+ (get-in (probe! [{:id "good-model"}] tool-calling-message)
+ [:connection-info :probed-model])))
+ (mt/with-dynamic-fn-redefs [http/request (let [server (probing-server [{:id "good-model"}]
+ {:tools {:message tool-calling-message
+ :finish_reason "tool_calls"}
+ :structured structured-success})]
+ (fn [{:keys [url] :as req}]
+ (if (re-find #"/api/ps$" (str url))
+ (throw (ex-info "ps is not there" {}))
+ (server req))))]
+ (is (= "good-model"
+ (get-in (ollama/list-models {:credentials credentials :probe? true})
+ [:connection-info :probed-model]))))))
+
+(defn- cloud-probe!
+ [models]
+ (probe-choice! models
+ {:message tool-calling-message :finish_reason "tool_calls"}
+ {:message structured-tool-message :finish_reason "tool_calls"}
+ cloud-credentials))
+
+(deftest preflight-gates-cloud-on-a-context-window-too-test
+ (testing (str "Cloud already runs a model at its largest window, so a window below the floor is the "
+ "model's own limit — which is a model Metabot cannot use, exactly as on a self-hosted "
+ "server that was started with too small a window")
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"4096 token context window, which is too small for Metabot\. Pick a model with a context window of at least 16384 tokens\.$"
+ (cloud-probe! [{:id "small-window-model" :context-length 4096}]))))
+ (testing "a Cloud model whose window clears the floor passes"
+ (is (= "big-window-model"
+ (get-in (cloud-probe! [{:id "big-window-model" :context-length 131072}])
+ [:connection-info :probed-model])))))
+
+(deftest preflight-rejects-a-model-that-cannot-call-tools-test
+ (testing "the fix is always a different model — Ollama drives tool calling from the model's own
+ template, so there is no server flag to point the admin at"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"answered with text instead of calling a tool.*supports tool calling"
+ (probe! [{:id "chatty-model"}] {:content "Sure! The table is orders."})))))
+
+(deftest preflight-rejects-a-model-that-calls-the-wrong-tool-test
+ (testing "the agent loop runs only registered tool names and drops the rest without a word, so a
+ model that invents one would leave Metabot doing nothing with nothing to show for it"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"called 'write_query' instead of the one tool it was offered"
+ (probe! [{:id "creative-model"}]
+ {:content ""
+ :tool_calls [{:id "call-1"
+ :type "function"
+ :function {:name "write_query" :arguments "{\"table_name\": \"orders\"}"}}]})))))
+
+(deftest preflight-rejects-a-model-that-omits-a-required-argument-test
+ (testing "an empty argument map is well-formed JSON and useless: the agent loop validates each call
+ against its tool's schema, so the connection would fail later rather than here"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"without its required table_name argument"
+ (probe! [{:id "sloppy-model"}]
+ {:content ""
+ :tool_calls [{:id "call-1"
+ :type "function"
+ :function {:name "record_table_name" :arguments "{}"}}]})))))
+
+(deftest preflight-rejects-a-model-that-leaks-its-thinking-into-chat-test
+ (testing "reasoning that arrives as chat text would appear inside Metabot's answers"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"streamed its reasoning as chat text"
+ (probe! [{:id "good-model"}] {:content "<think>hmm</think> orders"})))))
+
+(deftest preflight-probes-structured-output-the-way-the-runtime-asks-for-it-test
+ (testing "`tool_choice` is discarded by Ollama, so a probe that only sets it re-runs the tool-calling
+ check and can only pass. Each deployment is probed through the mechanism it will really use."
+ (let [bodies (atom [])
+ record (fn [models choice-by-probe]
+ (let [server (probing-server models choice-by-probe)]
+ (fn [{:keys [body] :as req}]
+ (when body (swap! bodies conj (json/decode+kw (str body))))
+ (server req))))]
+ (testing "self-hosted constrains the decoder, and offers no tool to call"
+ (reset! bodies [])
+ (mt/with-dynamic-fn-redefs
+ [http/request (record [{:id "good-model"}] {:tools {:message tool-calling-message :finish_reason "tool_calls"}
+ :structured structured-success})]
+ (ollama/list-models {:credentials credentials :probe? true}))
+ (let [structured (m/find-first :response_format @bodies)]
+ (is (some? structured)
+ "the structured probe must send response_format")
+ (is (= "json_schema" (get-in structured [:response_format :type])))
+ (is (nil? (:tools structured))
+ "nothing to call under a grammar — a tool here would probe the wrong path")))
+ (testing "Cloud has no grammar to fall back on, so it probes the instruction-and-tool path"
+ (reset! bodies [])
+ (mt/with-dynamic-fn-redefs
+ [http/request (record [{:id "good-model"}] {:tools {:message tool-calling-message :finish_reason "tool_calls"}
+ :structured {:message structured-tool-message :finish_reason "tool_calls"}})]
+ (ollama/list-models {:credentials cloud-credentials :probe? true}))
+ (let [structured (m/find-first #(some (fn [t] (= "structured_output" (get-in t [:function :name])))
+ (:tools %))
+ @bodies)]
+ (is (some? structured)
+ "the structured probe must offer the schema tool under the name the agent loop reads")
+ (is (nil? (:response_format structured))
+ "Cloud serves no structured outputs — probing one would pass on a promise it cannot keep")
+ (is (some #(re-find #"structured_output" (str (:content %))) (:messages structured))
+ "and must carry the instruction that stands in for the discarded tool_choice"))))))
+
+(deftest preflight-rejects-a-self-hosted-server-that-ignores-the-schema-test
+ (testing "an Ollama too old to read `response_format` ignores it and the model answers in prose.
+ That is a server problem, not a model one, so the message names both remedies."
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"did not answer with JSON matching the schema.*response_format"
+ (probe-choice! [{:id "good-model"}]
+ {:message tool-calling-message :finish_reason "tool_calls"}
+ {:message {:content "Sure! How about \"Late orders\"?"} :finish_reason "stop"})))))
+
+(deftest preflight-rejects-a-self-hosted-answer-that-is-json-but-not-the-schema-test
+ (testing "JSON alone is not the contract — the caller reads named fields out of it"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"did not answer with JSON matching the schema"
+ (probe-choice! [{:id "good-model"}]
+ {:message tool-calling-message :finish_reason "tool_calls"}
+ {:message {:content "{\"summary\": \"Late orders\"}"} :finish_reason "stop"})))))
+
+(deftest preflight-rejects-a-cloud-model-that-will-not-call-the-tool-test
+ (testing "Cloud cannot force the call, and the runtime re-ask cannot rescue a model that never makes
+ it, so this has to fail at connect rather than on every title"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"would not answer Ollama Cloud's structured-output request with a tool call"
+ (probe-choice! [{:id "good-model"}]
+ {:message tool-calling-message :finish_reason "tool_calls"}
+ {:message {:content "Late orders"} :finish_reason "stop"}
+ cloud-credentials)))))
+
+(deftest preflight-truncated-structured-output-is-its-own-diagnosis-test
+ (testing "a schema too large for the token budget looks nothing like a model that cannot follow one"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"ceiling before completing the structured answer"
+ (probe-choice! [{:id "good-model"}]
+ {:message tool-calling-message :finish_reason "tool_calls"}
+ {:message {:content "{\"title\": \"Late or"} :finish_reason "length"})))))
+
+(deftest preflight-rejects-a-model-the-server-has-not-pulled-test
+ (testing "falling back to another pulled model would pass every check and then persist a provider
+ string naming a model the server does not have"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"missing-model is not available.*Models on offer: good-model, chatty-model"
+ (mt/with-dynamic-fn-redefs
+ [http/request (probing-server ollama-catalog
+ {"auto" {:message tool-calling-message :finish_reason "tool_calls"}
+ "required" {:message tool-calling-message :finish_reason "tool_calls"}})]
+ (ollama/list-models {:credentials credentials :probe? true :model "missing-model"}))))))
+
+(deftest preflight-on-an-empty-catalog-is-its-own-failure-test
+ (testing "a reachable server offering nothing is a distinct failure from a model that misbehaves"
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] {:status 200 :body {:object "list" :data []}})]
+ (is (= {:models []} (ollama/list-models {:credentials credentials})))
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"offering no models"
+ (ollama/list-models {:credentials credentials :probe? true}))))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Streaming requests
+;;; ──────────────────────────────────────────────────────────────────
+
+(defn- raw!
+ "Run `ollama-raw` to completion against whatever the surrounding redefs stub out, returning the
+ exception it threw."
+ []
+ (try (into [] (ollama/ollama-raw {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :credentials credentials}))
+ (catch clojure.lang.ExceptionInfo e e)))
+
+(defn- failing-stream
+ "Stub the SSE stream so consuming it throws `e`, exercising `io-guarded` rather than the request."
+ [e]
+ (fn [_] (reify clojure.lang.IReduceInit
+ (reduce [_ _rf _init] (throw e)))))
+
+(deftest stream-socket-timeout-is-tagged-and-not-retryable-test
+ (testing "a timeout mid-stream is tagged rather than raw, and must not be replayed — a retry
+ costs a full cold prefill"
+ (with-redefs [self.core/sse-reducible (failing-stream (SocketTimeoutException. "Read timed out"))
+ debug/capture-stream (fn [r _] r)
+ http/request (fn [_] {:body nil})]
+ (let [e (raw!)]
+ (is (= :ollama-timeout (:error-code (ex-data e))))
+ (is (re-find #"stopped responding" (ex-message e)))
+ (is (false? (#'self/retryable-error? e)))))))
+
+(deftest stream-severed-mid-body-is-tagged-and-not-retryable-test
+ (testing "a connection dropped mid-stream is a distinct diagnosis from a timeout, and equally
+ must not be replayed"
+ (with-redefs [self.core/sse-reducible (failing-stream (java.io.IOException. "Connection reset"))
+ debug/capture-stream (fn [r _] r)
+ http/request (fn [_] {:body nil})]
+ (let [e (raw!)]
+ (is (= :ollama-stream-interrupted (:error-code (ex-data e))))
+ (is (re-find #"interrupted before the response finished" (ex-message e)))
+ (is (false? (#'self/retryable-error? e)))))))
+
+(deftest http-errors-reach-the-status-specific-message-test
+ (testing "the IOException catch runs first, so a non-2xx must still be translated by the
+ status-specific table rather than surfacing as a bare clj-http error"
+ (are [status pattern] (thrown-with-msg? clojure.lang.ExceptionInfo pattern
+ ;; `rethrow-api-error!` only translates a response that
+ ;; carries a body, which is what clj-http actually raises
+ (with-redefs [http/request (fn [_] (throw (ex-info "clj-http"
+ {:status status
+ :body "{}"})))]
+ (ollama/ollama-raw {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :credentials credentials})))
+ 401 #"Ollama rejected the API key"
+ 404 #"base URL should end in /v1"
+ 429 #"Ollama is rate limiting this instance"
+ 500 #"internal server error")))
+
+(defn- chunk-with
+ "One Chat Completions chunk carrying `delta`, plus the top-level fields the shared translation reads
+ to open the stream."
+ ([delta] (chunk-with delta nil))
+ ([delta finish-reason]
+ {:id "chatcmpl-1"
+ :model "good-model"
+ :choices [(cond-> {:index 0 :delta delta}
+ finish-reason (assoc :finish_reason finish-reason))]}))
+
+(defn- streamed-parts!
+ "Run `ollama` end to end over `responses` — one realized chunk sequence per HTTP request, in order —
+ and return the AISDK parts a caller would read, along with how many requests it took."
+ [responses opts]
+ (let [remaining (atom (vec responses))
+ calls (atom 0)]
+ (with-redefs [self.core/sse-reducible identity
+ debug/capture-stream (fn [r _] r)
+ ;; `/api/show` is capability metadata, not a generation: answered without
+ ;; capabilities, and not counted, so `:calls` stays a count of real requests
+ http/request (fn [{:keys [url]}]
+ (if (re-find #"/api/show$" (str url))
+ {:status 200 :body {}}
+ (do
+ (swap! calls inc)
+ (let [[head & tail] @remaining]
+ (reset! remaining (vec tail))
+ {:body head}))))]
+ {:parts (into [] (self.core/aisdk-xf) (ollama/ollama opts))
+ :calls @calls})))
+
+(defn- parts-of
+ "The parts of one `:type` from a [[streamed-parts!]] result. Every streaming test asks this question,
+ and inlining the filter made the assertions about the filter rather than about the answer."
+ [result kind]
+ (filterv #(= kind (:type %)) (:parts result)))
+
+(defn- usage-part [result] (first (parts-of result :usage)))
+
+(def ^:private usage-chunk
+ "The separate final chunk carrying usage, which is where the finish reason is reported."
+ {:id "chatcmpl-1" :model "good-model" :choices []
+ :usage {:prompt_tokens 10 :completion_tokens 5}})
+
+(defn- chat-opts
+ "An ordinary chat request: no schema, no tools."
+ []
+ {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :credentials credentials})
+
+(defn- structured-opts
+ "A `:schema` request against `creds` — the shape `call-llm-structured` sends."
+ ([] (structured-opts credentials))
+ ([creds]
+ {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :schema {:type "object"}
+ :credentials creds}))
+
+(def ^:private constrained-stream
+ "What a self-hosted server under a decoding grammar streams back: JSON in the content channel, split
+ across deltas like any other text, and no tool call anywhere."
+ [(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:content "{\"title\":"})
+ (chunk-with {:content " \"Late orders\"}"})
+ (chunk-with {} "stop")])
+
+(deftest constrained-output-reaches-the-caller-as-a-tool-call-test
+ (testing "`call-llm-structured` reads a tool call, but a constrained answer arrives as content. The
+ adapter owns that difference: downstream must see what a forced-tool provider produces."
+ (let [result (streamed-parts! [constrained-stream]
+ (structured-opts))]
+ (is (= [{:type :tool-input
+ :function "structured_output"
+ :arguments {:title "Late orders"}}]
+ (mapv #(select-keys % [:type :function :arguments])
+ (parts-of result :tool-input))))
+ (testing "and no text part — the raw JSON must not also surface as an answer"
+ (is (empty? (parts-of result :text)))))))
+
+(deftest constrained-output-reports-the-finish-reason-of-a-tool-call-test
+ (testing "a constrained answer finishes with `stop`; the stream should say what it really produced"
+ (let [result (streamed-parts! [(conj (vec (butlast constrained-stream))
+ (chunk-with {} "stop")
+ usage-chunk)]
+ (structured-opts))]
+ (is (= "tool-calls" (:finish-reason (usage-part result)))))))
+
+(deftest constrained-output-keeps-truncation-visible-test
+ (testing "`length` must survive the rewrite — a schema too large for the budget is a real failure,
+ and dressing it up as a completed tool call would hide it behind a JSON parse error"
+ (let [result (streamed-parts! [[(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:content "{\"title\": \"Late or"})
+ (chunk-with {} "length")
+ usage-chunk]]
+ (structured-opts))]
+ (is (= "length" (:finish-reason (usage-part result)))))))
+
+(deftest constrained-output-forwards-reasoning-alongside-the-json-test
+ (testing "the grammar binds the answer channel only, so a thinking model still streams its thinking"
+ (let [result (streamed-parts! [[(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:reasoning "which orders..."})
+ (chunk-with {:content "{\"title\": \"Late orders\"}"})
+ (chunk-with {} "stop")]]
+ (structured-opts))]
+ (is (= ["which orders..."] (mapv :text (parts-of result :reasoning))))
+ (is (= [{:title "Late orders"}]
+ (mapv :arguments (parts-of result :tool-input)))))))
+
+(def ^:private tool-union-stream
+ "What a self-hosted server under the union grammar streams back: a tool call named and argued in the
+ content channel, with `tool_calls` empty."
+ [(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:content "{\"name\": \"fake_tool\","})
+ (chunk-with {:content " \"parameters\": {\"a\": \"x\"}}"})
+ (chunk-with {} "stop")])
+
+(defn- forced-tool-opts []
+ {:model "good-model"
+ :input [{:role :user :content "hi"}]
+ :tools [fake-tool other-fake-tool]
+ :tool_choice "required"
+ :credentials credentials})
+
+(deftest union-grammar-answer-reaches-the-agent-loop-as-the-tool-it-named-test
+ (testing "the grammar guarantees a call, but puts it in the content channel. The agent loop reads
+ tool calls, so the adapter has to name the tool the answer chose — not a fixed one"
+ (let [result (streamed-parts! [tool-union-stream] (forced-tool-opts))]
+ (is (= [{:type :tool-input
+ :function "fake_tool"
+ :arguments {:a "x"}}]
+ (mapv #(select-keys % [:type :function :arguments])
+ (parts-of result :tool-input))))
+ (testing "and the JSON must not also surface as an answer the user would see"
+ (is (empty? (parts-of result :text)))))))
+
+(deftest union-grammar-truncation-is-not-dressed-up-as-a-call-test
+ (testing "a half-written union answer has no readable tool name. Emitting a call anyway would turn a
+ truncation into a confusing parse error; `length` is the true diagnosis and must survive"
+ (let [result (streamed-parts! [[(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:content "{\"name\": \"fake_to"})
+ (chunk-with {} "length")
+ usage-chunk]]
+ (forced-tool-opts))]
+ (is (empty? (parts-of result :tool-input)))
+ (is (= "length" (:finish-reason (usage-part result)))))))
+
+(deftest an-auto-tool-turn-is-left-unconstrained-test
+ (testing "no grammar on an `auto` turn, so nothing rewrites the model's text into a tool call it
+ never made — forcing every turn would stop these profiles ever answering"
+ (let [result (streamed-parts! [[(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:content "no tool needed"})
+ (chunk-with {} "stop")]]
+ (dissoc (forced-tool-opts) :tool_choice))]
+ (is (= ["no tool needed"] (mapv :text (parts-of result :text))))
+ (is (empty? (parts-of result :tool-input))))))
+
+(deftest cloud-forced-calls-are-asked-for-not-rewritten-test
+ (testing "Cloud has neither lever, so its tool call arrives the ordinary way and must pass through
+ untouched — and a model that answers in chat produces the caller's ordinary failure rather
+ than anything this adapter invents"
+ (let [taken (streamed-parts! [[(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:tool_calls [{:id "call-9"
+ :type "function"
+ :function {:name "structured_output"
+ :arguments "{\"title\": \"Late orders\"}"}}]})
+ (chunk-with {} "tool_calls")]]
+ (structured-opts cloud-credentials))
+ ignored (streamed-parts! [[(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:content "How about \"Late orders\"?"})
+ (chunk-with {} "stop")]]
+ (structured-opts cloud-credentials))]
+ (is (= [{:title "Late orders"}]
+ (mapv :arguments (parts-of taken :tool-input))))
+ (testing "no re-ask: one request, and the missing call stays missing for the caller to report"
+ (is (= 1 (:calls ignored)))
+ (is (empty? (parts-of ignored :tool-input)))
+ (is (= ["How about \"Late orders\"?"]
+ (mapv :text (parts-of ignored :text))))))))
+
+(deftest an-unstructured-request-is-streamed-straight-through-test
+ (testing "none of the structured machinery may touch ordinary chat — no buffering, no rewriting"
+ (let [result (streamed-parts! [[(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:content "hello"})
+ (chunk-with {} "stop")]]
+ (chat-opts))]
+ (is (= 1 (:calls result)))
+ (is (= ["hello"] (mapv :text (parts-of result :text))))
+ (is (empty? (parts-of result :error))
+ "a stream that says why it ended is complete"))))
+
+(deftest a-stream-that-ends-without-a-finish-reason-fails-the-turn-test
+ (testing (str "Ollama reports a generation that failed after the first token as an empty chunk and closes "
+ "the stream with no finish reason, so the half answer must not pass as a complete one")
+ (let [result (streamed-parts! [[(chunk-with {:role "assistant" :content ""})
+ (chunk-with {:content "The answer is"})
+ {:id "chatcmpl-1" :model "good-model" :choices []}]]
+ (chat-opts))]
+ (is (= ["The answer is"] (mapv :text (parts-of result :text))))
+ (is (=? [{:error {:message "The Ollama server ended the response before finishing it."}}]
+ (parts-of result :error))))))
+
+(deftest a-request-without-a-model-fails-before-any-io-test
+ (testing "a blank model is a configuration problem, not something to discover from the server"
+ (with-redefs [http/request (fn [_] (throw (ex-info "should never be called" {})))]
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"No Ollama model is set"
+ (ollama/ollama-raw {:model "" :input [] :credentials credentials}))))))
+
+;;; ──────────────────────────────────────────────────────────────────
+;;; Registration
+;;; ──────────────────────────────────────────────────────────────────
+
+(deftest ollama-is-a-registered-provider-type-test
+ (testing "registered, so it reaches the admin dropdown and the generated docs page"
+ (is (some? (llm.provider/provider-type "ollama")))
+ (testing "with no default model — the operator serves whatever they pulled"
+ (is (nil? (llm.provider/default-model "ollama"))))))
+
+(deftest ollama-form-asks-for-an-address-and-a-key-test
+ (let [by-key (u/index-by :key (:fields (llm.provider/provider-type "ollama")))
+ base-url (:base-url by-key)]
+ (testing "two fields, and nothing that says which kind of server is behind them: the address does"
+ (is (= #{:base-url :api-key} (set (keys by-key)))))
+ (testing (str "the address is required and has no default: a self-hosted Ollama is wherever the operator put "
+ "it, and on a real install localhost is the Metabase container rather than that host")
+ (is (true? (:required? base-url)))
+ (is (nil? (:default base-url))))
+ (testing "the key is optional, since a self-hosted server takes none"
+ (is (not (:required? (:api-key by-key)))))))
+
+(deftest ollama-requires-an-address-test
+ (testing "an address is a complete connection, whether it is a server of the operator's or Ollama Cloud's"
+ (is (true? (llm.provider/credentials-complete? "ollama" {:base-url base-url})))
+ (is (true? (llm.provider/credentials-complete? "ollama" cloud-credentials))))
+ (testing "a key with nowhere to send it is not"
+ (is (false? (llm.provider/credentials-complete? "ollama" {:api-key "sk-x"})))
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"API base URL is required for ollama"
+ (llm.provider/validate-config! "ollama" {:api-key "sk-x"}))))
+ (testing "an address is judged by the network policy"
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"not allowed to connect to"
+ (llm.provider/validate-config! "ollama" {:base-url "http://192.168.1.20:11434/v1"}))))
+ (testing "a trailing slash still cannot double up when a path is joined onto it"
+ (is (= "http://host:11434/v1"
+ (:base-url (llm.provider/with-field-defaults "ollama" {:base-url "http://host:11434/v1///"}))))))
+
+(deftest ollama-calls-the-address-it-was-given-test
+ ;; Credentials go through `with-field-defaults` on every real path — `resolve-model-ref` for
+ ;; requests, the provider API for connect — so these run through it too.
+ (letfn [(url-of [raw-config] (:url (captured-request raw-config)))]
+ (testing "Ollama Cloud is reached at its own address, like any other server"
+ (is (= "https://ollama.com/v1/models" (url-of cloud-credentials))))
+ (testing "a self-hosted connection goes exactly where it was told"
+ (is (= (str base-url "/models") (url-of credentials))))
+ (testing "a connection with no address throws rather than guessing at one"
+ (let [e (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"No Ollama base URL is set"
+ (url-of {:api-key "proxy-key"})))]
+ (testing "tagged so the admin API renders it under the field rather than as a 500"
+ (is (= {:status-code 400 :field :base-url} (select-keys (ex-data e) [:status-code :field]))))))))
+
+(deftest ollama-is-configurable-from-the-environment-test
+ (is (= {:base-url "MB_LLM_OLLAMA_API_BASE_URL"
+ :api-key "MB_LLM_OLLAMA_API_KEY"}
+ (llm.provider/connection-env-vars "ollama")))
+ (letfn [(connection-config []
+ (:config (first (llm.provider/connections))))
+ (url-for [config]
+ (let [called (atom nil)]
+ (mt/with-dynamic-fn-redefs [http/request (fn [req]
+ (reset! called (:url req))
+ {:status 200 :body {:data []}})]
+ (ollama/list-models
+ {:credentials (llm.provider/with-field-defaults "ollama" config)}))
+ @called))]
+ (mt/with-temporary-setting-values [llm-providers []]
+ (testing "a base URL alone synthesizes a connection and reaches that server"
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url base-url]
+ (is (= {:base-url base-url} (connection-config)))
+ (is (= (str base-url "/models") (url-for (connection-config))))))
+ (testing "Cloud's address plus a key synthesizes a Cloud connection and reaches Cloud"
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-base-url "https://ollama.com/v1"
+ mb-llm-ollama-api-key "sk-env"]
+ (is (= {:base-url "https://ollama.com/v1" :api-key "sk-env"} (connection-config)))
+ (is (= "https://ollama.com/v1/models" (url-for (connection-config))))))
+ (testing "a key on its own brings no connection into existence: it has nowhere to be sent"
+ (mt/with-temp-env-var-value! [mb-llm-ollama-api-key "sk-env"]
+ (is (empty? (llm.provider/connections))))))))
+
+(deftest ollama-takes-one-key-for-either-deployment-test
+ (testing "it is the only secret the type stores"
+ (is (= #{:api-key} (llm.provider/secret-field-keys "ollama"))))
+ (testing "and it authenticates against Cloud and a self-hosted server alike, since both take the same Bearer header"
+ (letfn [(bearer [creds] (get-in (captured-request creds) [:headers "Authorization"]))]
+ (is (= "Bearer sk-cloud-key" (bearer cloud-credentials)))
+ (is (= "Bearer proxy-key" (bearer keyed-credentials)))
+ (testing "a plain self-hosted server takes no key, and must not get an empty header"
+ (is (nil? (bearer credentials)))))))
+
+(deftest ollama-has-no-model-allow-list-test
+ (testing "`known-models` returns nil rather than throwing — the catalog is whatever is pulled"
+ (is (nil? (self/known-models "ollama")))))
diff --git a/test/metabase/metabot/self/openai/chat_completions_test.clj b/test/metabase/metabot/self/openai/chat_completions_test.clj
index d04c8e0322c..84d4cd2fada 100644
--- a/test/metabase/metabot/self/openai/chat_completions_test.clj
+++ b/test/metabase/metabot/self/openai/chat_completions_test.clj
@@ -117,6 +117,24 @@
[{:type :reasoning :id "r1" :text "alone"}]
{:reasoning-part->message top-level-hook}))))))
+(deftest ^:parallel whitespace-only-argument-fragments-survive-test
+ (testing (str "fragments are joined verbatim, so a whitespace-only one cannot be dropped: inside a "
+ "JSON string those characters are content — indentation in streamed SQL, and syntax "
+ "in streamed Python")
+ (let [fragments ["{\"code\": \"def f():" "\\n" " " "return 1\"}"]
+ chunk (fn [delta] {:id "c" :model "m" :choices [{:index 0 :delta delta}]})
+ parts (into [] (chat-completions/chat-completions->aisdk-chunks-xf
+ chat-completions/stop-reasons {})
+ (concat [(chunk {:role "assistant" :content ""})
+ (chunk {:tool_calls [{:index 0 :id "call-1" :type "function"
+ :function {:name "run_python"
+ :arguments (first fragments)}}]})]
+ (for [f (rest fragments)]
+ (chunk {:tool_calls [{:index 0 :function {:arguments f}}]}))))]
+ (is (= (apply str fragments)
+ (apply str (keep :inputTextDelta parts)))
+ "the indented line is not silently deleted from the arguments"))))
+
(deftest ^:parallel parts->cc-messages-tool-call-test
(testing "text + tool call merges into single assistant message"
(is (=? [{:role "assistant"
@@ -625,6 +643,115 @@
:function {:arguments ": \"Berlin\"}"}}]}}]}
{:choices [{:index 0 :delta {} :finish_reason "tool_calls"}]}])))))
+(deftest ^:parallel chunks-xf-batched-tool-calls-all-reach-the-caller-test
+ (testing "a server may put a whole turn's worth of calls in one delta's `tool_calls` array — Ollama
+ does when a model's tool parser flushes more than one at a time. Reading one call per delta
+ kept the first and silently dropped the rest, and a dropped call is a tool that never runs."
+ (is (= [{:type :start :messageId "chatcmpl-b"}
+ {:type :tool-input-start :toolCallId "call-a" :toolName "get_table"}
+ {:type :tool-input-delta :toolCallId "call-a" :inputTextDelta "{\"id\":1}"}
+ {:type :tool-input-available :toolCallId "call-a" :toolName "get_table"}
+ {:type :tool-input-start :toolCallId "call-b" :toolName "get_table"}
+ {:type :tool-input-delta :toolCallId "call-b" :inputTextDelta "{\"id\":2}"}
+ {:type :tool-input-available :toolCallId "call-b" :toolName "get_table"}]
+ (into [] (chat-completions/chat-completions->aisdk-chunks-xf)
+ [{:id "chatcmpl-b" :model "m" :choices [{:index 0 :delta {:role "assistant" :content ""}}]}
+ {:id "chatcmpl-b"
+ :model "m"
+ :choices [{:index 0
+ :delta {:tool_calls [{:index 0 :id "call-a" :type "function"
+ :function {:name "get_table" :arguments "{\"id\":1}"}}
+ {:index 1 :id "call-b" :type "function"
+ :function {:name "get_table" :arguments "{\"id\":2}"}}]}}]}
+ {:choices [{:index 0 :delta {} :finish_reason "tool_calls"}]}])))))
+
+(deftest ^:parallel chunks-xf-batched-tool-calls-end-the-turn-once-test
+ (testing "`finish_reason` and `usage` riding the batched delta are reported once, after the calls —
+ asserted over the whole sequence, since the ordering is the claim"
+ (is (= [{:type :start :messageId "chatcmpl-b"}
+ {:type :tool-input-start :toolCallId "call-a" :toolName "t"}
+ {:type :tool-input-delta :toolCallId "call-a" :inputTextDelta "{}"}
+ {:type :tool-input-available :toolCallId "call-a" :toolName "t"}
+ {:type :tool-input-start :toolCallId "call-b" :toolName "t"}
+ {:type :tool-input-delta :toolCallId "call-b" :inputTextDelta "{}"}
+ {:type :tool-input-available :toolCallId "call-b" :toolName "t"}
+ {:type :usage
+ :usage {:promptTokens 10 :completionTokens 4
+ :cacheCreationTokens 0 :cacheReadTokens 0}
+ :id "chatcmpl-b"
+ :model "m"
+ :finish-reason "tool-calls"
+ :raw-finish-reason "tool_calls"}]
+ (into [] (chat-completions/chat-completions->aisdk-chunks-xf)
+ [{:id "chatcmpl-b"
+ :model "m"
+ :choices [{:index 0
+ :delta {:tool_calls [{:index 0 :id "call-a" :type "function"
+ :function {:name "t" :arguments "{}"}}
+ {:index 1 :id "call-b" :type "function"
+ :function {:name "t" :arguments "{}"}}]}
+ :finish_reason "tool_calls"}]
+ :usage {:prompt_tokens 10 :completion_tokens 4}}])))))
+
+(deftest ^:parallel chunks-xf-content-and-tool-calls-in-one-delta-test
+ (testing "a delta can carry text and tool calls together. Ollama sends this shape when its parser flushes text
+ and a call at the same time. The text comes first and its block closes, then every call follows in
+ wire order."
+ (is (= [{:type :start :messageId "chatcmpl-b"}
+ {:type :text-start}
+ {:type :text-delta :delta "on it"}
+ {:type :text-end}
+ {:type :tool-input-start :toolCallId "call-a" :toolName "t"}
+ {:type :tool-input-delta :toolCallId "call-a" :inputTextDelta "{}"}
+ {:type :tool-input-available :toolCallId "call-a" :toolName "t"}
+ {:type :tool-input-start :toolCallId "call-b" :toolName "t"}
+ {:type :tool-input-delta :toolCallId "call-b" :inputTextDelta "{}"}
+ {:type :tool-input-available :toolCallId "call-b" :toolName "t"}]
+ (mapv #(dissoc % :id)
+ (into [] (chat-completions/chat-completions->aisdk-chunks-xf)
+ [{:id "chatcmpl-b"
+ :model "m"
+ :choices [{:index 0
+ :delta {:content "on it"
+ :tool_calls [{:index 0 :id "call-a" :type "function"
+ :function {:name "t" :arguments "{}"}}
+ {:index 1 :id "call-b" :type "function"
+ :function {:name "t" :arguments "{}"}}]}}]}]))))))
+
+(deftest ^:parallel chunks-xf-empty-continuation-arguments-emit-nothing-test
+ (testing "a continuation fragment carrying \"\" emits no delta, since an empty string adds nothing
+ to the joined arguments. This pins the chunk stream, which is what a consumer counting
+ deltas would notice."
+ (is (= [{:type :start :messageId "chatcmpl-s"}
+ {:type :tool-input-start :toolCallId "call-1" :toolName "t"}
+ {:type :tool-input-delta :toolCallId "call-1" :inputTextDelta "{\"a\":1}"}
+ {:type :tool-input-available :toolCallId "call-1" :toolName "t"}]
+ (into [] (chat-completions/chat-completions->aisdk-chunks-xf)
+ [{:id "chatcmpl-s"
+ :model "m"
+ :choices [{:index 0
+ :delta {:tool_calls [{:index 0 :id "call-1" :type "function"
+ :function {:name "t" :arguments "{\"a\":1}"}}]}}]}
+ ;; the empty fragment some servers send between meaningful ones
+ {:choices [{:index 0 :delta {:tool_calls [{:index 0 :function {:arguments ""}}]}}]}
+ {:choices [{:index 0 :delta {} :finish_reason "tool_calls"}]}])))))
+
+(deftest ^:parallel chunks-xf-arguments-without-an-open-call-are-dropped-test
+ (testing "arguments that belong to no open call emit no delta, so aggregation never sees a delta without a toolCallId"
+ (doseq [[label entries]
+ {"a new id without a name"
+ [{:index 0 :id "call-1" :type "function" :function {:name "t" :arguments "{\"a\":1}"}}
+ {:index 1 :id "call-2" :type "function" :function {:arguments "{\"b\":2}"}}]}]
+ (testing label
+ (is (=? [{:type :start}
+ {:type :tool-input :id "call-1" :function "t" :arguments {:a 1}}]
+ (into [] (comp (chat-completions/chat-completions->aisdk-chunks-xf)
+ (self.core/aisdk-xf))
+ (concat [{:id "chatcmpl-o" :model "m" :choices [{:index 0 :delta {:role "assistant"}}]}]
+ (for [entry entries]
+ {:choices [{:index 0 :delta {:tool_calls [entry]}}]})
+ [{:choices [{:index 0 :delta {} :finish_reason "tool_calls"}]}]))))))))
+
(deftest ^:parallel chunks-xf-reasoning-deltas-open-no-text-block-test
(testing "reasoning_content deltas and empty-string content produce no chunks"
;; Without `:forward-reasoning?` reasoning deltas are dropped rather than surfaced as text.
diff --git a/test/metabase/metabot/self/vllm_test.clj b/test/metabase/metabot/self/vllm_test.clj
index ce0f6a752da..932baf66b7f 100644
--- a/test/metabase/metabot/self/vllm_test.clj
+++ b/test/metabase/metabot/self/vllm_test.clj
@@ -215,7 +215,7 @@
(deftest ^:parallel request-body-supplies-a-default-temperature-test
(testing "a caller that supplies none gets the adapter default rather than vLLM's own 1.0"
- (is (= @#'vllm/default-temperature
+ (is (= adapter/default-temperature
(:temperature (vllm/vllm-request-body {:model "vllm-test"
:input [{:role :user :content "hi"}]}))))))
@@ -457,8 +457,7 @@
tc)]
(testing "index arrives in contiguous runs, not interleaved"
(is (= [0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1] (mapv :index deltas))))
- (testing "id arrives on each call's opening delta only — a provider repeating it would lose the
- arguments, since neither the start branch nor the argument-delta branch would fire"
+ (testing "id arrives on each call's opening delta only, so each id opens exactly one block"
(is (= [0 1] (keep #(when (:id %) (:index %)) deltas)))))))
;;; ──────────────────────────────────────────────────────────────────
@@ -797,7 +796,7 @@
(deftest list-models-fails-closed-on-a-body-that-is-not-a-catalog-test
(testing "a 2xx whose body carries no model list throws, naming the base URL"
- (doseq [body [{:status "ok" :service "some-other-thing"} {:object "list"} "<html>404</html>"]]
+ (doseq [body [{:status "ok" :service "some-other-thing"} {:object "list"} 404]]
(testing (str "body " (pr-str body))
(mt/with-dynamic-fn-redefs [http/request (fn [_] {:status 200 :body body})]
(is (thrown-with-msg?
@@ -812,6 +811,28 @@
#"reachable but is not serving any models"
(vllm/list-models {:credentials credentials :probe? true}))))))
+(deftest list-models-fails-closed-on-a-2xx-that-is-not-json-test
+ (testing "a 2xx whose body is not JSON — a proxy's HTML, or a base URL missing /v1 — is a server
+ that answered, so it reads as a bad address rather than an unreachable one. The stub throws
+ what `:as :json` throws: clj-http parses a 2xx whatever its content type."
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (json/decode "<html>404 Not Found</html>"))]
+ (is (=? {:error-code :malformed-model-catalog
+ :status-code 400}
+ (try
+ (vllm/list-models {:credentials credentials})
+ (catch clojure.lang.ExceptionInfo e (ex-data e)))))
+ (is (thrown-with-msg?
+ clojure.lang.ExceptionInfo
+ #"vLLM returned an unexpected model list response.*http://vllm\.internal:8000/v1"
+ (vllm/list-models {:credentials credentials}))))))
+
+(deftest list-models-keeps-the-parse-error-as-the-cause-test
+ (testing "the parse error travels as the cause, so a log shows whether the body was HTML, empty or cut off"
+ (mt/with-dynamic-fn-redefs [http/request (fn [_] (json/decode "<html>404 Not Found</html>"))]
+ (is (instance? com.fasterxml.jackson.core.JsonProcessingException
+ (try (vllm/list-models {:credentials credentials})
+ (catch clojure.lang.ExceptionInfo e (ex-cause e))))))))
+
(deftest list-models-fails-closed-before-probing-test
(testing "a malformed catalog throws without issuing a probe request"
(let [chat-requests (atom 0)]
diff --git a/test/metabase/metabot/settings_test.clj b/test/metabase/metabot/settings_test.clj
index cccdabf9257..7f8e81ba3e9 100644
--- a/test/metabase/metabot/settings_test.clj
+++ b/test/metabase/metabot/settings_test.clj
@@ -1,11 +1,14 @@
(ns metabase.metabot.settings-test
(:require
+ [clj-http.client :as http]
[clojure.test :refer [deftest is testing use-fixtures]]
[metabase.metabot.self :as metabot.self]
+ [metabase.metabot.self.ollama.capabilities :as ollama.capabilities]
[metabase.metabot.settings :as metabot.settings]
[metabase.settings.core :as setting]
[metabase.test :as mt]
- [metabase.test.fixtures :as fixtures]))
+ [metabase.test.fixtures :as fixtures]
+ [metabase.util.json :as json]))
(use-fixtures :once (fixtures/initialize :db))
@@ -286,6 +289,64 @@
(with-selected-model "vllm/vllm-test"
(is (false? (metabot.settings/llm-metabot-supports-reasoning?)))))))
+(def ^:private ollama-credentials
+ {:base-url "http://ollama.internal:11434/v1"})
+
+(def ^:private ollama-connection
+ (connection "ollama" "ollama" ollama-credentials))
+
+(defn- showing-capabilities
+ "Stub `http/request` so `/api/show` reports `capabilities-by-model`, and a model absent from it comes
+ back without a `capabilities` key — an Ollama too old to report them."
+ [capabilities-by-model]
+ (fn [{:keys [body]}]
+ (let [model (:model (json/decode+kw (str body)))]
+ {:status 200
+ :body (cond-> {:model model}
+ (contains? capabilities-by-model model)
+ (assoc :capabilities (get capabilities-by-model model)))})))
+
+(defn- supports-reasoning?
+ "The setting's answer for `model`, once the adapter has been given the chance to learn about it.
+
+ The setting itself only reads what is known — it is public, so it never calls Ollama. Warming here
+ goes through the accessor the request path uses, which is one of the two places that fills the
+ cache in production; the other is listing a connection's models."
+ [model]
+ (ollama.capabilities/reasoning-model? ollama-credentials model)
+ (with-selected-model (str "ollama/" model)
+ (metabot.settings/llm-metabot-supports-reasoning?)))
+
+(defn- with-capabilities!
+ "Call `f` against a `/api/show` stub reporting `by-model`, with the capability cache emptied on both
+ sides so no test inherits another's models."
+ [by-model f]
+ (ollama.capabilities/clear-cache!)
+ (try
+ (mt/with-dynamic-fn-redefs [http/request (showing-capabilities by-model)]
+ (f))
+ (finally
+ (ollama.capabilities/clear-cache!))))
+
+(deftest metabot-supports-reasoning-ollama-test
+ (testing "Ollama answers per model rather than per connection: one connection serves as many models as
+ the operator has pulled, and Metabot and the mini model need not be on the same one"
+ (with-capabilities! {"thinking-model" ["completion" "tools" "thinking"]
+ "chat-model" ["completion" "tools"]}
+ (fn []
+ (with-connections [ollama-connection]
+ (testing "thinking-model" (is (true? (supports-reasoning? "thinking-model"))))
+ (testing "chat-model" (is (false? (supports-reasoning? "chat-model"))))))))
+ (testing "and a model nothing has looked up yet reads as non-reasoning rather than guessing: the
+ setting is public, so it answers from what is already known instead of asking"
+ ;; the stub reports this model as thinking, so `false?` holds only while the setting answers from the
+ ;; cache rather than asking
+ (with-capabilities! {"unlooked-model" ["completion" "tools" "thinking"]}
+ (fn []
+ (with-connections [ollama-connection]
+ (with-selected-model "ollama/unlooked-model"
+ (is (false? (metabot.settings/llm-metabot-supports-reasoning?)))))))))
+
(deftest metabot-supports-reasoning-managed-proxy-test
(testing "the managed connection answers from the model's own provider segment"
(with-connections [(connection "metabase" "metabase")]