Skip to content

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.

docs: LLM provider configuration (#6529)

Myles Scolnick715b076e5773
diff --git a/docs/_static/docs-provider-config.png b/docs/_static/docs-provider-config.png
new file mode 100644
index 0000000000..0a62c7f0d8
Binary files /dev/null and b/docs/_static/docs-provider-config.png differ
diff --git a/docs/guides/configuration/llm_providers.md b/docs/guides/configuration/llm_providers.md
new file mode 100644
index 0000000000..8a0763f983
--- /dev/null
+++ b/docs/guides/configuration/llm_providers.md
@@ -0,0 +1,450 @@
+# Configuring LLM providers
+
+Connect marimo to an LLM via the notebook Settings panel (recommended) or by editing
+`marimo.toml` directly. **Prefer going through the notebook settings menu to avoid errors with the config file.**
+
+To locate your config file:
+
+```bash
+marimo config show
+```
+
+The path to `marimo.toml` is printed at the top of the output.
+
+## Model configuration
+
+### Model roles and routing
+
+marimo uses three roles:
+
+* **`chat_model`**: chat panel
+* **`edit_model`**: refactor cells (Cmd/Ctrl-Shift-E) and Generate with AI
+* **`autocomplete_model`**: inline code completion
+
+Models are written as `provider/model`, and the provider prefix routes to the matching config section:
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "openai/gpt-4o-mini" # Routes to OpenAI config
+edit_model = "anthropic/claude-3-sonnet" # Routes to Anthropic config
+autocomplete_model = "ollama/codellama" # Routes to Ollama config
+autocomplete_model = "some_other_provider/some_model" # Routes to OpenAI compatible config
+```
+
+### Custom models
+
+Add custom entries to the model dropdown:
+
+```toml title="marimo.toml"
+[ai.models]
+custom_models = ["ollama/somemodel"]
+```
+
+## Supported providers
+
+<div align="center">
+<figure>
+<img src="/_static/docs-provider-config.png" width="740px"/>
+<figcaption>Configure LLM providers in the notebook settings panel.</figcaption>
+</figure>
+</div>
+
+You can configure the following providers:
+
+* Anthropic
+* AWS Bedrock
+* GitHub
+* Google AI
+* DeepSeek
+* xAI
+* LM Studio
+* Mistral
+* Ollama
+* OpenAI
+* OpenRouter
+* Together AI
+* Vercel v0
+* and any OpenAI-compatible provider
+
+Below we describe how to connect marimo to your AI provider.
+
+### OpenAI
+
+**Requirements**
+
+* `pip install openai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "openai/gpt-4o-mini"
+edit_model = "openai/gpt-4o"
+# See https://platform.openai.com/docs/models for the latest list
+
+[ai.open_ai]
+# Get an API key at https://platform.openai.com/account/api-keys
+api_key = "sk-proj-..."
+```
+
+!!! note "OpenAI-compatible providers"
+ If your model does not start with `openai/`, it will not be routed to the OpenAI config, and likely will be routed to the OpenAI-compatible config.
+
+??? tip "Reasoning models (o1, o3, etc.)"
+ These models can incur higher costs due to separate reasoning tokens. Prefer smaller responses for refactors or autocompletion, and review your provider limits.
+
+### Anthropic
+
+**Requirements**
+
+* Create an account and key: [Anthropic Console](https://console.anthropic.com/settings/keys)
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "anthropic/claude-3-7-sonnet-latest" # other options: claude-3-haiku, claude-3-opus
+# See Anthropic model list: https://docs.anthropic.com/en/docs/about-claude/models
+
+[ai.anthropic]
+api_key = "sk-ant-..."
+```
+
+### AWS Bedrock
+
+AWS Bedrock exposes multiple foundation models via a unified AWS API.
+
+**Requirements**
+
+* `pip install boto3`
+* Enable model access in the Bedrock console
+* AWS credentials via `aws configure`, env vars, or `~/.aws/credentials`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "bedrock/anthropic.claude-3-sonnet-latest"
+# Example model families include Anthropic Claude, Meta Llama, Cohere Command
+
+[ai.bedrock]
+region_name = "us-east-1" # AWS region where Bedrock is available
+# Optional AWS profile name (from ~/.aws/credentials)
+profile_name = "my-profile"
+```
+
+Use `profile_name` for a non-default named profile, or rely on env vars/standard AWS resolution.
+
+??? tip "Required AWS Bedrock permissions"
+ Ensure your IAM policy allows `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` for the models you plan to use.
+
+### Google AI
+
+**Requirements**
+
+* API key from [Google AI Studio](https://aistudio.google.com/app/apikey)
+* `pip install google-genai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "google/gemini-2.5-pro" # also see gemini-2.0-flash, etc.
+# Model list: https://ai.google.dev/gemini-api/docs/models/gemini
+
+[ai.google]
+api_key = "AI..."
+```
+
+### GitHub Copilot
+
+Use Copilot for code refactoring or the chat panel (Copilot subscription required).
+
+**Requirements**
+
+* Install the [gh CLI](https://cli.github.com/)
+* Get a token: `gh auth token`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "github/gpt-4o-mini"
+
+[ai.github]
+api_key = "gho_..."
+```
+
+??? question "My token starts with `ghp_` instead of `gho_`?"
+
+ This usually happens when you previously authenticated `gh` by pasting a _personal_ access token (`ghp_...`). However, GitHub Copilot is not available through `ghp_...`, and you will encounter errors such as:
+
+ > bad request: Personal Access Tokens are not supported for this endpoint
+
+ To resolve this issue, you could switch to an _OAuth_ access token (`gho_...`):
+
+ 1. Re-authenticate by running `gh auth login`.
+ 2. Choose _Login with a web browser_ (instead of _Paste an authentication token_) this time.
+
+### OpenRouter
+
+Route to many providers through OpenRouter with a single API.
+
+**Requirements**
+
+* Create an API key: [OpenRouter Dashboard](https://openrouter.ai/)
+* `pip install openai` (OpenRouter is OpenAI‑compatible)
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+# Use OpenRouter's model slugs (vendor/model). Examples:
+chat_model = "openrouter/openai/gpt-4o-mini"
+edit_model = "openrouter/anthropic/claude-3-7-sonnet"
+
+[ai.openrouter]
+api_key = "sk-or-..."
+base_url = "https://openrouter.ai/api/v1/"
+# Optional but recommended per OpenRouter best practices
+# extra_headers = { "HTTP-Referer" = "https://your-app.example", "X-Title" = "Your App Name" }
+```
+
+See available models at `https://openrouter.ai/models`. Make sure to prepend `openrouter/` to the model slug (e.g., `openrouter/deepseek/deepseek-chat`, `openrouter/meta-llama/llama-3.1-8b-instruct`).
+
+### Local models with Ollama { #using-ollama }
+
+Run open-source LLMs locally and connect via an OpenAI‑compatible API.
+
+**Requirements**
+
+* Install [Ollama](https://ollama.com/)
+* Pull a model
+
+ ```bash
+ # View available models at https://ollama.com/library
+ ollama pull llama3.1
+ ollama pull codellama # recommended for code generation
+
+ # View your installed models
+ ollama ls
+ ```
+
+3. Start the Ollama server:
+
+ ```bash
+ ollama serve
+ # In another terminal, run a model (optional)
+ ollama run codellama
+ ```
+
+4. Visit <http://127.0.0.1:11434> to confirm that the server is running.
+
+!!! note "Port already in use"
+ If you get a "port already in use" error, you may need to close an existing Ollama instance. On Windows, click the up arrow in the taskbar, find the Ollama icon, and select "Quit". This is a known issue (see [Ollama Issue #3575](https://github.com/ollama/ollama/issues/3575)). Once you've closed the existing Ollama instance, you should be able to run `ollama serve` successfully.
+
+5. Install the OpenAI client (`pip install openai` or `uv add openai`).
+
+6. Start marimo:
+
+ ```bash
+ marimo edit notebook.py
+ ```
+
+??? warning "Important: Use the `/v1` endpoint"
+
+ marimo uses Ollama's OpenAI‑compatible API. Ensure your `base_url` includes `/v1`:
+
+ ```toml
+ [ai.ollama]
+ base_url = "http://127.0.0.1:11434/v1" # ✅ Correct - includes /v1
+ ```
+
+ Common mistake:
+ ```toml
+ base_url = "http://127.0.0.1:11434" # ❌ Will cause 404 errors
+ ```
+
+ If you see 404s, verify the model is installed with `ollama ls` and test the endpoint:
+ ```bash
+ curl http://127.0.0.1:11434/v1/models
+ ```
+
+7. Configure `marimo.toml` (or use Settings):
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "ollama/llama3.1:latest"
+edit_model = "ollama/codellama"
+autocomplete_model = "ollama/codellama" # or another model from `ollama ls`
+```
+
+### OpenAI-compatible providers
+
+Many providers expose OpenAI-compatible endpoints. Point `base_url` at the provider and use their models.
+Common examples include [GROQ](https://console.groq.com/docs/openai), DeepSeek, xAI, Together AI, and LM Studio.
+
+**Requirements**
+
+* Provider API key
+* Provider OpenAI-compatible `base_url`
+* `pip install openai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "provider-x/some-model"
+
+[ai.open_ai_compatible]
+api_key = "..."
+base_url = "https://api.provider-x.com/"
+```
+
+??? tip "Use the `/v1` path if required by your provider"
+ Some OpenAI-compatible providers expose their API under `/v1` (e.g., `https://host/v1`). If you see 404s, add `/v1` to your `base_url`.
+
+??? tip "Using OpenAI-compatible providers"
+
+ === "Via UI Settings"
+
+ 1. Open marimo's Settings panel
+ 2. Navigate to the AI section
+ 3. Enter your provider's API key in the "OpenAI-Compatible" section
+ 4. Set Base URL to your provider (e.g., `https://api.provider-x.com`)
+ 5. Set Model to your provider's model slug
+
+ === "Via marimo.toml"
+
+ Add this to `marimo.toml`:
+
+ ```toml
+ [ai.models]
+ chat_model = "provider-x/some-model"
+
+ [ai.open_ai_compatible]
+ api_key = "px-..."
+ base_url = "https://api.provider-x.com/"
+ ```
+
+### DeepSeek
+
+Use DeepSeek via its OpenAI‑compatible API.
+
+**Requirements**
+
+* DeepSeek API key
+* `pip install openai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "deepseek/deepseek-chat" # or "deepseek-reasoner"
+
+[ai.open_ai_compatible]
+api_key = "dsk-..."
+base_url = "https://api.deepseek.com/"
+```
+
+### xAI
+
+Use Grok models via xAI's OpenAI‑compatible API.
+
+**Requirements**
+
+* xAI API key
+* `pip install openai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "xai/grok-2-latest"
+
+[ai.open_ai_compatible]
+api_key = "xai-..."
+base_url = "https://api.x.ai/v1/"
+```
+
+### LM Studio
+
+Connect to a local model served by LM Studio's OpenAI‑compatible endpoint.
+
+**Requirements**
+
+* Install LM Studio and start its server
+* `pip install openai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "lmstudio/qwen2.5-coder-7b"
+
+[ai.open_ai_compatible]
+base_url = "http://127.0.0.1:1234/v1" # LM Studio server
+```
+
+### Mistral
+
+Use Mistral via its OpenAI‑compatible API.
+
+**Requirements**
+
+* Mistral API key
+* `pip install openai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "mistral/mistral-small-latest" # e.g., codestral-latest, mistral-large-latest
+
+[ai.open_ai_compatible]
+api_key = "mistral-..."
+base_url = "https://api.mistral.ai/v1/"
+```
+
+### Together AI
+
+Access multiple hosted models via Together AI's OpenAI‑compatible API.
+
+**Requirements**
+
+* Together AI API key
+* `pip install openai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "together/mistralai/Mixtral-8x7B-Instruct-v0.1"
+
+[ai.open_ai_compatible]
+api_key = "tg-..."
+base_url = "https://api.together.xyz/v1/"
+```
+
+### Vercel v0
+
+Use Vercel's v0 OpenAI‑compatible models for app-oriented generation.
+
+**Requirements**
+
+* v0 API key
+* `pip install openai`
+
+**Configuration**
+
+```toml title="marimo.toml"
+[ai.models]
+chat_model = "v0/v0-1.5-md"
+
+[ai.open_ai_compatible]
+api_key = "v0-..."
+base_url = "https://api.v0.dev/" # Verify the endpoint in v0 docs
+```
+
+See the [LiteLLM provider list](https://litellm.vercel.app/docs/providers) for more options. For non‑compatible APIs, submit a
+[feature request](https://github.com/marimo-team/marimo/issues/new?template=feature_request.yaml).
diff --git a/docs/guides/editor_features/ai_completion.md b/docs/guides/editor_features/ai_completion.md
index ca3b6911fa..e3f9e1131a 100644
--- a/docs/guides/editor_features/ai_completion.md
+++ b/docs/guides/editor_features/ai_completion.md
@@ -144,250 +144,7 @@ You can configure the following providers:
* Ollama
* and any OpenAI-compatible provider
-To locate your configuration file, run:
-
-```bash
-marimo config show
-```
-
-At the top, the path to your `marimo.toml` file will be shown.
-
-#### Model roles and provider routing
-
-marimo supports three different AI model roles, each serving a specific purpose:
-
-* **`chat_model`**: Used for the chat panel
-* **`edit_model`**: Used for refactoring existing cells (Ctrl/Cmd-Shift-E) and generating new cells with the "Generate with AI" button.
-* **`autocomplete_model`**: Used for inline code autocompletion
-
-Models are specified using the format `provider/model-name`, where the provider prefix routes the request to the appropriate configuration section:
-
-```toml title="marimo.toml"
-[ai.models]
-chat_model = "openai/gpt-4o-mini" # Routes to OpenAI config
-edit_model = "anthropic/claude-3-sonnet" # Routes to Anthropic config
-autocomplete_model = "ollama/codellama" # Routes to Ollama config
-```
-
-#### Custom models
-
-You can also add custom models to appear in the model selection dropdown.
-
-```toml title="marimo.toml"
-[ai.models]
-custom_models = ["ollama/somemodel"]
-```
-
-Below we describe how to connect marimo to your AI provider.
-
-#### OpenAI
-
-1. Install openai: `pip install openai`
-
-2. Add the following to your `marimo.toml` (or configure in the UI settings in the editor):
-
-```toml title="marimo.toml"
-[ai.models]
-# Choose for chat
-chat_model = "openai/gpt-4o-mini"
-# Or choose for edit
-edit_model = "openai/gpt-4o-mini"
-
-[ai.open_ai]
-# Get your API key from https://platform.openai.com/account/api-keys
-api_key = "sk-proj-..."
-# Available models: gpt-4o-mini, gpt-4o, gpt-4, gpt-3.5-turbo
-# See https://platform.openai.com/docs/models for all available models
-
-# Change the base_url if you are using a different OpenAI-compatible API
-# different from one of the providers listed below
-base_url = "https://api.openai.com/v1"
-```
-
-#### Anthropic
-
-To use Anthropic with marimo:
-
-1. Sign up for an account at [Anthropic](https://console.anthropic.com/) and grab your [Anthropic Key](https://console.anthropic.com/settings/keys).
-2. Add the following to your `marimo.toml` (or configure in the UI settings in the editor):
-
-```toml title="marimo.toml"
-[ai.models]
-chat_model = "anthropic/claude-3-7-sonnet-latest"
-# or any model from https://docs.anthropic.com/en/docs/about-claude/models
-
-[ai.anthropic]
-api_key = "sk-ant-..."
-```
-
-#### AWS Bedrock
-
-AWS Bedrock provides access to foundation models from leading AI companies through a unified AWS API.
-
-To use AWS Bedrock with marimo:
-
-1. Set up an [AWS account](https://aws.amazon.com/) with access to the AWS Bedrock service.
-2. [Enable model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) for the specific models you want to use in the AWS Bedrock console.
-3. Install the boto3 Python client: `pip install boto3`
-4. Configure AWS credentials using one of these methods:
- * AWS CLI: Run `aws configure` to set up credentials
- * Environment variables: Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
- * AWS credentials file at `~/.aws/credentials`
-5. Add the following to your `marimo.toml`:
-
-```toml title="marimo.toml"
-[ai.models]
-chat_model = "bedrock/anthropic.claude-3-sonnet-latest"
-
-[ai.bedrock]
-region_name = "us-east-1" # AWS region where Bedrock is available
-# Optional AWS profile name (from ~/.aws/credentials)
-profile_name = "my-profile"
-```
-
-If you're using an AWS named profile different from your default, specify the profile_name. For explicit credentials (not recommended), you can use environment variables instead.
-
-#### Google AI
-
-To use Google AI with marimo:
-
-1. Sign up for an account at [Google AI Studio](https://aistudio.google.com/app/apikey) and obtain your API key.
-2. Install the Google AI Python client: `pip install google-genai`
-3. Add the following to your `marimo.toml` (or configure in the UI settings in the editor):
-
-```toml title="marimo.toml"
-[ai.models]
-chat_model = "google/gemini-2.5-pro"
-# or any model from https://ai.google.dev/gemini-api/docs/models/gemini
-
-[ai.google]
-api_key = "AI..."
-```
-
-#### GitHub Copilot
-
-You can use your GitHub Copilot for code refactoring or the chat panel. This requires a GitHub Copilot subscription.
-
-1. Download the `gh` CLI from [here](https://cli.github.com/).
-2. Create a token with `gh auth token` and copy the token.
-3. Add the token to your `marimo.toml` (or configure in the UI settings in the editor).
-
-```toml title="marimo.toml"
-[ai.models]
-chat_model = "github/gpt-4o-mini"
-
-[ai.github]
-api_key = "gho_..."
-```
-
-??? question "My token starts with `ghp_` instead of `gho_`?"
-
- This usually happens when you previously authenticated `gh` by pasting a _personal_ access token (`ghp_...`). However, GitHub Copilot is not available through `ghp_...`, and you will encounter errors such as:
-
- > bad request: Personal Access Tokens are not supported for this endpoint
-
- To resolve this issue, you could switch to an _OAuth_ access token (`gho_...`):
-
- 1. Re-authenticate by running `gh auth login`.
- 2. Choose _Login with a web browser_ (instead of _Paste an authentication token_) this time.
-
-#### Local models with Ollama { #using-ollama }
-
-Ollama allows you to run open-source LLMs on your local machine. To integrate Ollama with marimo:
-
-1. Download and install [Ollama](https://ollama.com/).
-2. Download the model you want to use:
-
- ```bash
- # View available models at https://ollama.com/library
- ollama pull llama3.1
- ollama pull codellama # recommended for code generation
-
- # View your installed models
- ollama ls
- ```
-
-3. Start the Ollama server in a terminal:
-
- ```bash
- ollama serve
- # In a new terminal
- ollama run codellama # or any model from ollama ls
- ```
-
-4. Visit <http://127.0.0.1:11434> to confirm that the server is running.
-
-!!! note "Port already in use"
- If you get a "port already in use" error, you may need to close an existing Ollama instance. On Windows, click the up arrow in the taskbar, find the Ollama icon, and select "Quit". This is a known issue (see [Ollama Issue #3575](https://github.com/ollama/ollama/issues/3575)). Once you've closed the existing Ollama instance, you should be able to run `ollama serve` successfully.
-
-5. Open a new terminal and install the openai client (e.g. `pip install openai`, `uv add openai`)
-
-6. Start marimo:
-
- ```bash
- marimo edit notebook.py
- ```
-
-??? warning "Important: Use the `/v1` endpoint"
-
- marimo requires Ollama's OpenAI-compatible API endpoint. Always ensure your `base_url` includes the `/v1` path:
-
- ```toml
- [ai.ollama]
- base_url = "http://127.0.0.1:11434/v1" # ✅ Correct - includes /v1
- ```
-
- **Common mistake:**
- ```toml
- base_url = "http://127.0.0.1:11434" # ❌ Will cause 404 errors
- ```
-
- If you encounter 404 errors, verify your model is installed with `ollama ls` and test the endpoint:
- ```bash
- curl http://127.0.0.1:11434/v1/models
- ```
-
-7. Add the following to your `marimo.toml` (or configure in the UI settings in the editor):
-
-```toml title="marimo.toml"
-[ai.models]
-chat_model = "ollama/llama3.1:latest"
-edit_model = "ollama/codellama"
-autocomplete_model = "ollama/codellama" # or another model from `ollama ls`
-```
-
-#### Other AI providers
-
-marimo supports OpenAI's API by default. Many providers offer OpenAI API-compatible endpoints, which can be used by simply changing the `base_url` in your configuration. For example, providers like [GROQ](https://console.groq.com/docs/openai) and [DeepSeek](https://platform.deepseek.com) follow this pattern.
-
-??? tip "Using OpenAI-compatible providers (e.g., DeepSeek)"
-
- === "Via UI Settings"
-
- 1. Open marimo's Settings panel
- 2. Navigate to the AI section
- 3. Enter your provider's API key in the "OpenAI API Key" field
- 4. Under AI Assist settings:
-
- - Set Base URL to your provider's endpoint (e.g., `https://api.deepseek.com`)
- - Set Model to your chosen model (e.g., `deepseek-chat` or `deepseek-reasoner`)
-
- === "Via marimo.toml"
-
- Add the following configuration to your `marimo.toml` file:
-
- ```toml
- [ai.models]
- chat_model = "deepseek/deepseek-chat" # or "deepseek-reasoner"
-
- [ai.open_ai_compatible]
- api_key = "dsk-..." # Your provider's API key
- base_url = "https://api.deepseek.com/"
- ```
-
-For a comprehensive list of compatible providers and their configurations, please refer to the [liteLLM Providers documentation](https://litellm.vercel.app/docs/providers).
-
-For providers not compatible with OpenAI's API, please submit a [feature request](https://github.com/marimo-team/marimo/issues/new?template=feature_request.yaml) or "thumbs up" an existing one.
+See the [llm_providers](../configuration/llm_providers.md) guide for detailed instructions on how to configure each provider.
## Agents
diff --git a/mkdocs.yml b/mkdocs.yml
index b979736ab1..4b6b7633bb 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -172,6 +172,7 @@ nav:
- Configuration:
- Configuration: guides/configuration/index.md
- Runtime configuration: guides/configuration/runtime_configuration.md
+ - LLM Providers: guides/configuration/llm_providers.md
- HTML head: guides/configuration/html_head.md
- Theming: guides/configuration/theming.md
- Snippets: guides/configuration/snippets.md