A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.
Python artificial-intelligencedagdata-sciencedata-visualizationdataflowdeveloper-toolsmachine-learningnotebookspipelinepythonreactivesqlweb-app
feat(ai): add capabilities, docs, caching (#10052)
Shahmir Varqha985a19cd59dc
diff --git a/docs/guides/editor_features/ai_completion.md b/docs/guides/editor_features/ai_completion.mdindex 421ed5c14f..2615662c3a 100644--- a/docs/guides/editor_features/ai_completion.md+++ b/docs/guides/editor_features/ai_completion.md@@ -86,7 +86,8 @@ The chat panel currently supports the following modes:- **Manual**: No tool access; the AI responds based only on the conversation and manually injected context- **Ask**: Enables read-only [AI tools](tools.md) and [tools from added MCP Client servers](mcp.md#mcp-client) for context gathering, allowing the assistant to inspect your notebooks-- **Agent** (beta): Enables all tools in **Ask Mode** plus additional tools to [edit notebook cells (add, remove, update) and run stale cells](tools.md#editing-agent-mode-only).+- **Agent**: Enables all tools in **Ask Mode** plus additional tools to [edit notebook cells (add, remove, update) and run stale cells](tools.md#editing-agent-mode-only)+- **Code mode**: Gives the assistant access to the [notebook kernel](tools.md#code-mode) which allows the assistant to inspect the marimo runtime and manipulate the notebook in powerful ways.??? tip "See the chat panel in action"diff --git a/docs/guides/editor_features/tools.md b/docs/guides/editor_features/tools.mdindex 0ea388ea38..793e0faaff 100644--- a/docs/guides/editor_features/tools.md+++ b/docs/guides/editor_features/tools.md@@ -7,7 +7,16 @@ marimo exposes a set of tools that allow AI assistants to interact with your not## Using tools-These tools are available when using the [chat panel in ask mode](ai_completion.md#chat-panel). External AI applications can also access these tools through the [marimo MCP server](mcp.md#mcp-server).+Tool availability depends on which [chat panel mode](ai_completion.md#chat-panel) you use:++| Mode | Marimo notebook tools |+|------|----------------------|+| **Manual** | — |+| **Ask** | Read-only inspection, data, debugging, and reference tools |+| **Agent** | All **Ask** tools plus editing tools |+| **Code mode** | Code execution tool and on-demand reference guides (see below) |++External AI applications can also access the **Ask** and **Agent** notebook tools through the [marimo MCP server](mcp.md#mcp-server).## Available tools@@ -51,6 +60,57 @@ These tools are available when using the [chat panel in ask mode](ai_completion.| **edit_notebook** | Add, remove, or update cells in the notebook. Takes cell operations and modifications as parameters. Allows the AI agent to generate diffs that modify notebook structure and content. || **run_stale_cells** | Run cells that are stale (outdated due to upstream changes). Triggers execution of affected cells to update the notebook state. |+## Web search and fetch++In any chat panel mode (**Manual**, **Ask**, **Agent**, or **Code mode**), marimo can give the assistant access to web search and URL fetching. These are [provider-adaptive capabilities](https://pydantic.dev/docs/ai/core-concepts/capabilities/#provider-adaptive-tools) from [Pydantic AI](https://ai.pydantic.dev): marimo enables them automatically based on your installed packages and the model you are using.++### How capabilities are enabled++marimo picks the best available option for each capability:++| Capability | Local fallback (installed in your environment) | Native (model provider supports it) |+|------------|-----------------------------------------------|-------------------------------------|+| **Web search** | DuckDuckGo search when `ddgs` is installed | Provider-native web search (e.g. Anthropic, OpenAI Responses) |+| **Web fetch** | URL fetching via `markdownify` when installed | Provider-native web fetch |+| **X search** | — | xAI models with native X search support |++Local fallbacks take priority when their packages are installed. Otherwise, marimo uses the provider's native tools when your configured model supports them.++### Install local web search and fetch++To enable web search and fetch on any model — including local models via Ollama — install the optional Pydantic AI extras:++```bash+pip install "pydantic-ai-slim[duckduckgo,web-fetch]"+```++### Use provider-native tools++When local packages are not installed, marimo enables native tools only if your model supports them. For example:++- **Anthropic** and **OpenAI Responses** models can use native web search and web fetch+- **xAI** models (e.g. `xai/grok-2-latest`) can use native web search and X search++Configure xAI in your `marimo.toml` or through the notebook settings — see the [xAI provider guide](../configuration/llm_providers.md#xai).+++## Code mode++!!! warning "Experimental"+ Code mode gives the assistant direct access to your notebook's kernel so it can make destructive changes to your notebook.++Code mode is available from the chat panel mode selector. Instead of the inspection and editing tools above, the assistant uses a different toolset oriented around running Python in the live kernel:++| Tool / capability | Description |+|-------------------|-------------|+| **execute_code** | Run Python in the notebook kernel's scratchpad. The assistant uses this for all notebook mutations — adding cells, updating code, inspecting variables, and running logic. |+| **gotchas** | On-demand reference for name redefinition, cached module proxies, and other notebook traps. |+| **notebook-improvements** | On-demand reference for improving, optimizing, or cleaning up an existing notebook. |+| **rich-representations** | On-demand reference for custom widgets, visual encodings, and interactive output. |++Code mode loads the [marimo pair](../generate_with_ai/marimo_pair.md) skill as its system prompt, so the assistant follows the same conventions as external agent CLIs paired on your notebook.++## Related documentation- [Model Context Protocol (MCP)](mcp.md) - Learn how to expose tools through the marimo MCP serverdiff --git a/marimo/_dependencies/dependencies.py b/marimo/_dependencies/dependencies.pyindex 8b3e32006e..349609deee 100644--- a/marimo/_dependencies/dependencies.py+++ b/marimo/_dependencies/dependencies.py@@ -253,6 +253,8 @@ class DependencyManager:"pydantic_ai", pkg_name_to_install="pydantic-ai-slim")pydantic = Dependency("pydantic")+ duckduckgo_search = Dependency("ddgs") # For web search+ markdownify = Dependency("markdownify") # For web fetchzmq = Dependency("zmq") # pyzmq for sandbox IPC kernelstorch = Dependency("torch")flax = Dependency("flax")diff --git a/marimo/_server/ai/providers.py b/marimo/_server/ai/providers.pyindex 79eebeb619..7fe043cad6 100644--- a/marimo/_server/ai/providers.py+++ b/marimo/_server/ai/providers.py@@ -2,6 +2,7 @@from __future__ import annotationsimport functools+import inspectimport osfrom abc import ABC, abstractmethodfrom dataclasses import dataclass@@ -44,16 +45,23 @@ from marimo._utils.http import HTTPStatusfrom marimo._utils.typing import overrideif TYPE_CHECKING:+ from collections.abc import Sequence+from openai import AsyncOpenAIfrom pydantic_ai import Agent, DeferredToolRequests, FunctionToolset+ from pydantic_ai.capabilities import AbstractCapabilityfrom pydantic_ai.models import Model- from pydantic_ai.models.bedrock import BedrockConverseModel+ from pydantic_ai.models.anthropic import AnthropicModelSettings+ from pydantic_ai.models.bedrock import (+ BedrockConverseModel,+ BedrockModelSettings,+ )from pydantic_ai.models.google import GoogleModelfrom pydantic_ai.models.openai import (- OpenAIChatModel,OpenAIResponsesModel,OpenAIResponsesModelSettings,)+ from pydantic_ai.models.openrouter import OpenRouterModelSettingsfrom pydantic_ai.output import OutputSpecfrom pydantic_ai.providers import Providerfrom pydantic_ai.providers.anthropic import (@@ -65,6 +73,7 @@ if TYPE_CHECKING:from pydantic_ai.providers.google import GoogleProvider as PydanticGooglefrom pydantic_ai.providers.openai import OpenAIProvider as PydanticOpenAIfrom pydantic_ai.settings import ModelSettings, ThinkingLevel+ from pydantic_ai.toolsets import AbstractToolsetfrom pydantic_ai.ui.vercel_ai.request_types import UIMessage, UIMessagePartfrom starlette.requests import Requestfrom starlette.responses import StreamingResponse@@ -83,6 +92,12 @@ class StreamOptions:accept: str | None = None+@dataclass(frozen=True)+class BaseModelSettings:+ max_tokens: int | None+ thinking: ThinkingLevel | None++ProviderT = TypeVar("ProviderT", bound="Provider", covariant=True)@@ -117,46 +132,66 @@ class PydanticProvider(ABC, Generic[ProviderT]):"""Create a provider for the given config."""@abstractmethod- def create_model(self, max_tokens: int | None) -> Model:- """Create a Pydantic AI model for the given max tokens."""+ def create_model(self) -> Model:+ """Create a Pydantic AI model."""def create_agent(self,*,name: str,max_tokens: int | None,- tools: list[ToolDefinition],+ tools: list[ToolDefinition] | None = None,+ toolsets: Sequence[AbstractToolset[None]] | None = None,+ extra_capabilities: Sequence[AbstractCapability[None]] | None = None,system_prompt: str,) -> Agent[None, DeferredToolRequests | str]:- """Create a Pydantic AI agent"""+ """Create a Pydantic AI agent."""from pydantic_ai import Agent- model = self.create_model(max_tokens)- toolset, output_type = self._get_toolsets_and_output_type(tools)+ model = self.create_model()+ capabilities = self._build_agent_capabilities(model)+ if extra_capabilities:+ capabilities.extend(extra_capabilities)+ agent_toolsets, output_type = self._resolve_agent_toolsets(+ tools or [], toolsets+ )+return Agent(model,name=name,- model_settings=self._build_agent_settings(model),- toolsets=[toolset] if tools else None,+ model_settings=self._build_model_settings(model, max_tokens),+ toolsets=agent_toolsets,instructions=system_prompt,+ capabilities=capabilities,output_type=output_type,+ deps_type=type(None),)- def _build_agent_settings(self, model: Model) -> ModelSettings | None:+ def _build_model_settings(+ self, model: Model, max_tokens: int | None+ ) -> ModelSettings | None:"""Settings applied at agent level on every request."""- from pydantic_ai.settings import ModelSettings+ values = self._base_model_settings(model, max_tokens)+ settings: ModelSettings = {}- thinking = self._default_thinking(model)- if thinking is None:- return None+ if values.max_tokens is not None:+ settings["max_tokens"] = values.max_tokens+ if values.thinking is not None:+ settings["thinking"] = values.thinking+ return settings- if not (+ def _base_model_settings(+ self, model: Model, max_tokens: int | None+ ) -> BaseModelSettings:+ """Shared request setting values, before provider-specific keys."""+ thinking = self._default_thinking(model)+ if thinking is not None and not (profile_get(model.profile, "supports_thinking", False)or profile_get(model.profile, "thinking_always_enabled", False)):- return None+ thinking = None- return ModelSettings(thinking=thinking)+ return BaseModelSettings(max_tokens=max_tokens, thinking=thinking)def _default_thinking(self, model: Model) -> ThinkingLevel | None:"""Default unified thinking flag. Return None to skip."""@@ -178,11 +213,7 @@ class PydanticProvider(ABC, Generic[ProviderT]):stream_options: StreamOptions,) -> StreamingResponse:"""Return a streaming response from the given messages. The response are AI SDK events."""- from pydantic_ai.ui.vercel_ai import VercelAIAdapter- from pydantic_ai.ui.vercel_ai.request_types import SubmitMessage-tools = (self.config.tools or []) + additional_tools- stream_options.span_info.tool_count = len(tools)agent = self.create_agent(name=stream_options.span_info.endpoint,@@ -190,13 +221,27 @@ class PydanticProvider(ABC, Generic[ProviderT]):tools=tools,system_prompt=system_prompt,)+ stream_options.span_info.tool_count = len(tools) + len(+ agent.root_capability.capabilities+ )++ return self._vercel_streaming_response(agent, messages, stream_options)++ def _vercel_streaming_response(+ self,+ agent: Agent[None, DeferredToolRequests | str],+ messages: list[ServerUIMessage],+ stream_options: StreamOptions,+ ) -> StreamingResponse:+ """Run `agent` over `messages` and return an AI SDK streaming response."""+ from pydantic_ai.ui.vercel_ai import VercelAIAdapter+ from pydantic_ai.ui.vercel_ai.request_types import SubmitMessagerun_input = SubmitMessage(id=generate_id("submit-message"),trigger="submit-message",messages=self.convert_messages(messages),)-adapter = VercelAIAdapter(agent=agent,run_input=run_input,@@ -220,14 +265,15 @@ class PydanticProvider(ABC, Generic[ProviderT]):from pydantic_ai.ui.vercel_ai import VercelAIAdaptertools = (self.config.tools or []) + additional_tools- span_info.tool_count = len(tools)-agent = self.create_agent(name=span_info.endpoint,max_tokens=max_tokens,tools=tools,system_prompt=system_prompt,)+ span_info.tool_count = len(tools) + len(+ agent.root_capability.capabilities+ )with trace_completion(span_info):result = await agent.run(@@ -247,42 +293,85 @@ class PydanticProvider(ABC, Generic[ProviderT]):max_tokens: int | None,stream_options: StreamOptions,) -> StreamingResponse:- """Experimental method to return code-mode streaming responses"""- from pydantic_ai import Agent- from pydantic_ai.ui.vercel_ai import VercelAIAdapter- from pydantic_ai.ui.vercel_ai.request_types import SubmitMessage-+ """Return code-mode streaming responses"""from marimo._server.ai.tools.code_mode import (build_execute_code_toolset,references_capability,)- model = self.create_model(max_tokens=max_tokens)- capabilities = references_capability()- agent = Agent(- model,- model_settings=self._build_agent_settings(model),+ agent = self.create_agent(+ name=stream_options.span_info.endpoint,+ max_tokens=max_tokens,+ tools=[],toolsets=[build_execute_code_toolset(session, request)],- instructions=system_prompt,- capabilities=capabilities,+ extra_capabilities=references_capability(),+ system_prompt=system_prompt,)- run_input = SubmitMessage(- id=generate_id("submit-message"),- trigger="submit-message",- messages=self.convert_messages(messages),+ # One for the execute code toolset, plus the agent's native capabilities+ stream_options.span_info.tool_count = 1 + len(+ agent.root_capability.capabilities)- stream_options.span_info.tool_count = 1 + len(capabilities)- adapter = VercelAIAdapter(- agent=agent,- run_input=run_input,- accept=stream_options.accept,- sdk_version=AI_SDK_VERSION,+ return self._vercel_streaming_response(agent, messages, stream_options)++ def _build_agent_capabilities(+ self, model: Model+ ) -> list[AbstractCapability[None]]:+ """+ Build agent capabilities for the given model.++ Referenced from https://pydantic.dev/docs/ai/core-concepts/capabilities#provider-adaptive-tools+ """+ from pydantic_ai.capabilities import WebFetch, WebSearch, XSearch+ from pydantic_ai.native_tools import (+ WebFetchTool,+ WebSearchTool,+ XSearchTool,)- event_stream = adapter.run_stream()- event_stream = trace_stream(event_stream, stream_options.span_info)- return adapter.streaming_response(event_stream)++ supported = profile_get(model.profile, "supported_native_tools", [])+ LOGGER.debug(+ "Supported native tools: %s for model: %s", supported, self.model+ )+ capabilities: list[AbstractCapability[None]] = []++ if DependencyManager.duckduckgo_search.has():+ capabilities.append(WebSearch(local="duckduckgo"))+ elif WebSearchTool in supported:+ capabilities.append(WebSearch())++ if DependencyManager.markdownify.has():+ capabilities.append(WebFetch(local=True))+ elif WebFetchTool in supported:+ capabilities.append(WebFetch())++ if XSearchTool in supported:+ capabilities.append(XSearch())++ LOGGER.debug(+ "Capabilities: %s for model: %s", capabilities, self.model+ )+ return capabilities++ def _resolve_agent_toolsets(+ self,+ tools: list[ToolDefinition],+ extra_toolsets: Sequence[AbstractToolset[None]] | None,+ ) -> tuple[+ list[AbstractToolset[None]] | None,+ OutputSpec[str | DeferredToolRequests],+ ]:+ all_toolsets: list[AbstractToolset[None]] = []+ if tools:+ toolset, output_type = self._get_toolsets_and_output_type(tools)+ all_toolsets.append(toolset)+ else:+ output_type = str++ if extra_toolsets:+ all_toolsets.extend(extra_toolsets)+ return all_toolsets or None, output_typedef _get_toolsets_and_output_type(self, tools: list[ToolDefinition]@@ -338,17 +427,10 @@ class GoogleProvider(PydanticProvider["PydanticGoogle"]):return provider@override- def create_model(self, max_tokens: int | None) -> GoogleModel:- from pydantic_ai.models.google import GoogleModel, GoogleModelSettings+ def create_model(self) -> GoogleModel:+ from pydantic_ai.models.google import GoogleModel- settings: GoogleModelSettings = (- {"max_tokens": max_tokens} if max_tokens is not None else {}- )- return GoogleModel(- model_name=self.model,- provider=self.provider,- settings=settings,- )+ return GoogleModel(model_name=self.model, provider=self.provider)class OpenAIClientMixin:@@ -448,30 +530,33 @@ class OpenAIProvider(OpenAIClientMixin, PydanticProvider["PydanticOpenAI"]):return PydanticOpenAI(openai_client=client)@override- def create_model(self, max_tokens: int | None) -> OpenAIResponsesModel:+ def create_model(self) -> OpenAIResponsesModel:from pydantic_ai.models.openai import (OpenAIResponsesModel,)- settings: OpenAIResponsesModelSettings = (- {"max_tokens": max_tokens} if max_tokens is not None else {}- )return OpenAIResponsesModel(- model_name=self.model,- provider=self.provider,- settings=settings,+ model_name=self.model, provider=self.provider)@override- def _build_agent_settings(self, model: Model) -> ModelSettings | None:+ def _build_model_settings(+ self, model: Model, max_tokens: int | None+ ) -> ModelSettings | None:# `reasoning.summary` is only valid for OpenAI reasoning models (gpt-5# and the o-series).- settings = super()._build_agent_settings(model)- if settings is not None and "thinking" in settings:- extra: OpenAIResponsesModelSettings = {- "openai_reasoning_summary": self.DEFAULT_REASONING_SUMMARY,- }- settings.update(extra)+ values = self._base_model_settings(model, max_tokens)+ settings: OpenAIResponsesModelSettings = {}++ if values.max_tokens is not None:+ settings["max_tokens"] = values.max_tokens++ if values.thinking is not None:+ settings["thinking"] = values.thinking+ settings["openai_reasoning_summary"] = (+ self.DEFAULT_REASONING_SUMMARY+ )+return settings@override@@ -725,54 +810,32 @@ class CustomProvider(OpenAIClientMixin, PydanticProvider["Provider"]):def _create_custom_provider(self, provider_class: type[Provider], config: AnyProviderConfig) -> Provider:- """- Create a custom provider based on the provider class. These providers are not OpenAI-compatible.- Import on-demand to avoid requiring the provider packages to be installed.- """+ """Instantiate a non-OpenAI-compatible pydantic-ai provider."""provider_name = provider_class.__name__- LOGGER.debug(f"Creating custom provider: {provider_name}")-- if provider_name == "CerebrasProvider":- from pydantic_ai.providers.cerebras import CerebrasProvider-- return CerebrasProvider(api_key=config.api_key)- if provider_name == "CohereProvider":- from pydantic_ai.providers.cohere import CohereProvider+ LOGGER.debug("Creating custom provider: %s", provider_name)- return CohereProvider(api_key=config.api_key)- if provider_name == "GroqProvider":- from pydantic_ai.providers.groq import GroqProvider-- return GroqProvider(- api_key=config.api_key, base_url=config.base_url+ kwargs: dict[str, str] = {}+ try:+ params = inspect.signature(provider_class).parameters+ if "api_key" in params and config.api_key:+ kwargs["api_key"] = config.api_key+ if "base_url" in params and config.base_url:+ kwargs["base_url"] = config.base_url+ except (TypeError, ValueError):+ LOGGER.debug(+ "Could not inspect signature for provider %s", provider_name)- if provider_name == "HuggingFaceProvider":- from pydantic_ai.providers.huggingface import HuggingFaceProvider-- if config.base_url:- return HuggingFaceProvider(- api_key=config.api_key, base_url=config.base_url- )- return HuggingFaceProvider(api_key=config.api_key)- if provider_name == "MistralProvider":- from pydantic_ai.providers.mistral import MistralProvider-- return MistralProvider(api_key=config.api_key)- LOGGER.warning(- f"Unknown provider: {provider_name}. Create a GitHub issue to request support."- )- # Try returning provider directly (use env vars)try:- return provider_class()+ return provider_class(**kwargs)except Exception as e:from pydantic_ai.providers.openai import (OpenAIProvider as PydanticOpenAI,)fallback_msg = (- f"Failed to create provider {provider_class.__name__}: {e}. "+ f"Failed to create provider {provider_name}: {e}. "f"Falling back to OpenAIProvider.")LOGGER.warning(fallback_msg)@@ -780,38 +843,17 @@ class CustomProvider(OpenAIClientMixin, PydanticProvider["Provider"]):return PydanticOpenAI(openai_client=client)@override- def create_model(self, max_tokens: int | None) -> OpenAIChatModel:- """Default to OpenAIChatModel"""-- from pydantic_ai.models.openai import (- OpenAIChatModel,- OpenAIChatModelSettings,- )-- settings: OpenAIChatModelSettings = (- {"max_tokens": max_tokens} if max_tokens is not None else {}- )- return OpenAIChatModel(- model_name=self.model,- provider=self.provider,- settings=settings,- )-- @override- def create_agent(- self,- *,- name: str,- max_tokens: int | None,- tools: list[ToolDefinition],- system_prompt: str,- ) -> Agent[None, DeferredToolRequests | str]:- """Create a Pydantic AI agent"""- from pydantic_ai import Agent, UserError+ def create_model(self) -> Model:+ """+ Infer the model from pydantic-ai's registry, falling back to a+ generic OpenAI-compatible chat model.+ """+ from pydantic_ai import UserErrorfrom pydantic_ai.models import infer_model+ from pydantic_ai.models.openai import OpenAIChatModeltry:- model = infer_model(+ return infer_model(f"{self._provider_name}:{self.model}",provider_factory=lambda _: self.provider,)@@ -821,27 +863,36 @@ class CustomProvider(OpenAIClientMixin, PydanticProvider["Provider"]):"Falling back to OpenAIChatModel.")LOGGER.debug(model_not_found_msg)- model = self.create_model(max_tokens)except Exception as e:LOGGER.error(f"Error creating model: {e}. Falling back to OpenAIChatModel.")- model = self.create_model(max_tokens)- agent_settings: ModelSettings = (- {"max_tokens": max_tokens} if max_tokens is not None else {}- )- agent_settings.update(self._build_agent_settings(model) or {})+ return OpenAIChatModel(model_name=self.model, provider=self.provider)- toolset, output_type = self._get_toolsets_and_output_type(tools)- return Agent(- model,- name=name,- model_settings=agent_settings,- toolsets=[toolset] if tools else None,- instructions=system_prompt,- output_type=output_type,- )+ @override+ def _build_model_settings(+ self, model: Model, max_tokens: int | None+ ) -> ModelSettings:+ values = self._base_model_settings(model, max_tokens)+ model_settings: ModelSettings = {}+ if values.max_tokens is not None:+ model_settings["max_tokens"] = values.max_tokens+ if values.thinking is not None:+ model_settings["thinking"] = values.thinking++ if self._provider_name == "openrouter":+ openrouter_settings: OpenRouterModelSettings = {}+ if values.max_tokens is not None:+ openrouter_settings["max_tokens"] = values.max_tokens+ if values.thinking is not None:+ openrouter_settings["thinking"] = values.thinking+ openrouter_settings["openrouter_cache_instructions"] = True+ openrouter_settings["openrouter_cache_messages"] = True+ openrouter_settings["openrouter_cache_tool_definitions"] = "1h"+ return openrouter_settings++ return model_settings@overridedef _default_thinking(self, model: Model) -> ThinkingLevel | None:@@ -862,23 +913,29 @@ class AnthropicProvider(PydanticProvider["PydanticAnthropic"]):return PydanticAnthropic(api_key=config.api_key)@override- def create_model(self, max_tokens: int | None) -> Model:- from pydantic_ai.models.anthropic import (- AnthropicModel,- AnthropicModelSettings,- )+ def create_model(self) -> Model:+ from pydantic_ai.models.anthropic import AnthropicModel- settings: AnthropicModelSettings = {- "max_tokens": max_tokens- if max_tokens is not None- else ANTHROPIC_DEFAULT_MAX_TOKENS- }+ return AnthropicModel(model_name=self.model, provider=self.provider)- return AnthropicModel(- model_name=self.model,- provider=self.provider,- settings=settings,+ @override+ def _build_model_settings(+ self, model: Model, max_tokens: int | None+ ) -> AnthropicModelSettings:+ # Anthropic provider needs to set the max tokens to 32768+ # https://github.com/marimo-team/marimo/pull/9703+ values = self._base_model_settings(+ model,+ max_tokens+ if max_tokens is not None+ else ANTHROPIC_DEFAULT_MAX_TOKENS,)+ settings: AnthropicModelSettings = {"anthropic_cache": True}+ if values.max_tokens is not None:+ settings["max_tokens"] = values.max_tokens+ if values.thinking is not None:+ settings["thinking"] = values.thinking+ return settings@overridedef convert_messages(@@ -937,21 +994,29 @@ class BedrockProvider(PydanticProvider["PydanticBedrock"]):return PydanticBedrock(region_name=config.base_url)@override- def create_model(self, max_tokens: int | None) -> BedrockConverseModel:- from pydantic_ai.models.bedrock import (- BedrockConverseModel,- BedrockModelSettings,- )+ def create_model(self) -> BedrockConverseModel:+ from pydantic_ai.models.bedrock import BedrockConverseModel- settings: BedrockModelSettings = (- {"max_tokens": max_tokens} if max_tokens is not None else {}- )return BedrockConverseModel(- model_name=self.model,- provider=self.provider,- settings=settings,+ model_name=self.model, provider=self.provider)+ @override+ def _build_model_settings(+ self, model: Model, max_tokens: int | None+ ) -> BedrockModelSettings:+ values = self._base_model_settings(model, max_tokens)+ settings: BedrockModelSettings = {+ "bedrock_cache_instructions": True,+ "bedrock_cache_messages": True,+ "bedrock_cache_tool_definitions": "1h",+ }+ if values.max_tokens is not None:+ settings["max_tokens"] = values.max_tokens+ if values.thinking is not None:+ settings["thinking"] = values.thinking+ return settings+def get_completion_provider(config: AnyProviderConfig, model: strdiff --git a/marimo/_server/ai/skills/marimo-pair/references/gotchas.md b/marimo/_server/ai/skills/marimo-pair/references/gotchas.mdindex 3f55dd5d10..147dc25f52 100644--- a/marimo/_server/ai/skills/marimo-pair/references/gotchas.md+++ b/marimo/_server/ai/skills/marimo-pair/references/gotchas.md@@ -1,7 +1,6 @@# Gotchas-Loaded on demand via the `gotchas` capability (`load_capability`). Do not read-this file from disk.+Loaded on demand via the `gotchas` capability (`load_capability`).## Private variables are cell-scopeddiff --git a/marimo/_server/ai/skills/marimo-pair/references/notebook-improvements.md b/marimo/_server/ai/skills/marimo-pair/references/notebook-improvements.mdindex db8f2d6fe4..ec6f7669a6 100644--- a/marimo/_server/ai/skills/marimo-pair/references/notebook-improvements.md+++ b/marimo/_server/ai/skills/marimo-pair/references/notebook-improvements.md@@ -1,7 +1,6 @@# Notebook ImprovementsLoaded on demand via the `notebook-improvements` capability (`load_capability`).-Do not read this file from disk.When the user asks to improve, optimize, or clean up their notebook, scan thecurrent cells for these opportunities. Use your judgment — don't over-apply,diff --git a/marimo/_server/ai/skills/marimo-pair/references/rich-representations.md b/marimo/_server/ai/skills/marimo-pair/references/rich-representations.mdindex 913f1fe6a8..67987ad588 100644--- a/marimo/_server/ai/skills/marimo-pair/references/rich-representations.md+++ b/marimo/_server/ai/skills/marimo-pair/references/rich-representations.md@@ -1,7 +1,6 @@# Rich RepresentationsLoaded on demand via the `rich-representations` capability (`load_capability`).-Do not read this file from disk.Custom visual encodings for data that go beyond standard charts and tables.diff --git a/tests/_server/ai/skills/test_skills_utils.py b/tests/_server/ai/skills/test_skills_utils.pyindex f5777dfbe2..ec177d702a 100644--- a/tests/_server/ai/skills/test_skills_utils.py+++ b/tests/_server/ai/skills/test_skills_utils.py@@ -46,5 +46,4 @@ def test_reference_files_point_to_load_capability() -> None:content = load_reference(reference_name)assert "load_capability" in contentassert f"`{reference_name}` capability" in content- assert "Do not read" in content- assert "from disk." in content+ assert "Loaded on demand via" in contentdiff --git a/tests/_server/ai/test_providers.py b/tests/_server/ai/test_providers.pyindex 719a565b09..1563e9fbd4 100644--- a/tests/_server/ai/test_providers.py+++ b/tests/_server/ai/test_providers.py@@ -1,6 +1,6 @@"""Tests for the LLM providers in marimo._server.ai.providers."""-from typing import Any+from typing import Any, castfrom unittest.mock import AsyncMock, MagicMock, patchimport pytest@@ -252,8 +252,8 @@ def test_openai_default_thinking(if provider_kind == "azure"else OpenAIProvider(model_name, config))- model = provider.create_model(max_tokens=512)- settings = provider._build_agent_settings(model)+ model = provider.create_model()+ settings = provider._build_model_settings(model, max_tokens=512)has_thinking = settings is not None and settings.get("thinking") is Truehas_summary = (@@ -273,31 +273,31 @@ def test_openai_default_thinking(pytest.param("claude-opus-4-7",# Opus 4.7 disallows sampling settings, so no temperature.- {"max_tokens": 1024},+ {"max_tokens": 1024, "anthropic_cache": True, "thinking": True},True,id="opus_4_7_adaptive_no_sampling",),pytest.param("claude-opus-4-6",- {"max_tokens": 1024},+ {"max_tokens": 1024, "anthropic_cache": True, "thinking": True},True,id="opus_4_6",),pytest.param("claude-sonnet-4-6",- {"max_tokens": 1024},+ {"max_tokens": 1024, "anthropic_cache": True, "thinking": True},True,id="sonnet_4_6",),pytest.param("claude-opus-4-5-20251101",- {"max_tokens": 1024},+ {"max_tokens": 1024, "anthropic_cache": True, "thinking": True},True,id="opus_4_5",),pytest.param("claude-3-7-sonnet-20250219",- {"max_tokens": 1024},+ {"max_tokens": 1024, "anthropic_cache": True, "thinking": True},True,id="sonnet_3_7",),@@ -309,7 +309,7 @@ def test_openai_default_thinking(# corrected upstream, behavior here will follow automatically.pytest.param("claude-3-5-sonnet-20241022",- {"max_tokens": 1024},+ {"max_tokens": 1024, "anthropic_cache": True, "thinking": True},True,id="sonnet_3_5_trusts_profile",),@@ -325,15 +325,15 @@ def test_anthropic_settings_split(expected_model_settings: dict[str, Any],expected_agent_thinking: bool,) -> None:- """Verify model-level settings and agent-level thinking flag."""+ """Verify request settings carry Anthropic max tokens/cache/thinking."""config = AnyProviderConfig(api_key="test-key", base_url=None)provider = AnthropicProvider(model_name, config)- model = provider.create_model(max_tokens=1024)- assert dict(model.settings) == expected_model_settings+ model = provider.create_model()+ model_settings = provider._build_model_settings(model, max_tokens=1024)+ assert dict(model_settings) == expected_model_settings- agent_settings = provider._build_agent_settings(model)actual_thinking = (- agent_settings is not None and agent_settings.get("thinking") is True+ model_settings is not None and model_settings.get("thinking") is True)assert actual_thinking == expected_agent_thinking@@ -370,23 +370,23 @@ def test_anthropic_thinking_payload_translation(and rejects `{"type": "enabled", "budget_tokens": ...}` with HTTP 400."""from pydantic_ai.models import ModelRequestParameters- from pydantic_ai.models.anthropic import AnthropicModel+ from pydantic_ai.models.anthropic import (+ AnthropicModel,+ AnthropicModelSettings,+ )config = AnyProviderConfig(api_key="test-key", base_url=None)provider = AnthropicProvider(model_name, config)- model = provider.create_model(max_tokens=1024)+ model = provider.create_model()assert isinstance(model, AnthropicModel)- # Combine model-level settings with the agent-level thinking flag the way- # pydantic-ai does at request time.- merged = dict(model.settings or {})- merged.update(provider._build_agent_settings(model) or {})-+ settings = provider._build_model_settings(model, max_tokens=1024)prepared_settings, prepared_params = model.prepare_request(- merged, ModelRequestParameters()+ settings, ModelRequestParameters())payload = model._translate_thinking( # type: ignore[attr-defined]- prepared_settings or {}, prepared_params+ cast("AnthropicModelSettings", prepared_settings or {}),+ prepared_params,)if expected_payload_kind == "adaptive":assert payload == {"type": "adaptive"}@@ -416,8 +416,8 @@ def test_google_default_thinking("""Google's profile correctly distinguishes thinking vs non-thinking models."""config = AnyProviderConfig(api_key="test-key", base_url=None)provider = GoogleProvider(model_name, config)- model = provider.create_model(max_tokens=512)- settings = provider._build_agent_settings(model)+ model = provider.create_model()+ settings = provider._build_model_settings(model, max_tokens=512)actual = settings is not None and settings.get("thinking") is Trueassert actual == expected_thinking@@ -460,6 +460,34 @@ async def test_completion_does_not_pass_redundant_instructions() -> None:assert instructions == "Test prompt"[email protected]("pydantic_ai")+async def test_completion_tool_count_includes_capabilities() -> None:+ """`completion` reports tools plus the agent's native capabilities, so its+ telemetry matches the streaming paths."""+ config = AnyProviderConfig(api_key="test-key", base_url="http://test-url")+ provider = OpenAIProvider("gpt-4", config)++ agent = MagicMock(name="agent")+ agent.root_capability.capabilities = [MagicMock(), MagicMock()]+ result = MagicMock()+ result.output = "hi"+ agent.run = AsyncMock(return_value=result)++ span_info = SpanInfo(endpoint="completion", model="openai/gpt-4")++ with patch.object(provider, "create_agent", return_value=agent):+ await provider.completion(+ messages=[],+ system_prompt="x",+ max_tokens=100,+ additional_tools=[MagicMock(name="tool")],+ span_info=span_info,+ )++ # 1 additional tool + 2 capabilities.+ assert span_info.tool_count == 3++@pytest.mark.skipif(not DependencyManager.anthropic.has()or not DependencyManager.pydantic_ai.has(),@@ -471,10 +499,9 @@ def test_anthropic_applies_default_floor_when_max_tokens_none() -> None:config = AnyProviderConfig(api_key="test-key", base_url=None)provider = AnthropicProvider("claude-sonnet-4-5", config)- model = provider.create_model(max_tokens=None)- assert (- dict(model.settings).get("max_tokens") == ANTHROPIC_DEFAULT_MAX_TOKENS- )+ model = provider.create_model()+ settings = provider._build_model_settings(model, max_tokens=None)+ assert dict(settings).get("max_tokens") == ANTHROPIC_DEFAULT_MAX_TOKENS@pytest.mark.skipif(@@ -486,8 +513,9 @@ def test_anthropic_override_wins_over_default_floor() -> None:"""An explicit max_tokens overrides the Anthropic default floor."""config = AnyProviderConfig(api_key="test-key", base_url=None)provider = AnthropicProvider("claude-sonnet-4-5", config)- model = provider.create_model(max_tokens=12345)- assert dict(model.settings).get("max_tokens") == 12345+ model = provider.create_model()+ settings = provider._build_model_settings(model, max_tokens=12345)+ assert dict(settings).get("max_tokens") == 12345@pytest.mark.requires("pydantic_ai")@@ -496,8 +524,9 @@ def test_openai_chat_omits_max_tokens_when_none() -> None:pydantic-ai falls through to the upstream provider's default."""config = AnyProviderConfig(api_key="test-key", base_url="http://test-url")provider = OpenAIProvider("gpt-4", config)- model = provider.create_model(max_tokens=None)- assert "max_tokens" not in dict(model.settings)+ model = provider.create_model()+ settings = provider._build_model_settings(model, max_tokens=None)+ assert "max_tokens" not in dict(settings or {})@pytest.mark.requires("pydantic_ai")@@ -505,22 +534,23 @@ def test_openai_chat_passes_explicit_max_tokens() -> None:"""Non-Anthropic providers pass through an explicit max_tokens."""config = AnyProviderConfig(api_key="test-key", base_url="http://test-url")provider = OpenAIProvider("gpt-4", config)- model = provider.create_model(max_tokens=12345)- assert dict(model.settings).get("max_tokens") == 12345+ model = provider.create_model()+ settings = provider._build_model_settings(model, max_tokens=12345)+ assert dict(settings or {}).get("max_tokens") == 12345@pytest.mark.requires("pydantic_ai")def test_custom_provider_agent_passes_explicit_max_tokens() -> None:- """The chat path builds the agent (not the model), so the agent's- model_settings must carry the explicit max_tokens."""- config = AnyProviderConfig(api_key="test-key", base_url="http://test-url")- provider = get_completion_provider(config, "openrouter/openai/gpt-4")- with patch("marimo._server.ai.providers.get_tool_manager") as mock_get_tm:- mock_get_tm.return_value = MagicMock()- agent = provider.create_agent(- name="test", max_tokens=12345, tools=[], system_prompt="x"- )- assert dict(agent.model_settings or {}).get("max_tokens") == 12345+ """Custom providers pass explicit max_tokens through agent settings."""+ config = AnyProviderConfig(+ api_key="test-key", base_url="https://my.internal.llm/v1"+ )+ provider = CustomProvider(+ AiModelId.from_model("my_provider/my-model"), config+ )+ model = provider.create_model()+ settings = provider._build_model_settings(model, max_tokens=12345)+ assert dict(settings).get("max_tokens") == 12345@pytest.mark.requires("pydantic_ai")@@ -533,7 +563,8 @@ def test_custom_provider_agent_omits_max_tokens_when_none() -> None:agent = provider.create_agent(name="test", max_tokens=None, tools=[], system_prompt="x")- assert "max_tokens" not in dict(agent.model_settings or {})+ settings = cast("dict[str, Any]", agent.model_settings or {})+ assert "max_tokens" not in settings@pytest.mark.parametrize(@@ -629,7 +660,7 @@ def test_custom_provider_inherits_profile_from_base_url() -> None:assert provider._provider_name == "deepseek"assert provider.provider.name == "deepseek"- model = provider.create_model(max_tokens=None)+ model = provider.create_model()profile = model.profileif isinstance(profile, dict):assert profile.get("openai_chat_thinking_field") == "reasoning_content"@@ -653,7 +684,7 @@ def test_custom_provider_unknown_base_url_stays_generic() -> None:assert provider._provider_name == "my_provider"assert provider.provider.name == "openai"- model = provider.create_model(max_tokens=None)+ model = provider.create_model()profile = model.profileif isinstance(profile, dict):assert profile.get("openai_chat_thinking_field") is None@@ -661,6 +692,24 @@ def test_custom_provider_unknown_base_url_stays_generic() -> None:assert profile.openai_chat_thinking_field is None[email protected]("pydantic_ai")+def test_custom_provider_create_model_infers_from_registry() -> None:+ """`create_model` resolves the model through pydantic-ai's registry using+ the `provider:model` id so we get the provider-tuned model class."""+ config = AnyProviderConfig(api_key="test-key", base_url="http://test-url")+ provider = get_completion_provider(config, "openrouter/openai/gpt-4")+ assert isinstance(provider, CustomProvider)++ sentinel = MagicMock(name="inferred-model")+ with patch(+ "pydantic_ai.models.infer_model", return_value=sentinel+ ) as mock_infer:+ model = provider.create_model()++ assert model is sentinel+ assert mock_infer.call_args.args[0] == "openrouter:openai/gpt-4"++@pytest.mark.requires("pydantic_ai")def test_custom_provider_known_name_not_overridden_by_base_url() -> None:"""A recognized provider name is used as-is; the base URL never overrides@@ -692,15 +741,25 @@ async def test_stream_completion_harness_wires_execute_code_toolset() -> None:span_info=SpanInfo(endpoint="chat", model="openai/gpt-4"),)+ def build_mock_agent(*_args: Any, **kwargs: Any) -> MagicMock:+ # `tool_count` reads back the agent's aggregated capabilities, so the+ # mock must expose the capabilities it was constructed with.+ agent = MagicMock(name="agent")+ agent.root_capability.capabilities = kwargs.get("capabilities", [])+ return agent+with (patch.object(provider, "create_model", return_value=MagicMock()),- patch.object(provider, "_build_agent_settings", return_value={}),+ patch.object(provider, "_build_model_settings", return_value={}),+ # Isolate from provider-adaptive web tools (covered separately) so the+ # only capabilities are the three references capabilities.+ patch.object(provider, "_build_agent_capabilities", return_value=[]),patch.object(provider, "convert_messages", return_value=[]),patch("marimo._server.ai.tools.code_mode.build_execute_code_toolset",return_value=toolset,) as mock_build_toolset,- patch("pydantic_ai.Agent") as mock_agent,+ patch("pydantic_ai.Agent", side_effect=build_mock_agent) as mock_agent,patch("pydantic_ai.ui.vercel_ai.VercelAIAdapter",return_value=adapter,@@ -733,3 +792,156 @@ async def test_stream_completion_harness_wires_execute_code_toolset() -> None:"rich-representations",}assert all(capability.defer_loading for capability in capabilities)++[email protected]("pydantic_ai")+def test_create_custom_provider_passes_supported_credentials() -> None:+ """`_create_custom_provider` constructs the resolved provider class,+ passing only the credentials its constructor accepts."""+ config = AnyProviderConfig(api_key="test-key", base_url="https://x/v1")+ provider = get_completion_provider(config, "openrouter/openai/gpt-4")+ assert isinstance(provider, CustomProvider)++ captured: dict[str, Any] = {}++ class FakeProvider:+ def __init__(+ self, *, api_key: str | None = None, base_url: str | None = None+ ) -> None:+ captured["api_key"] = api_key+ captured["base_url"] = base_url++ result = provider._create_custom_provider(FakeProvider, config) # type: ignore[arg-type]+ assert isinstance(result, FakeProvider)+ assert captured == {"api_key": "test-key", "base_url": "https://x/v1"}++[email protected]("pydantic_ai")+def test_create_custom_provider_omits_unsupported_base_url() -> None:+ """A provider whose constructor has no `base_url` parameter is built with+ just the api key, even when the config carries a base URL."""+ config = AnyProviderConfig(api_key="test-key", base_url="https://x/v1")+ provider = get_completion_provider(config, "openrouter/openai/gpt-4")+ assert isinstance(provider, CustomProvider)++ captured: dict[str, Any] = {}++ class ApiKeyOnlyProvider:+ def __init__(self, *, api_key: str | None = None) -> None:+ captured["api_key"] = api_key++ result = provider._create_custom_provider(ApiKeyOnlyProvider, config) # type: ignore[arg-type]+ assert isinstance(result, ApiKeyOnlyProvider)+ assert captured == {"api_key": "test-key"}++[email protected]("pydantic_ai")+def test_create_custom_provider_falls_back_to_openai_on_error() -> None:+ """If constructing the provider raises, we fall back to a generic+ OpenAI-compatible provider rather than propagating the error."""+ from pydantic_ai.providers.openai import (+ OpenAIProvider as PydanticOpenAI,+ )++ config = AnyProviderConfig(api_key="test-key", base_url="https://x/v1")+ provider = get_completion_provider(config, "openrouter/openai/gpt-4")+ assert isinstance(provider, CustomProvider)++ class BrokenProvider:+ def __init__(self, *, api_key: str | None = None) -> None:+ del api_key+ raise RuntimeError("boom")++ result = provider._create_custom_provider(BrokenProvider, config) # type: ignore[arg-type]+ assert isinstance(result, PydanticOpenAI)++[email protected]("pydantic_ai")+def test_custom_provider_applies_openrouter_cache_settings() -> None:+ """OpenRouter agents opt into prompt caching at the agent-settings level."""+ config = AnyProviderConfig(api_key="test-key", base_url="http://test-url")+ provider = get_completion_provider(config, "openrouter/openai/gpt-4")+ assert isinstance(provider, CustomProvider)++ model = provider.create_model()+ settings = cast(+ "dict[str, Any]",+ provider._build_model_settings(model, max_tokens=100),+ )++ assert settings["openrouter_cache_instructions"] is True+ assert settings["openrouter_cache_messages"] is True+ assert settings["openrouter_cache_tool_definitions"] == "1h"++[email protected]("pydantic_ai")+def test_custom_provider_non_openrouter_omits_cache_settings() -> None:+ """Non-OpenRouter custom providers don't get OpenRouter-specific settings."""+ config = AnyProviderConfig(+ api_key="test-key", base_url="https://my.internal.llm/v1"+ )+ provider = CustomProvider(+ AiModelId.from_model("my_provider/my-model"), config+ )++ model = provider.create_model()+ settings = provider._build_model_settings(model, max_tokens=None)++ assert "openrouter_cache_instructions" not in settings+ assert "max_tokens" not in settings++[email protected]("pydantic_ai")+def test_build_agent_capabilities_from_native_tool_support() -> None:+ """Provider-adaptive tools are enabled based on the model profile's+ supported native tools when no local search/fetch deps are installed."""+ from pydantic_ai.native_tools import (+ WebFetchTool,+ WebSearchTool,+ XSearchTool,+ )++ config = AnyProviderConfig(api_key="test-key", base_url="http://test-url")+ provider = OpenAIProvider("gpt-4", config)++ model = MagicMock(name="model")+ model.profile.supported_native_tools = {+ WebSearchTool,+ WebFetchTool,+ XSearchTool,+ }++ with (+ patch.object(+ DependencyManager.duckduckgo_search, "has", return_value=False+ ),+ patch.object(DependencyManager.markdownify, "has", return_value=False),+ ):+ capabilities = provider._build_agent_capabilities(model)++ assert sorted(type(c).__name__ for c in capabilities) == [+ "WebFetch",+ "WebSearch",+ "XSearch",+ ]++[email protected]("pydantic_ai")+def test_build_agent_capabilities_empty_without_support_or_deps() -> None:+ """No capabilities are added when the model supports no native tools and+ no local search/fetch deps are installed."""+ config = AnyProviderConfig(api_key="test-key", base_url="http://test-url")+ provider = OpenAIProvider("gpt-4", config)++ model = MagicMock(name="model")+ model.profile.supported_native_tools = set()++ with (+ patch.object(+ DependencyManager.duckduckgo_search, "has", return_value=False+ ),+ patch.object(DependencyManager.markdownify, "has", return_value=False),+ ):+ capabilities = provider._build_agent_capabilities(model)++ assert capabilities == []