A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.
Python artificial-intelligencedagdata-sciencedata-visualizationdataflowdeveloper-toolsmachine-learningnotebookspipelinepythonreactivesqlweb-app
Add streaming support for chat: async generator bug and add documentation (#7187)
Adam Draperda684ecbc021
diff --git a/docs/api/inputs/chat.md b/docs/api/inputs/chat.mdindex 40073bcf91..344d953c40 100644--- a/docs/api/inputs/chat.md+++ b/docs/api/inputs/chat.md@@ -132,6 +132,92 @@ mo.ui.chat()```+## Streaming Responses++Chatbots can stream responses in real-time, creating a more interactive experience+similar to ChatGPT where you see the response appear word-by-word as it's generated.++### With Built-in Models++For built-in models (OpenAI, Anthropic, Google, Groq, Bedrock), set `stream=True` in the model constructor:++```python+import marimo as mo++chat = mo.ui.chat(+ mo.ai.llm.openai(+ "gpt-4o",+ system_message="You are a helpful assistant.",+ stream=True, # Enable streaming+ ),+ show_configuration_controls=True+)+chat+```++This works for all built-in models:++- `mo.ai.llm.openai("gpt-4o", stream=True)`+- `mo.ai.llm.anthropic("claude-3-5-sonnet-20240620", stream=True)`+- `mo.ai.llm.google("gemini-1.5-pro-latest", stream=True)`+- `mo.ai.llm.groq("llama-3.1-70b-versatile", stream=True)`+- `mo.ai.llm.bedrock("anthropic.claude-3-7-sonnet-20250219-v1:0", stream=True)`++### With Custom Models++For custom models, you can use either regular (sync) or async generator functions that yield intermediate results:++**Sync generator (simpler):**++```python+import marimo as mo+import time++def streaming_model(messages, config):+ """Stream responses word by word."""+ response = "This response will appear word by word!"+ words = response.split()+ accumulated = ""++ for word in words:+ accumulated += word + " "+ yield accumulated+ time.sleep(0.1) # Simulate processing delay++chat = mo.ui.chat(streaming_model)+chat+```++**Async generator (for async operations):**++```python+import marimo as mo+import asyncio++async def async_streaming_model(messages, config):+ """Stream responses word by word asynchronously."""+ response = "This response will appear word by word!"+ words = response.split()+ accumulated = ""++ for word in words:+ accumulated += word + " "+ yield accumulated+ await asyncio.sleep(0.1) # Async processing delay++chat = mo.ui.chat(async_streaming_model)+chat+```++Each `yield` sends an update to the frontend, and the chat UI will display+the progressively accumulated response in real-time.++!!! tip "See streaming examples"+ For complete working examples, check out:++ - [`streaming_openai.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/streaming_openai.py) - Streaming with OpenAI models+ - [`streaming_custom.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/streaming_custom.py) - Custom streaming chatbot+## Built-in Modelsmarimo provides several built-in AI models that you can use with the chat UIdiff --git a/examples/ai/chat/README.md b/examples/ai/chat/README.mdindex abdf5d3000..4cdd0e244f 100644--- a/examples/ai/chat/README.md+++ b/examples/ai/chat/README.md@@ -3,7 +3,9 @@These examples show how to make chatbots with marimo, using [`mo.ui.chat`](https://docs.marimo.io/api/inputs/chat.html#marimo.ui.chat).- `custom.py` shows how to make a custom chatbot.+- `streaming_custom.py` shows how to make a custom chatbot with streaming responses.- `openai_example.py` shows how to make a chatbot powered by OpenAI models.+- `streaming_openai.py` shows how to make a chatbot powered by OpenAI models with streaming responses.- `anthropic_example.py` shows how to make a chatbot powered by Anthropic models.- `gemini.py` shows how to make a chatbot powered by Google models like Gemini.- `groq_example.py` shows how to make a chatbot powered by Groq models.@@ -14,6 +16,15 @@ These examples show how to make chatbots with marimo, using [`mo.ui.chat`](https- `simplemind_example.py` shows how to integrate [simplemind](https://github.com/kennethreitz/simplemind).- `generative_ui.py` shows how to make a chatbot that can generate UI code.+## Streaming Responses++Chatbots can stream responses in real-time, creating a more interactive experience similar to ChatGPT. To enable streaming:++- For **custom models**: create an async generator function that yields intermediate results+- For **built-in models** (OpenAI, Anthropic, etc.): set `stream=True` in the model constructor++See `streaming_custom.py` and `streaming_openai.py` for complete examples.+Chatbot's in marimo are _reactive_: when the chatbot responds with a message,all other cells referencing the chatbot are automatically run or markedstale, with the chatbot's response stored in the object's `value` attribute.diff --git a/examples/ai/chat/anthropic_example.py b/examples/ai/chat/anthropic_example.pyindex d738feb24d..460d066f99 100644--- a/examples/ai/chat/anthropic_example.py+++ b/examples/ai/chat/anthropic_example.py@@ -56,6 +56,7 @@ def _(key, mo):"claude-3-7-sonnet-20250219",system_message="You are a helpful assistant.",api_key=key,+ # stream=True, # Uncomment to enable streaming responses),allow_attachments=["image/png",diff --git a/examples/ai/chat/bedrock_example.py b/examples/ai/chat/bedrock_example.pyindex acc1a54870..c65575d452 100644--- a/examples/ai/chat/bedrock_example.py+++ b/examples/ai/chat/bedrock_example.py@@ -192,6 +192,7 @@ def _(config_form, max_tokens, mo, temperature):"model": model,"region_name": region,"system_message": system_message,+ # "stream": True, # Uncomment to enable streaming responses}# Add profile if specifieddiff --git a/examples/ai/chat/gemini.py b/examples/ai/chat/gemini.pyindex 989ee77915..bf758a1cbb 100644--- a/examples/ai/chat/gemini.py+++ b/examples/ai/chat/gemini.py@@ -2,12 +2,13 @@# requires-python = ">=3.11"# dependencies = [# "marimo",+# "google-genai>=1.20.0",# ]# ///import marimo-__generated_with = "0.17.4"+__generated_with = "0.17.8"app = marimo.App(width="medium")@@ -52,9 +53,10 @@ def _(input_key, mo, os_key):def _(key, mo):chatbot = mo.ui.chat(mo.ai.llm.google(- "gemini-1.5-pro-latest",+ "gemini-2.5-flash",system_message="You are a helpful assistant.",api_key=key,+ # stream=True, # Uncomment to enable streaming responses),prompts=["Hello",diff --git a/examples/ai/chat/groq_example.py b/examples/ai/chat/groq_example.pyindex b030f4fcc3..ae10e553bc 100644--- a/examples/ai/chat/groq_example.py+++ b/examples/ai/chat/groq_example.py@@ -73,6 +73,7 @@ def _(key, mo):model="llama-3.1-70b-versatile",system_message="You are a helpful assistant.",api_key=key,+ # stream=True, # Uncomment to enable streaming responses),prompts=["Hello",diff --git a/examples/ai/chat/openai_example.py b/examples/ai/chat/openai_example.pyindex 8556a126b0..0e75e45dba 100644--- a/examples/ai/chat/openai_example.py+++ b/examples/ai/chat/openai_example.py@@ -56,6 +56,7 @@ def _(mo, openai_key):"gpt-4o",system_message="You are a helpful assistant.",api_key=openai_key,+ # stream=True, # Uncomment to enable streaming responses),prompts=["Hello",diff --git a/examples/ai/chat/streaming_custom.py b/examples/ai/chat/streaming_custom.pynew file mode 100644index 0000000000..84218a67f1--- /dev/null+++ b/examples/ai/chat/streaming_custom.py@@ -0,0 +1,102 @@+# /// script+# requires-python = ">=3.11"+# dependencies = [+# "marimo",+# ]+# ///++import marimo++__generated_with = "0.17.8"+app = marimo.App(width="medium")+++def _():+ import marimo as mo+ import asyncio+ return asyncio, mo++[email protected](hide_code=True)+def _(mo):+ mo.md("""+ # Custom streaming chatbot++ This example shows how to make a chatbot that streams responses.+ Create an async generator function that yields intermediate results,+ and watch the response appear incrementally!+ """)+ return+++def _(asyncio, mo):+ async def streaming_echo_model(messages, config):+ """This chatbot echoes what the user says, word by word."""+ # Get the user's message+ user_message = messages[-1].content++ # Stream the response word by word+ response = f"You said: '{user_message}'. Here's my response streaming word by word!"+ words = response.split()+ accumulated = ""++ for word in words:+ accumulated += word + " "+ yield accumulated+ await asyncio.sleep(0.2) # Delay to make streaming visible++ chatbot = mo.ui.chat(+ streaming_echo_model,+ prompts=["Hello", "Tell me a story", "What is streaming?"],+ show_configuration_controls=True+ )+ return (chatbot,)+++def _(chatbot):+ chatbot+ return+++def _(mo):+ mo.md("""+ ## How it works++ The key is to make your model function an **async generator**:++ ```python+ async def my_model(messages, config):+ response = 'Building up text...'+ accumulated = ''+ for part in response.split():+ accumulated += part + ' '+ yield accumulated # Each yield updates the UI+ await asyncio.sleep(0.1)+ ```++ Each `yield` sends an update to the frontend, creating a smooth streaming effect!+ """)+ return+++def _(mo):+ mo.md("""+ Access the chatbot's historical messages with `chatbot.value`.+ """)+ return+++def _(chatbot):+ # chatbot.value is the list of chat messages+ chatbot.value+ return+++if __name__ == "__main__":+ app.run()diff --git a/examples/ai/chat/streaming_openai.py b/examples/ai/chat/streaming_openai.pynew file mode 100644index 0000000000..4c19223c68--- /dev/null+++ b/examples/ai/chat/streaming_openai.py@@ -0,0 +1,117 @@+# /// script+# requires-python = ">=3.11"+# dependencies = [+# "marimo",+# "openai>=1.55.3",+# ]+# ///++import marimo++__generated_with = "0.17.8"+app = marimo.App(width="medium")+++def _():+ import marimo as mo+ return (mo,)++[email protected](hide_code=True)+def _(mo):+ mo.md("""+ # OpenAI streaming chatbot++ This example shows how to use OpenAI's API with streaming responses.+ The built-in `mo.ai.llm.openai()` model automatically streams tokens+ as they arrive from the API!++ Enter your API key below to try it out.+ """)+ return+++def _(mo):+ api_key_input = mo.ui.text(+ placeholder="sk-...",+ label="OpenAI API Key",+ kind="password",+ )+ api_key_input+ return (api_key_input,)+++def _(api_key_input, mo):+ if api_key_input.value:+ chatbot = mo.ui.chat(+ mo.ai.llm.openai(+ "gpt-4o-mini",+ system_message="You are a helpful assistant. Keep responses concise and friendly.",+ api_key=api_key_input.value,+ stream=True, # Enable streaming+ ),+ prompts=[+ "Tell me a short joke",+ "What is Python?",+ "Explain streaming in one sentence",+ ],+ show_configuration_controls=True,+ )+ else:+ chatbot = mo.md("*Enter your OpenAI API key above to start chatting*")+ return (chatbot,)+++def _(chatbot):+ chatbot+ return+++def _(mo):+ mo.md("""+ ## How it works++ The built-in OpenAI model returns an async generator that yields tokens+ as they stream from the API:++ ```python+ mo.ui.chat(+ mo.ai.llm.openai(+ "gpt-4o-mini",+ api_key="your-key",+ stream=True, # Enable streaming!+ )+ )+ ```++ Set `stream=True` to enable streaming responses. 🚀++ Other built-in models (`anthropic`, `google`, `groq`) work the same way.+ """)+ return+++def _(chatbot, mo):+ # Show chat history+ if hasattr(chatbot, 'value'):+ mo.md(f"**Chat history:** {len(chatbot.value)} messages")+ return+++def _(chatbot):+ # Display full history+ if hasattr(chatbot, 'value'):+ chatbot.value+ return+++if __name__ == "__main__":+ app.run()+diff --git a/frontend/src/plugins/impl/chat/ChatPlugin.tsx b/frontend/src/plugins/impl/chat/ChatPlugin.tsxindex ae134eec0c..fe95eba8f7 100644--- a/frontend/src/plugins/impl/chat/ChatPlugin.tsx+++ b/frontend/src/plugins/impl/chat/ChatPlugin.tsx@@ -89,6 +89,7 @@ export const ChatPlugin = createPlugin<{ messages: ChatMessage[] }>(send_prompt={props.functions.send_prompt}value={props.value?.messages || Arrays.EMPTY}setValue={(messages) => props.setValue({ messages })}+ host={props.host}/></Suspense></TooltipProvider>diff --git a/frontend/src/plugins/impl/chat/chat-ui.tsx b/frontend/src/plugins/impl/chat/chat-ui.tsxindex c2683e0f15..99afa72eee 100644--- a/frontend/src/plugins/impl/chat/chat-ui.tsx+++ b/frontend/src/plugins/impl/chat/chat-ui.tsx@@ -12,6 +12,7 @@ import {DownloadIcon,HelpCircleIcon,PaperclipIcon,+ RotateCwIcon,SendIcon,SettingsIcon,Trash2Icon,@@ -42,7 +43,12 @@ import {import { Tooltip } from "@/components/ui/tooltip";import { toast } from "@/components/ui/use-toast";import { moveToEndOfEditor } from "@/core/codemirror/utils";+import { MarimoIncomingMessageEvent } from "@/core/dom/events";import { useAsyncData } from "@/hooks/useAsyncData";+import {+ type HTMLElementNotDerivedFromRef,+ useEventListener,+} from "@/hooks/useEventListener";import { cn } from "@/utils/cn";import { copyToClipboard } from "@/utils/copy";import { Logger } from "@/utils/Logger";@@ -63,6 +69,7 @@ interface Props extends PluginFunctions {allowAttachments: boolean | string[];value: ChatMessage[];setValue: (messages: ChatMessage[]) => void;+ host: HTMLElement;}export const Chatbot: React.FC<Props> = (props) => {@@ -74,6 +81,12 @@ export const Chatbot: React.FC<Props> = (props) => {const codeMirrorInputRef = useRef<ReactCodeMirrorRef>(null);const scrollContainerRef = useRef<HTMLDivElement>(null);+ // Track streaming state - maps backend message_id to frontend message index+ const streamingStateRef = useRef<{+ backendMessageId: string | null;+ frontendMessageIndex: number | null;+ }>({ backendMessageId: null, frontendMessageIndex: null });+const { data: initialMessages } = useAsyncData(async () => {const chatMessages = await props.get_chat_history({});const messages: UIMessage[] = chatMessages.messages.map((message, idx) => ({@@ -113,6 +126,19 @@ export const Chatbot: React.FC<Props> = (props) => {.join("\n"),parts: m.parts,}));++ // Create a placeholder message for streaming+ const messageId = Date.now().toString();++ setMessages((prev) => [+ ...prev,+ {+ id: messageId,+ role: "assistant",+ parts: [{ type: "text", text: "" }],+ },+ ]);+const response = await props.send_prompt({messages: messages,config: {@@ -124,18 +150,35 @@ export const Chatbot: React.FC<Props> = (props) => {presence_penalty: config.presence_penalty,},});- // Update local state with AI response- setMessages((prev) => [- ...prev,- {- id: Date.now().toString(),- role: "assistant",- parts: [{ type: "text", text: response }],- },- ]);++ // If streaming didn't happen (non-generator response), update the message+ // Check if streaming state is still set (meaning no chunks were received)+ if (+ streamingStateRef.current.backendMessageId === null &&+ streamingStateRef.current.frontendMessageIndex === null+ ) {+ setMessages((prev) => {+ const updated = [...prev];+ const index = updated.findIndex((m) => m.id === messageId);+ if (index !== -1) {+ updated[index] = {+ ...updated[index],+ parts: [{ type: "text", text: response }],+ };+ }+ return updated;+ });+ }+return new Response(response);// eslint-disable-next-line @typescript-eslint/no-explicit-any} catch (error: any) {+ // Clear streaming state on error+ streamingStateRef.current = {+ backendMessageId: null,+ frontendMessageIndex: null,+ };+// HACK: strip the error message to clean up the responseconst strippedError = error.message.split("failed with exception ")@@ -152,12 +195,98 @@ export const Chatbot: React.FC<Props> = (props) => {fileInputRef.current.value = "";}Logger.debug("Finished streaming message:", message);++ // Clear streaming state+ streamingStateRef.current = {+ backendMessageId: null,+ frontendMessageIndex: null,+ };},onError: (error) => {Logger.error("An error occurred:", error);+ // Clear streaming state on error+ streamingStateRef.current = {+ backendMessageId: null,+ frontendMessageIndex: null,+ };},});+ // Listen for streaming chunks from backend+ useEventListener(+ props.host as HTMLElementNotDerivedFromRef,+ MarimoIncomingMessageEvent.TYPE,+ (e) => {+ const message = e.detail.message;+ if (+ typeof message === "object" &&+ message !== null &&+ "type" in message &&+ message.type === "stream_chunk"+ ) {+ const chunkMessage = message as {+ type: string;+ message_id: string;+ content: string;+ is_final: boolean;+ };++ // Initialize streaming state on first chunk if not already set+ if (streamingStateRef.current.backendMessageId === null) {+ // Find the last assistant message (which should be the placeholder we created)+ setMessages((prev) => {+ const updated = [...prev];+ // Find the last assistant message+ for (let i = updated.length - 1; i >= 0; i--) {+ if (updated[i].role === "assistant") {+ streamingStateRef.current = {+ backendMessageId: chunkMessage.message_id,+ frontendMessageIndex: i,+ };+ break;+ }+ }+ return updated;+ });+ }++ // Only process chunks for the current streaming message+ const frontendIndex = streamingStateRef.current.frontendMessageIndex;+ if (+ streamingStateRef.current.backendMessageId ===+ chunkMessage.message_id &&+ frontendIndex !== null+ ) {+ setMessages((prev) => {+ const updated = [...prev];+ const index = frontendIndex;++ // Update the message at the tracked index+ if (index < updated.length) {+ const messageToUpdate = updated[index];+ if (messageToUpdate.role === "assistant") {+ updated[index] = {+ ...messageToUpdate,+ parts: [{ type: "text", text: chunkMessage.content }],+ };+ }+ }++ return updated;+ });++ // Clear streaming state when final chunk arrives+ if (chunkMessage.is_final) {+ streamingStateRef.current = {+ backendMessageId: null,+ frontendMessageIndex: null,+ };+ }+ }+ }+ },+ );+const isLoading = status === "submitted" || status === "streaming";const handleDelete = (id: string) => {@@ -283,13 +412,14 @@ export const Chatbot: React.FC<Props> = (props) => {<Buttonvariant="text"size="icon"+ disabled={messages.length === 0}onClick={() => {setMessages([]);props.setValue([]);props.delete_chat_history({});}}>- <Trash2Icon className="h-3 w-3" />+ <RotateCwIcon className="h-3 w-3" /></Button></div><divdiff --git a/marimo/_ai/llm/_impl.py b/marimo/_ai/llm/_impl.pyindex a1459a9378..d3e77ffae6 100644--- a/marimo/_ai/llm/_impl.py+++ b/marimo/_ai/llm/_impl.py@@ -2,7 +2,10 @@from __future__ import annotationsimport os-from typing import Any, Callable, Optional, cast+from typing import TYPE_CHECKING, Any, Callable, Optional, cast++if TYPE_CHECKING:+ from collections.abc import Generatorfrom marimo._ai._convert import (convert_to_anthropic_messages,@@ -55,6 +58,10 @@ class openai(ChatModel):If not provided, the API key will be retrievedfrom the OPENAI_API_KEY environment variable or the user's config.base_url: The base URL to use+ stream: Whether to stream the response token-by-token as it's generated.+ When True, the response appears progressively (like ChatGPT), creating+ a more responsive user experience. When False, the complete response+ is returned all at once. Defaults to False for backward compatibility."""def __init__(@@ -64,11 +71,13 @@ class openai(ChatModel):system_message: str = DEFAULT_SYSTEM_MESSAGE,api_key: Optional[str] = None,base_url: Optional[str] = None,+ stream: bool = False,):self.model = modelself.system_message = system_messageself.api_key = api_keyself.base_url = base_url+ self.stream = stream@propertydef _require_api_key(self) -> str:@@ -96,6 +105,21 @@ class openai(ChatModel):"set OPENAI_API_KEY as an environment variable")+ def _stream_response(self, response: Any) -> Generator[str, None, None]:+ """Helper method for streaming - separate to avoid mixing yield/return."""+ accumulated = ""+ chunk_count = 0+ for chunk in response:+ chunk_count += 1+ if chunk.choices and len(chunk.choices) > 0:+ delta = chunk.choices[0].delta+ if delta.content:+ accumulated += delta.content+ yield accumulated+ # Always yield final accumulated result to ensure complete response+ # This handles cases where the last chunk has no content+ yield accumulated+def __call__(self, messages: list[ChatMessage], config: ChatModelConfig) -> object:@@ -135,6 +159,7 @@ class openai(ChatModel):[ChatMessage(role="system", content=self.system_message)]+ messages)+response = client.chat.completions.create(model=self.model,messages=openai_messages,@@ -143,12 +168,17 @@ class openai(ChatModel):top_p=config.top_p,frequency_penalty=config.frequency_penalty,presence_penalty=config.presence_penalty,- stream=False,+ stream=self.stream,)- choice = response.choices[0]- content = choice.message.content- return content or ""+ if self.stream:+ # Return generator for streaming+ return self._stream_response(response)+ else:+ # Non-streaming response - return string directly+ choice = cast(Any, response).choices[0]+ content = choice.message.content+ return content or ""class anthropic(ChatModel):@@ -164,6 +194,10 @@ class anthropic(ChatModel):from the ANTHROPIC_API_KEY environment variableor the user's config.base_url: The base URL to use+ stream: Whether to stream the response token-by-token as it's generated.+ When True, the response appears progressively (like ChatGPT), creating+ a more responsive user experience. When False, the complete response+ is returned all at once. Defaults to False for backward compatibility."""def __init__(@@ -173,11 +207,13 @@ class anthropic(ChatModel):system_message: str = DEFAULT_SYSTEM_MESSAGE,api_key: Optional[str] = None,base_url: Optional[str] = None,+ stream: bool = False,):self.model = modelself.system_message = system_messageself.api_key = api_keyself.base_url = base_url+ self.stream = stream@propertydef _require_api_key(self) -> str:@@ -205,6 +241,19 @@ class anthropic(ChatModel):"set ANTHROPIC_API_KEY as an environment variable")+ def _stream_response(+ self, client: Any, params: Any+ ) -> Generator[str, None, None]:+ """Helper method for streaming - separate to avoid mixing yield/return."""+ accumulated = ""+ with client.messages.stream(**params) as stream:+ for text in stream.text_stream:+ accumulated += text+ yield accumulated+ # Yield final accumulated result to ensure complete response+ if accumulated:+ yield accumulated+def __call__(self, messages: list[ChatMessage], config: ChatModelConfig) -> object:@@ -224,7 +273,7 @@ class anthropic(ChatModel):"system": self.system_message,"max_tokens": config.max_tokens or 4096,"messages": anthropic_messages,- "stream": False,+ "stream": self.stream,}if config.top_p is not None:params["top_p"] = config.top_p@@ -233,15 +282,23 @@ class anthropic(ChatModel):if config.temperature is not None:params["temperature"] = config.temperature- response = client.messages.create(**params)-- content = response.content- if len(content) > 0:- if content[0].type == "text":- return content[0].text- elif content[0].type == "tool_use":- return content- return ""+ if self.stream:+ # Return generator for streaming+ return self._stream_response(client, params)+ else:+ # Non-streaming response - return directly+ response = client.messages.create(**params)++ content = response.content+ if len(content) > 0:+ if content[0].type == "text":+ return content[0].text+ elif content[0].type == "tool_use":+ return content+ else:+ return ""+ else:+ return ""class google(ChatModel):@@ -256,6 +313,10 @@ class google(ChatModel):If not provided, the API key will be retrievedfrom the GOOGLE_AI_API_KEY environment variableor the user's config.+ stream: Whether to stream the response token-by-token as it's generated.+ When True, the response appears progressively (like ChatGPT), creating+ a more responsive user experience. When False, the complete response+ is returned all at once. Defaults to False for backward compatibility."""def __init__(@@ -264,10 +325,12 @@ class google(ChatModel):*,system_message: str = DEFAULT_SYSTEM_MESSAGE,api_key: Optional[str] = None,+ stream: bool = False,):self.model = modelself.system_message = system_messageself.api_key = api_key+ self.stream = stream@propertydef _require_api_key(self) -> str:@@ -306,22 +369,51 @@ class google(ChatModel):client = genai.Client(api_key=self._require_api_key)google_messages = convert_to_google_messages(messages)- response = client.models.generate_content(++ # Build config once to avoid duplication+ generation_config = {+ "system_instruction": self.system_message,+ "max_output_tokens": config.max_tokens,+ "temperature": config.temperature,+ "top_p": config.top_p,+ "top_k": config.top_k,+ "frequency_penalty": config.frequency_penalty,+ "presence_penalty": config.presence_penalty,+ }++ if self.stream:+ # Return generator for streaming+ return self._stream_response(+ client, google_messages, generation_config+ )+ else:+ # Non-streaming response - return directly+ response = client.models.generate_content(+ model=self.model,+ contents=google_messages,+ config=cast(Any, generation_config),+ )++ content = response.text+ return content or ""++ def _stream_response(+ self, client: Any, google_messages: Any, generation_config: Any+ ) -> Generator[str, None, None]:+ """Helper method for streaming - separate to avoid mixing yield/return."""+ accumulated = ""+ response = client.models.generate_content_stream(model=self.model,contents=google_messages,- config={- "system_instruction": self.system_message,- "max_output_tokens": config.max_tokens,- "temperature": config.temperature,- "top_p": config.top_p,- "top_k": config.top_k,- "frequency_penalty": config.frequency_penalty,- "presence_penalty": config.presence_penalty,- },+ config=generation_config,)-- content = response.text- return content or ""+ for chunk in response:+ if chunk.text:+ accumulated += chunk.text+ yield accumulated+ # Yield final accumulated result to ensure complete response+ if accumulated:+ yield accumulatedclass groq(ChatModel):@@ -336,6 +428,10 @@ class groq(ChatModel):If not provided, the API key will be retrievedfrom the GROQ_API_KEY environment variable or the user's config.base_url: The base URL to use+ stream: Whether to stream the response token-by-token as it's generated.+ When True, the response appears progressively (like ChatGPT), creating+ a more responsive user experience. When False, the complete response+ is returned all at once. Defaults to False for backward compatibility."""def __init__(@@ -345,11 +441,13 @@ class groq(ChatModel):system_message: str = DEFAULT_SYSTEM_MESSAGE,api_key: Optional[str] = None,base_url: Optional[str] = None,+ stream: bool = False,):self.model = modelself.system_message = system_messageself.api_key = api_keyself.base_url = base_url+ self.stream = stream@propertydef _require_api_key(self) -> str:@@ -378,6 +476,31 @@ class groq(ChatModel):"set GROQ_API_KEY as an environment variable")+ def _stream_response(+ self, client: Any, groq_messages: Any, config: ChatModelConfig+ ) -> Generator[str, None, None]:+ """Helper method for streaming - separate to avoid mixing yield/return."""+ stream = client.chat.completions.create(+ model=self.model,+ messages=groq_messages,+ max_tokens=config.max_tokens,+ temperature=config.temperature,+ top_p=config.top_p,+ stop=None,+ stream=True,+ )++ accumulated = ""+ for chunk in stream:+ if chunk.choices and len(chunk.choices) > 0:+ delta = chunk.choices[0].delta+ if delta.content:+ accumulated += delta.content+ yield accumulated+ # Yield final accumulated result to ensure complete response+ if accumulated:+ yield accumulated+def __call__(self, messages: list[ChatMessage], config: ChatModelConfig) -> object:@@ -392,19 +515,25 @@ class groq(ChatModel):[ChatMessage(role="system", content=self.system_message)]+ messages)- response = client.chat.completions.create(- model=self.model,- messages=groq_messages,- max_tokens=config.max_tokens,- temperature=config.temperature,- top_p=config.top_p,- stop=None,- stream=False,- )- choice = response.choices[0]- content = choice.message.content- return content or ""+ if self.stream:+ # Return generator for streaming+ return self._stream_response(client, groq_messages, config)+ else:+ # Non-streaming response - return directly+ response = client.chat.completions.create(+ model=self.model,+ messages=groq_messages,+ max_tokens=config.max_tokens,+ temperature=config.temperature,+ top_p=config.top_p,+ stop=None,+ stream=False,+ )++ choice = response.choices[0]+ content = choice.message.content+ return content or ""class bedrock(ChatModel):@@ -421,6 +550,10 @@ class bedrock(ChatModel):Dict with keys: "aws_access_key_id" and "aws_secret_access_key"If not provided, credentials will be retrieved from the environmentor the AWS configuration files.+ stream: Whether to stream the response token-by-token as it's generated.+ When True, the response appears progressively (like ChatGPT), creating+ a more responsive user experience. When False, the complete response+ is returned all at once. Defaults to False for backward compatibility."""def __init__(@@ -432,6 +565,7 @@ class bedrock(ChatModel):profile_name: Optional[str] = None,aws_access_key_id: Optional[str] = None,aws_secret_access_key: Optional[str] = None,+ stream: bool = False,):if not model.startswith("bedrock/"):model = f"bedrock/{model}"@@ -441,6 +575,7 @@ class bedrock(ChatModel):self.profile_name = profile_nameself.aws_access_key_id = aws_access_key_idself.aws_secret_access_key = aws_secret_access_key+ self.stream = streamdef _setup_credentials(self) -> None:# Use profile name if provided, otherwise use API key@@ -452,6 +587,37 @@ class bedrock(ChatModel):else:pass # Use default credential chain+ def _stream_response(+ self, messages: list[ChatMessage], config: ChatModelConfig+ ) -> Generator[str, None, None]:+ """Helper method for streaming - separate to avoid mixing yield/return."""+ from litellm import completion as litellm_completion++ response = litellm_completion(+ model=self.model,+ messages=convert_to_openai_messages(+ [ChatMessage(role="system", content=self.system_message)]+ + messages+ ),+ max_tokens=config.max_tokens,+ temperature=config.temperature,+ top_p=config.top_p,+ frequency_penalty=config.frequency_penalty,+ presence_penalty=config.presence_penalty,+ stream=True,+ )++ accumulated = ""+ for chunk in response:+ if chunk.choices and len(chunk.choices) > 0:+ delta = chunk.choices[0].delta+ if delta.content:+ accumulated += delta.content+ yield accumulated+ # Yield final accumulated result to ensure complete response+ if accumulated:+ yield accumulated+def __call__(self, messages: list[ChatMessage], config: ChatModelConfig) -> object:@@ -466,22 +632,29 @@ class bedrock(ChatModel):self._setup_credentials()try:- # Make API call- response = litellm_completion(- model=self.model,- messages=convert_to_openai_messages(- [ChatMessage(role="system", content=self.system_message)]- + messages- ),- max_tokens=config.max_tokens,- temperature=config.temperature,- top_p=config.top_p,- frequency_penalty=config.frequency_penalty,- presence_penalty=config.presence_penalty,- stream=False,- )-- return response.choices[0].message.content+ if self.stream:+ # Return generator for streaming+ return self._stream_response(messages, config)+ else:+ # Non-streaming response - return directly+ response = litellm_completion(+ model=self.model,+ messages=convert_to_openai_messages(+ [+ ChatMessage(+ role="system", content=self.system_message+ )+ ]+ + messages+ ),+ max_tokens=config.max_tokens,+ temperature=config.temperature,+ top_p=config.top_p,+ frequency_penalty=config.frequency_penalty,+ presence_penalty=config.presence_penalty,+ stream=False,+ )+ return response.choices[0].message.contentexcept Exception as e:# Handle common AWS exceptions with helpful messagesdiff --git a/marimo/_plugins/ui/_impl/chat/chat.py b/marimo/_plugins/ui/_impl/chat/chat.pyindex 140bfcf83c..3646e65b42 100644--- a/marimo/_plugins/ui/_impl/chat/chat.py+++ b/marimo/_plugins/ui/_impl/chat/chat.py@@ -2,6 +2,7 @@from __future__ import annotationsimport inspect+import uuidfrom dataclasses import dataclassfrom typing import Any, Callable, Final, Optional, Union, cast@@ -20,7 +21,7 @@ from marimo._runtime.functions import EmptyArgs, Functionfrom marimo._runtime.requests import SetUIElementValueRequestDEFAULT_CONFIG = ChatModelConfigDict(- max_tokens=100,+ max_tokens=4096,temperature=0.5,top_p=1,top_k=40,@@ -77,16 +78,23 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):return await my_async_function(messages)```+ Regular (sync) generators for streaming:```python- async def my_rag_model(messages):- for response in my_async_iterator(messages):- yield response+ def my_streaming_model(messages, config):+ for chunk in process_stream():+ yield chunk # Each yield updates the UI+ ```++ Async generators for streaming with async operations:+ ```python+ async def my_async_streaming_model(messages, config):+ async for chunk in async_process_stream():+ yield chunk # Each yield updates the UI```- The last value yielded by the async generator is treated as the model- response. ui.chat does not yet support streaming responses to the frontend.- Please file a GitHub issue if this is important to you:- https://github.com/marimo-team/marimo/issues+ The last value yielded by the generator is treated as the model+ response. Streaming responses are automatically streamed to the frontend+ as they are generated.Using a built-in model:```python@@ -206,6 +214,56 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):del self._chat_history[index]self._value = self._chat_history+ async def _handle_streaming_response(self, response: Any) -> str:+ """Handle streaming from both sync and async generators."""+ message_id = str(uuid.uuid4())+ latest_response = None+ accumulated_text = ""++ # Use async for if it's an async generator, otherwise regular for+ if inspect.isasyncgen(response):+ async for latest_response in response: # noqa: B007+ accumulated_text = str(latest_response)+ self._send_message(+ {+ "type": "stream_chunk",+ "message_id": message_id,+ "content": accumulated_text,+ "is_final": False,+ },+ buffers=None,+ )+ else:+ for latest_response in response: # noqa: B007+ accumulated_text = str(latest_response)+ self._send_message(+ {+ "type": "stream_chunk",+ "message_id": message_id,+ "content": accumulated_text,+ "is_final": False,+ },+ buffers=None,+ )++ # Send final message to indicate streaming is complete+ if latest_response is not None:+ self._send_message(+ {+ "type": "stream_chunk",+ "message_id": message_id,+ "content": accumulated_text,+ "is_final": True,+ },+ buffers=None,+ )++ return (+ str(latest_response)+ if latest_response is not None+ else accumulated_text+ )+async def _send_prompt(self, args: SendMessageRequest) -> str:messages = args.messages@@ -223,20 +281,11 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):if inspect.isawaitable(response):response = await response- elif inspect.isasyncgen(response):- # We support functions that stream the response with an async- # generator; each yielded value is the latest representation of the- # response, and the last value is the full value- latest_response = None- async for latest_response in response: # noqa: B007- # TODO(akshayka, mscolnick): Stream response to frontend- # once bidirectional communication is implemented.- #- # RPCs don't yet support bidirectional communication, so we- # just ignore all the initial responses; ideally we'd stream- # the response back to the frontend.- pass- response = latest_response+ elif inspect.isasyncgen(response) or inspect.isgenerator(response):+ # We support functions that stream the response with generators+ # (both sync and async); each yielded value is the latest+ # representation of the response, and the last value is the full value+ response = await self._handle_streaming_response(response)response_message = ChatMessage(role="assistant", content=response)self._chat_history = messages + [response_message]diff --git a/pixi.lock b/pixi.lockindex 70d9afb0de..48f4c828c2 100644--- a/pixi.lock+++ b/pixi.lock@@ -106,25 +106,17 @@ environments:- conda: https://conda.anaconda.org/conda-forge/noarch/zipp-3.22.0-pyhd8ed1ab_0.conda- conda: https://conda.anaconda.org/conda-forge/linux-64/zlib-1.3.1-hb9d3cd8_2.conda- conda: https://conda.anaconda.org/conda-forge/linux-64/zstandard-0.23.0-py312h66e93f0_2.conda- - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/8f/d7/9322c609343d929e75e7e5e6255e614fcc67572cfd083959cdef3b7aad79/docutils-0.21.2-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/c0/5a/9cac0c82afec3d09ccd97c8b6502d48f165f9124db81b4bcb90b4af974ee/jedi-0.19.2-py2.py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/7b/43/59fdc2f6267959b71dd23ce0bd8d4aeaf55566aa435a5d00f53d53c7eb24/jiter-0.11.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl- pypi: https://files.pythonhosted.org/packages/a3/48/f97f0920c20bc522c34f1eaec4c62d7a03c95c37da2dec858f65c8cb4325/loro-1.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl- pypi: https://files.pythonhosted.org/packages/51/3f/afe76f8e2246ffbc867440cbcf90525264df0e658f8a5ca1f872b3f6192a/markdown-3.8-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/87/51/0fa83662b036bdac504192e8067798b6d0ef912eec2af897361e60357808/msgspec_m-0.19.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl- pypi: https://files.pythonhosted.org/packages/50/3b/0e2c535c3e6970cfc5763b67f6cc31accaab35a7aa3e322fb6a12830450f/narwhals-2.6.0-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/8c/74/6bfc3adc81f6c2cea4439f2a734c40e3a420703bbcdc539890096a732bbd/openai-2.7.1-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/c6/ac/dac4a63f978e4dcb3c6d3a78c4d8e0192a113d288502a1216950c41b1027/parso-0.8.4-py2.py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/bf/b9/b0eb3f3cbcb734d930fdf839431606844a825b23eaf9a6ab371edac8162c/psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl- - pypi: https://files.pythonhosted.org/packages/bd/1f/73c53fcbfb0b5a78f91176df41945ca466e71e9d9d836e5c522abda39ee7/pydantic-2.11.10-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/f9/41/4b043778cf9c4285d59742281a769eac371b9e47e35f98ad321349cc5d61/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl- pypi: https://files.pythonhosted.org/packages/a7/d1/c54e608505776ce4e7966d03358ae635cfd51dff1da6ee421c090dbc797b/pymdown_extensions-10.15-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/8b/0c/9d30a4ebeb6db2b25a841afbb80f6ef9a854fc3b41be131d249a977b4959/starlette-0.46.2-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/b1/4b/4cef6ce21a2aaca9d852a6e84ef4f135d99fcd74fa75105e2fc0c8308acd/uvicorn-0.34.2-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/14/8f/aa61f528fba38578ec553c145857a181384c72b98156f858ca5c8e82d9d3/websockets-15.0.1-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl- pypi: ./@@ -211,25 +203,17 @@ environments:- conda: https://conda.anaconda.org/conda-forge/noarch/zipp-3.22.0-pyhd8ed1ab_0.conda- conda: https://conda.anaconda.org/conda-forge/osx-arm64/zlib-1.3.1-h8359307_2.conda- conda: https://conda.anaconda.org/conda-forge/osx-arm64/zstandard-0.23.0-py312hea69d52_2.conda- - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/8f/d7/9322c609343d929e75e7e5e6255e614fcc67572cfd083959cdef3b7aad79/docutils-0.21.2-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/c0/5a/9cac0c82afec3d09ccd97c8b6502d48f165f9124db81b4bcb90b4af974ee/jedi-0.19.2-py2.py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/f7/29/6c7de6b5d6e511d9e736312c0c9bfcee8f9b6bef68182a08b1d78767e627/jiter-0.11.1-cp312-cp312-macosx_11_0_arm64.whl- pypi: https://files.pythonhosted.org/packages/32/ee/7e042a13e93f09c24618dc804f01849735d81740f8b74827124ff7fc8396/loro-1.5.0-cp312-cp312-macosx_11_0_arm64.whl- pypi: https://files.pythonhosted.org/packages/3f/08/83871f3c50fc983b88547c196d11cf8c3340e37c32d2e9d6152abe2c61f7/Markdown-3.7-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/4c/b7/a93d524907162cb6150179eb47fac885bbbad025c82151b69ad1d62cda4d/msgspec_m-0.19.2-cp312-cp312-macosx_11_0_arm64.whl- pypi: https://files.pythonhosted.org/packages/7d/81/1cf7a468ef2f8e88266d5c3e5ab026a045aa76150e6640bfe9c5450a8c11/narwhals-2.3.0-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/8c/74/6bfc3adc81f6c2cea4439f2a734c40e3a420703bbcdc539890096a732bbd/openai-2.7.1-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/c6/ac/dac4a63f978e4dcb3c6d3a78c4d8e0192a113d288502a1216950c41b1027/parso-0.8.4-py2.py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/04/8b/30f930733afe425e3cbfc0e1468a30a18942350c1a8816acfade80c005c4/psutil-7.0.0-cp36-abi3-macosx_11_0_arm64.whl- - pypi: https://files.pythonhosted.org/packages/bd/1f/73c53fcbfb0b5a78f91176df41945ca466e71e9d9d836e5c522abda39ee7/pydantic-2.11.10-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/a1/02/6224312aacb3c8ecbaa959897af57181fb6cf3a3d7917fd44d0f2917e6f2/pydantic_core-2.33.2-cp312-cp312-macosx_11_0_arm64.whl- pypi: https://files.pythonhosted.org/packages/e4/06/43084e6cbd4b3bc0e80f6be743b2e79fbc6eed8de9ad8c629939fa55d972/pymdown_extensions-10.16.1-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/a0/4b/528ccf7a982216885a1ff4908e886b8fb5f19862d1962f56a3fce2435a70/starlette-0.46.1-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl- - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/61/14/33a3a1352cfa71812a3a21e8c9bfb83f60b0011f5e36f2b1399d51928209/uvicorn-0.34.0-py3-none-any.whl- pypi: https://files.pythonhosted.org/packages/3d/69/1a681dd6f02180916f116894181eab8b2e25b31e484c5d0eae637ec01f7c/websockets-15.0.1-cp312-cp312-macosx_11_0_arm64.whl- pypi: ./@@ -255,13 +239,6 @@ packages:purls: []size: 23621timestamp: 1650670423406-- pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl- name: annotated-types- version: 0.7.0- sha256: 1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53- requires_dist:- - typing-extensions>=4.0.0 ; python_full_version < '3.9'- requires_python: '>=3.8'- conda: https://conda.anaconda.org/conda-forge/noarch/anyio-4.9.0-pyh29332c3_0.condasha256: b28e0f78bb0c7962630001e63af25a89224ff504e135a02e50d4d80b6155d386md5: 9749a2c77a7c40d432ea0927662d7e52@@ -450,11 +427,6 @@ packages:- pkg:pypi/distlib?source=hash-mappingsize: 274151timestamp: 1733238487461-- pypi: https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl- name: distro- version: 1.9.0- sha256: 7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2- requires_python: '>=3.6'- pypi: https://files.pythonhosted.org/packages/8f/d7/9322c609343d929e75e7e5e6255e614fcc67572cfd083959cdef3b7aad79/docutils-0.21.2-py3-none-any.whlname: docutilsversion: 0.21.2@@ -793,16 +765,6 @@ packages:- pkg:pypi/jeepney?source=hash-mappingsize: 40015timestamp: 1740828380668-- pypi: https://files.pythonhosted.org/packages/7b/43/59fdc2f6267959b71dd23ce0bd8d4aeaf55566aa435a5d00f53d53c7eb24/jiter-0.11.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl- name: jiter- version: 0.11.1- sha256: eb68e7bf65c990531ad8715e57d50195daf7c8e6f1509e617b4e692af1108939- requires_python: '>=3.9'-- pypi: https://files.pythonhosted.org/packages/f7/29/6c7de6b5d6e511d9e736312c0c9bfcee8f9b6bef68182a08b1d78767e627/jiter-0.11.1-cp312-cp312-macosx_11_0_arm64.whl- name: jiter- version: 0.11.1- sha256: ecd4da91b5415f183a6be8f7158d127bdd9e6a3174138293c0d48d6ea2f2009d- requires_python: '>=3.9'- conda: https://conda.anaconda.org/conda-forge/noarch/keyring-25.6.0-pyh534df25_0.condasha256: c8b436fa9853bf8b836c96afbb7684a04955b80b37f5d5285fd836b6a8566cc5md5: d2c0c5bda93c249f877c7fceea9e63af@@ -1139,8 +1101,8 @@ packages:timestamp: 1727801725384- pypi: ./name: marimo- version: 0.17.7- sha256: cad6192c0a7ae8eff42fc50f18bd870864e322892608c70ca274c95b3308b632+ version: 0.17.8+ sha256: 45bc4e2f2d1b8607176ff73aa7bc6a442a4bd44498b7d9b2b17052470215b7bfrequires_dist:- click>=8.0,<9- jedi>=0.18.0@@ -1160,7 +1122,6 @@ packages:- narwhals>=2.0.0- packaging- msgspec-m>=0.19.2- - openai>=2.7.1- duckdb>=1.0.0 ; extra == 'sql'- polars[pyarrow]>=1.9.0 ; extra == 'sql'- sqlglot[rs]>=26.2.0 ; extra == 'sql'@@ -1430,28 +1391,6 @@ packages:purls: []size: 15490642timestamp: 1737401388520-- pypi: https://files.pythonhosted.org/packages/8c/74/6bfc3adc81f6c2cea4439f2a734c40e3a420703bbcdc539890096a732bbd/openai-2.7.1-py3-none-any.whl- name: openai- version: 2.7.1- sha256: 2f2530354d94c59c614645a4662b9dab0a5b881c5cd767a8587398feac0c9021- requires_dist:- - anyio>=3.5.0,<5- - distro>=1.7.0,<2- - httpx>=0.23.0,<1- - jiter>=0.10.0,<1- - pydantic>=1.9.0,<3- - sniffio- - tqdm>4- - typing-extensions>=4.11,<5- - aiohttp ; extra == 'aiohttp'- - httpx-aiohttp>=0.1.9 ; extra == 'aiohttp'- - numpy>=1 ; extra == 'datalib'- - pandas-stubs>=1.1.0.11 ; extra == 'datalib'- - pandas>=1.2.3 ; extra == 'datalib'- - websockets>=13,<16 ; extra == 'realtime'- - numpy>=2.0.2 ; extra == 'voice-helpers'- - sounddevice>=0.5.1 ; extra == 'voice-helpers'- requires_python: '>=3.8'- conda: https://conda.anaconda.org/conda-forge/linux-64/openssl-3.5.0-h7b32b05_1.condasha256: b4491077c494dbf0b5eaa6d87738c22f2154e9277e5293175ec187634bd808a0md5: de356753cfdbffcde5bb1e86e3aa6cd0@@ -1704,32 +1643,6 @@ packages:- pkg:pypi/pycparser?source=hash-mappingsize: 110100timestamp: 1733195786147-- pypi: https://files.pythonhosted.org/packages/bd/1f/73c53fcbfb0b5a78f91176df41945ca466e71e9d9d836e5c522abda39ee7/pydantic-2.11.10-py3-none-any.whl- name: pydantic- version: 2.11.10- sha256: 802a655709d49bd004c31e865ef37da30b540786a46bfce02333e0e24b5fe29a- requires_dist:- - annotated-types>=0.6.0- - pydantic-core==2.33.2- - typing-extensions>=4.12.2- - typing-inspection>=0.4.0- - email-validator>=2.0.0 ; extra == 'email'- - tzdata ; python_full_version >= '3.9' and sys_platform == 'win32' and extra == 'timezone'- requires_python: '>=3.9'-- pypi: https://files.pythonhosted.org/packages/a1/02/6224312aacb3c8ecbaa959897af57181fb6cf3a3d7917fd44d0f2917e6f2/pydantic_core-2.33.2-cp312-cp312-macosx_11_0_arm64.whl- name: pydantic-core- version: 2.33.2- sha256: 3c6db6e52c6d70aa0d00d45cdb9b40f0433b96380071ea80b09277dba021ddf7- requires_dist:- - typing-extensions>=4.6.0,!=4.7.0- requires_python: '>=3.9'-- pypi: https://files.pythonhosted.org/packages/f9/41/4b043778cf9c4285d59742281a769eac371b9e47e35f98ad321349cc5d61/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl- name: pydantic-core- version: 2.33.2- sha256: 8f57a69461af2a5fa6e6bbd7a5f60d3b7e6cebb687f55106933188e79ad155c1- requires_dist:- - typing-extensions>=4.6.0,!=4.7.0- requires_python: '>=3.9'- conda: https://conda.anaconda.org/conda-forge/noarch/pygments-2.19.1-pyhd8ed1ab_0.condasha256: 28a3e3161390a9d23bc02b4419448f8d27679d9e2c250e29849e37749c8de86bmd5: 232fb4577b6687b2d503ef8e254270c9@@ -2014,22 +1927,6 @@ packages:- pkg:pypi/tomlkit?source=hash-mappingsize: 37372timestamp: 1733230836889-- pypi: https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl- name: tqdm- version: 4.67.1- sha256: 26445eca388f82e72884e0d580d5464cd801a3ea01e63e5601bdff9ba6a48de2- requires_dist:- - colorama ; sys_platform == 'win32'- - pytest>=6 ; extra == 'dev'- - pytest-cov ; extra == 'dev'- - pytest-timeout ; extra == 'dev'- - pytest-asyncio>=0.24 ; extra == 'dev'- - nbval ; extra == 'dev'- - requests ; extra == 'discord'- - slack-sdk ; extra == 'slack'- - requests ; extra == 'telegram'- - ipywidgets>=6 ; extra == 'notebook'- requires_python: '>=3.7'- conda: https://conda.anaconda.org/conda-forge/noarch/trove-classifiers-2025.5.9.12-pyhd8ed1ab_0.condasha256: 455b7b0dc0cf7e4a6fcc41455b4fd7f646b3b842e6dc0d894438366827d7d9b2md5: 764db08a8d868de9e377d88277c75d83@@ -2041,13 +1938,6 @@ packages:- pkg:pypi/trove-classifiers?source=hash-mappingsize: 19516timestamp: 1746817031708-- pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl- name: typing-inspection- version: 0.4.2- sha256: 4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7- requires_dist:- - typing-extensions>=4.12.0- requires_python: '>=3.9'- conda: https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.13.2-pyh29332c3_0.condasha256: a8aaf351e6461de0d5d47e4911257e25eec2fa409d71f3b643bb2f748bde1c08md5: 83fc6ae00127671e301c9f44254c31b8diff --git a/tests/_plugins/ui/_impl/chat/test_chat.py b/tests/_plugins/ui/_impl/chat/test_chat.pyindex c7b729427b..0af644fae2 100644--- a/tests/_plugins/ui/_impl/chat/test_chat.py+++ b/tests/_plugins/ui/_impl/chat/test_chat.py@@ -139,6 +139,134 @@ async def test_chat_send_prompt_async_generator():assert chat._chat_history[1].content == "2"+async def test_chat_streaming_sends_messages():+ """Test that streaming async generators send messages via _send_message"""+ sent_messages = []++ async def mock_streaming_model(+ messages: list[ChatMessage], config: ChatModelConfig+ ) -> AsyncIterator[str]:+ del config, messages+ # Simulate streaming response+ accumulated = ""+ for word in ["Hello", "world", "!"]:+ accumulated += word + " "+ yield accumulated.strip()++ chat = ui.chat(mock_streaming_model)++ # Mock _send_message to capture calls+ original_send_message = chat._send_message++ def capture_send_message(message: dict[str, object], buffers): # noqa: ARG001+ sent_messages.append(message)+ # Don't actually send to avoid needing kernel context++ chat._send_message = capture_send_message++ request = SendMessageRequest(+ messages=[ChatMessage(role="user", content="Test")],+ config=ChatModelConfig(),+ )++ response: str = await chat._send_prompt(request)++ # Verify final response+ assert response == "Hello world !"++ # Verify streaming messages were sent+ # Should have sent chunks for each yield, plus final message+ assert len(sent_messages) >= 3++ # Check that messages have streaming structure+ for msg in sent_messages[:-1]: # All but last+ assert msg["type"] == "stream_chunk"+ assert "message_id" in msg+ assert "content" in msg+ assert not msg["is_final"]++ # Last message should be final+ assert sent_messages[-1]["type"] == "stream_chunk"+ assert sent_messages[-1]["is_final"]+ assert sent_messages[-1]["content"] == "Hello world !"+++async def test_chat_sync_generator_streaming():+ """Test that sync generators also work for streaming"""+ sent_messages = []++ def mock_streaming_model(+ messages: list[ChatMessage], config: ChatModelConfig+ ):+ del config, messages+ # Simulate streaming response with sync generator+ accumulated = ""+ for word in ["Hello", "world", "!"]:+ accumulated += word + " "+ yield accumulated.strip()++ chat = ui.chat(mock_streaming_model)++ def capture_send_message(message: dict[str, object], buffers): # noqa: ARG001+ sent_messages.append(message)++ chat._send_message = capture_send_message++ request = SendMessageRequest(+ messages=[ChatMessage(role="user", content="Test")],+ config=ChatModelConfig(),+ )++ response: str = await chat._send_prompt(request)++ # Verify final response+ assert response == "Hello world !"++ # Verify streaming messages were sent+ assert len(sent_messages) >= 3++ # Check that messages have streaming structure+ for msg in sent_messages[:-1]: # All but last+ assert msg["type"] == "stream_chunk"+ assert "message_id" in msg+ assert "content" in msg+ assert not msg["is_final"]++ # Last message should be final+ assert sent_messages[-1]["type"] == "stream_chunk"+ assert sent_messages[-1]["is_final"]+ assert sent_messages[-1]["content"] == "Hello world !"+++async def test_chat_streaming_complete_response():+ """Test that streaming returns complete response even with empty final chunks"""++ def mock_streaming_model_with_empty_final(+ messages: list[ChatMessage], config: ChatModelConfig+ ):+ del config, messages+ # Simulate OpenAI-style streaming where last chunk might be empty+ # This tests the bug where we might not yield the final accumulated value+ yield "Hello "+ yield "Hello world"+ yield "Hello world!"+ # Simulate final chunk with no content (like finish_reason only)+ # Generator continues but no more content - still need complete result++ chat = ui.chat(mock_streaming_model_with_empty_final)++ request = SendMessageRequest(+ messages=[ChatMessage(role="user", content="Test")],+ config=ChatModelConfig(),+ )++ response: str = await chat._send_prompt(request)++ # Verify we got the complete final response+ assert response == "Hello world!"+ assert chat._chat_history[-1].content == "Hello world!"++def test_chat_get_history():def mock_model(messages: list[ChatMessage], config: ChatModelConfig