Skip to content

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.

Add streaming support for chat: async generator bug and add documentation (#7187)

Adam Draperda684ecbc021
diff --git a/docs/api/inputs/chat.md b/docs/api/inputs/chat.md
index 40073bcf91..344d953c40 100644
--- a/docs/api/inputs/chat.md
+++ b/docs/api/inputs/chat.md
@@ -132,6 +132,92 @@ mo.ui.chat(
)
```
+## Streaming Responses
+
+Chatbots can stream responses in real-time, creating a more interactive experience
+similar to ChatGPT where you see the response appear word-by-word as it's generated.
+
+### With Built-in Models
+
+For built-in models (OpenAI, Anthropic, Google, Groq, Bedrock), set `stream=True` in the model constructor:
+
+```python
+import marimo as mo
+
+chat = mo.ui.chat(
+ mo.ai.llm.openai(
+ "gpt-4o",
+ system_message="You are a helpful assistant.",
+ stream=True, # Enable streaming
+ ),
+ show_configuration_controls=True
+)
+chat
+```
+
+This works for all built-in models:
+
+- `mo.ai.llm.openai("gpt-4o", stream=True)`
+- `mo.ai.llm.anthropic("claude-3-5-sonnet-20240620", stream=True)`
+- `mo.ai.llm.google("gemini-1.5-pro-latest", stream=True)`
+- `mo.ai.llm.groq("llama-3.1-70b-versatile", stream=True)`
+- `mo.ai.llm.bedrock("anthropic.claude-3-7-sonnet-20250219-v1:0", stream=True)`
+
+### With Custom Models
+
+For custom models, you can use either regular (sync) or async generator functions that yield intermediate results:
+
+**Sync generator (simpler):**
+
+```python
+import marimo as mo
+import time
+
+def streaming_model(messages, config):
+ """Stream responses word by word."""
+ response = "This response will appear word by word!"
+ words = response.split()
+ accumulated = ""
+
+ for word in words:
+ accumulated += word + " "
+ yield accumulated
+ time.sleep(0.1) # Simulate processing delay
+
+chat = mo.ui.chat(streaming_model)
+chat
+```
+
+**Async generator (for async operations):**
+
+```python
+import marimo as mo
+import asyncio
+
+async def async_streaming_model(messages, config):
+ """Stream responses word by word asynchronously."""
+ response = "This response will appear word by word!"
+ words = response.split()
+ accumulated = ""
+
+ for word in words:
+ accumulated += word + " "
+ yield accumulated
+ await asyncio.sleep(0.1) # Async processing delay
+
+chat = mo.ui.chat(async_streaming_model)
+chat
+```
+
+Each `yield` sends an update to the frontend, and the chat UI will display
+the progressively accumulated response in real-time.
+
+!!! tip "See streaming examples"
+ For complete working examples, check out:
+
+ - [`streaming_openai.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/streaming_openai.py) - Streaming with OpenAI models
+ - [`streaming_custom.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/streaming_custom.py) - Custom streaming chatbot
+
## Built-in Models
marimo provides several built-in AI models that you can use with the chat UI
diff --git a/examples/ai/chat/README.md b/examples/ai/chat/README.md
index abdf5d3000..4cdd0e244f 100644
--- a/examples/ai/chat/README.md
+++ b/examples/ai/chat/README.md
@@ -3,7 +3,9 @@
These examples show how to make chatbots with marimo, using [`mo.ui.chat`](https://docs.marimo.io/api/inputs/chat.html#marimo.ui.chat).
- `custom.py` shows how to make a custom chatbot.
+- `streaming_custom.py` shows how to make a custom chatbot with streaming responses.
- `openai_example.py` shows how to make a chatbot powered by OpenAI models.
+- `streaming_openai.py` shows how to make a chatbot powered by OpenAI models with streaming responses.
- `anthropic_example.py` shows how to make a chatbot powered by Anthropic models.
- `gemini.py` shows how to make a chatbot powered by Google models like Gemini.
- `groq_example.py` shows how to make a chatbot powered by Groq models.
@@ -14,6 +16,15 @@ These examples show how to make chatbots with marimo, using [`mo.ui.chat`](https
- `simplemind_example.py` shows how to integrate [simplemind](https://github.com/kennethreitz/simplemind).
- `generative_ui.py` shows how to make a chatbot that can generate UI code.
+## Streaming Responses
+
+Chatbots can stream responses in real-time, creating a more interactive experience similar to ChatGPT. To enable streaming:
+
+- For **custom models**: create an async generator function that yields intermediate results
+- For **built-in models** (OpenAI, Anthropic, etc.): set `stream=True` in the model constructor
+
+See `streaming_custom.py` and `streaming_openai.py` for complete examples.
+
Chatbot's in marimo are _reactive_: when the chatbot responds with a message,
all other cells referencing the chatbot are automatically run or marked
stale, with the chatbot's response stored in the object's `value` attribute.
diff --git a/examples/ai/chat/anthropic_example.py b/examples/ai/chat/anthropic_example.py
index d738feb24d..460d066f99 100644
--- a/examples/ai/chat/anthropic_example.py
+++ b/examples/ai/chat/anthropic_example.py
@@ -56,6 +56,7 @@ def _(key, mo):
"claude-3-7-sonnet-20250219",
system_message="You are a helpful assistant.",
api_key=key,
+ # stream=True, # Uncomment to enable streaming responses
),
allow_attachments=[
"image/png",
diff --git a/examples/ai/chat/bedrock_example.py b/examples/ai/chat/bedrock_example.py
index acc1a54870..c65575d452 100644
--- a/examples/ai/chat/bedrock_example.py
+++ b/examples/ai/chat/bedrock_example.py
@@ -192,6 +192,7 @@ def _(config_form, max_tokens, mo, temperature):
"model": model,
"region_name": region,
"system_message": system_message,
+ # "stream": True, # Uncomment to enable streaming responses
}
# Add profile if specified
diff --git a/examples/ai/chat/gemini.py b/examples/ai/chat/gemini.py
index 989ee77915..bf758a1cbb 100644
--- a/examples/ai/chat/gemini.py
+++ b/examples/ai/chat/gemini.py
@@ -2,12 +2,13 @@
# requires-python = ">=3.11"
# dependencies = [
# "marimo",
+# "google-genai>=1.20.0",
# ]
# ///
import marimo
-__generated_with = "0.17.4"
+__generated_with = "0.17.8"
app = marimo.App(width="medium")
@@ -52,9 +53,10 @@ def _(input_key, mo, os_key):
def _(key, mo):
chatbot = mo.ui.chat(
mo.ai.llm.google(
- "gemini-1.5-pro-latest",
+ "gemini-2.5-flash",
system_message="You are a helpful assistant.",
api_key=key,
+ # stream=True, # Uncomment to enable streaming responses
),
prompts=[
"Hello",
diff --git a/examples/ai/chat/groq_example.py b/examples/ai/chat/groq_example.py
index b030f4fcc3..ae10e553bc 100644
--- a/examples/ai/chat/groq_example.py
+++ b/examples/ai/chat/groq_example.py
@@ -73,6 +73,7 @@ def _(key, mo):
model="llama-3.1-70b-versatile",
system_message="You are a helpful assistant.",
api_key=key,
+ # stream=True, # Uncomment to enable streaming responses
),
prompts=[
"Hello",
diff --git a/examples/ai/chat/openai_example.py b/examples/ai/chat/openai_example.py
index 8556a126b0..0e75e45dba 100644
--- a/examples/ai/chat/openai_example.py
+++ b/examples/ai/chat/openai_example.py
@@ -56,6 +56,7 @@ def _(mo, openai_key):
"gpt-4o",
system_message="You are a helpful assistant.",
api_key=openai_key,
+ # stream=True, # Uncomment to enable streaming responses
),
prompts=[
"Hello",
diff --git a/examples/ai/chat/streaming_custom.py b/examples/ai/chat/streaming_custom.py
new file mode 100644
index 0000000000..84218a67f1
--- /dev/null
+++ b/examples/ai/chat/streaming_custom.py
@@ -0,0 +1,102 @@
+# /// script
+# requires-python = ">=3.11"
+# dependencies = [
+# "marimo",
+# ]
+# ///
+
+import marimo
+
+__generated_with = "0.17.8"
+app = marimo.App(width="medium")
+
+
+def _():
+ import marimo as mo
+ import asyncio
+ return asyncio, mo
+
+
[email protected](hide_code=True)
+def _(mo):
+ mo.md("""
+ # Custom streaming chatbot
+
+ This example shows how to make a chatbot that streams responses.
+ Create an async generator function that yields intermediate results,
+ and watch the response appear incrementally!
+ """)
+ return
+
+
+def _(asyncio, mo):
+ async def streaming_echo_model(messages, config):
+ """This chatbot echoes what the user says, word by word."""
+ # Get the user's message
+ user_message = messages[-1].content
+
+ # Stream the response word by word
+ response = f"You said: '{user_message}'. Here's my response streaming word by word!"
+ words = response.split()
+ accumulated = ""
+
+ for word in words:
+ accumulated += word + " "
+ yield accumulated
+ await asyncio.sleep(0.2) # Delay to make streaming visible
+
+ chatbot = mo.ui.chat(
+ streaming_echo_model,
+ prompts=["Hello", "Tell me a story", "What is streaming?"],
+ show_configuration_controls=True
+ )
+ return (chatbot,)
+
+
+def _(chatbot):
+ chatbot
+ return
+
+
+def _(mo):
+ mo.md("""
+ ## How it works
+
+ The key is to make your model function an **async generator**:
+
+ ```python
+ async def my_model(messages, config):
+ response = 'Building up text...'
+ accumulated = ''
+ for part in response.split():
+ accumulated += part + ' '
+ yield accumulated # Each yield updates the UI
+ await asyncio.sleep(0.1)
+ ```
+
+ Each `yield` sends an update to the frontend, creating a smooth streaming effect!
+ """)
+ return
+
+
+def _(mo):
+ mo.md("""
+ Access the chatbot's historical messages with `chatbot.value`.
+ """)
+ return
+
+
+def _(chatbot):
+ # chatbot.value is the list of chat messages
+ chatbot.value
+ return
+
+
+if __name__ == "__main__":
+ app.run()
diff --git a/examples/ai/chat/streaming_openai.py b/examples/ai/chat/streaming_openai.py
new file mode 100644
index 0000000000..4c19223c68
--- /dev/null
+++ b/examples/ai/chat/streaming_openai.py
@@ -0,0 +1,117 @@
+# /// script
+# requires-python = ">=3.11"
+# dependencies = [
+# "marimo",
+# "openai>=1.55.3",
+# ]
+# ///
+
+import marimo
+
+__generated_with = "0.17.8"
+app = marimo.App(width="medium")
+
+
+def _():
+ import marimo as mo
+ return (mo,)
+
+
[email protected](hide_code=True)
+def _(mo):
+ mo.md("""
+ # OpenAI streaming chatbot
+
+ This example shows how to use OpenAI's API with streaming responses.
+ The built-in `mo.ai.llm.openai()` model automatically streams tokens
+ as they arrive from the API!
+
+ Enter your API key below to try it out.
+ """)
+ return
+
+
+def _(mo):
+ api_key_input = mo.ui.text(
+ placeholder="sk-...",
+ label="OpenAI API Key",
+ kind="password",
+ )
+ api_key_input
+ return (api_key_input,)
+
+
+def _(api_key_input, mo):
+ if api_key_input.value:
+ chatbot = mo.ui.chat(
+ mo.ai.llm.openai(
+ "gpt-4o-mini",
+ system_message="You are a helpful assistant. Keep responses concise and friendly.",
+ api_key=api_key_input.value,
+ stream=True, # Enable streaming
+ ),
+ prompts=[
+ "Tell me a short joke",
+ "What is Python?",
+ "Explain streaming in one sentence",
+ ],
+ show_configuration_controls=True,
+ )
+ else:
+ chatbot = mo.md("*Enter your OpenAI API key above to start chatting*")
+ return (chatbot,)
+
+
+def _(chatbot):
+ chatbot
+ return
+
+
+def _(mo):
+ mo.md("""
+ ## How it works
+
+ The built-in OpenAI model returns an async generator that yields tokens
+ as they stream from the API:
+
+ ```python
+ mo.ui.chat(
+ mo.ai.llm.openai(
+ "gpt-4o-mini",
+ api_key="your-key",
+ stream=True, # Enable streaming!
+ )
+ )
+ ```
+
+ Set `stream=True` to enable streaming responses. 🚀
+
+ Other built-in models (`anthropic`, `google`, `groq`) work the same way.
+ """)
+ return
+
+
+def _(chatbot, mo):
+ # Show chat history
+ if hasattr(chatbot, 'value'):
+ mo.md(f"**Chat history:** {len(chatbot.value)} messages")
+ return
+
+
+def _(chatbot):
+ # Display full history
+ if hasattr(chatbot, 'value'):
+ chatbot.value
+ return
+
+
+if __name__ == "__main__":
+ app.run()
+
diff --git a/frontend/src/plugins/impl/chat/ChatPlugin.tsx b/frontend/src/plugins/impl/chat/ChatPlugin.tsx
index ae134eec0c..fe95eba8f7 100644
--- a/frontend/src/plugins/impl/chat/ChatPlugin.tsx
+++ b/frontend/src/plugins/impl/chat/ChatPlugin.tsx
@@ -89,6 +89,7 @@ export const ChatPlugin = createPlugin<{ messages: ChatMessage[] }>(
send_prompt={props.functions.send_prompt}
value={props.value?.messages || Arrays.EMPTY}
setValue={(messages) => props.setValue({ messages })}
+ host={props.host}
/>
</Suspense>
</TooltipProvider>
diff --git a/frontend/src/plugins/impl/chat/chat-ui.tsx b/frontend/src/plugins/impl/chat/chat-ui.tsx
index c2683e0f15..99afa72eee 100644
--- a/frontend/src/plugins/impl/chat/chat-ui.tsx
+++ b/frontend/src/plugins/impl/chat/chat-ui.tsx
@@ -12,6 +12,7 @@ import {
DownloadIcon,
HelpCircleIcon,
PaperclipIcon,
+ RotateCwIcon,
SendIcon,
SettingsIcon,
Trash2Icon,
@@ -42,7 +43,12 @@ import {
import { Tooltip } from "@/components/ui/tooltip";
import { toast } from "@/components/ui/use-toast";
import { moveToEndOfEditor } from "@/core/codemirror/utils";
+import { MarimoIncomingMessageEvent } from "@/core/dom/events";
import { useAsyncData } from "@/hooks/useAsyncData";
+import {
+ type HTMLElementNotDerivedFromRef,
+ useEventListener,
+} from "@/hooks/useEventListener";
import { cn } from "@/utils/cn";
import { copyToClipboard } from "@/utils/copy";
import { Logger } from "@/utils/Logger";
@@ -63,6 +69,7 @@ interface Props extends PluginFunctions {
allowAttachments: boolean | string[];
value: ChatMessage[];
setValue: (messages: ChatMessage[]) => void;
+ host: HTMLElement;
}
export const Chatbot: React.FC<Props> = (props) => {
@@ -74,6 +81,12 @@ export const Chatbot: React.FC<Props> = (props) => {
const codeMirrorInputRef = useRef<ReactCodeMirrorRef>(null);
const scrollContainerRef = useRef<HTMLDivElement>(null);
+ // Track streaming state - maps backend message_id to frontend message index
+ const streamingStateRef = useRef<{
+ backendMessageId: string | null;
+ frontendMessageIndex: number | null;
+ }>({ backendMessageId: null, frontendMessageIndex: null });
+
const { data: initialMessages } = useAsyncData(async () => {
const chatMessages = await props.get_chat_history({});
const messages: UIMessage[] = chatMessages.messages.map((message, idx) => ({
@@ -113,6 +126,19 @@ export const Chatbot: React.FC<Props> = (props) => {
.join("\n"),
parts: m.parts,
}));
+
+ // Create a placeholder message for streaming
+ const messageId = Date.now().toString();
+
+ setMessages((prev) => [
+ ...prev,
+ {
+ id: messageId,
+ role: "assistant",
+ parts: [{ type: "text", text: "" }],
+ },
+ ]);
+
const response = await props.send_prompt({
messages: messages,
config: {
@@ -124,18 +150,35 @@ export const Chatbot: React.FC<Props> = (props) => {
presence_penalty: config.presence_penalty,
},
});
- // Update local state with AI response
- setMessages((prev) => [
- ...prev,
- {
- id: Date.now().toString(),
- role: "assistant",
- parts: [{ type: "text", text: response }],
- },
- ]);
+
+ // If streaming didn't happen (non-generator response), update the message
+ // Check if streaming state is still set (meaning no chunks were received)
+ if (
+ streamingStateRef.current.backendMessageId === null &&
+ streamingStateRef.current.frontendMessageIndex === null
+ ) {
+ setMessages((prev) => {
+ const updated = [...prev];
+ const index = updated.findIndex((m) => m.id === messageId);
+ if (index !== -1) {
+ updated[index] = {
+ ...updated[index],
+ parts: [{ type: "text", text: response }],
+ };
+ }
+ return updated;
+ });
+ }
+
return new Response(response);
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} catch (error: any) {
+ // Clear streaming state on error
+ streamingStateRef.current = {
+ backendMessageId: null,
+ frontendMessageIndex: null,
+ };
+
// HACK: strip the error message to clean up the response
const strippedError = error.message
.split("failed with exception ")
@@ -152,12 +195,98 @@ export const Chatbot: React.FC<Props> = (props) => {
fileInputRef.current.value = "";
}
Logger.debug("Finished streaming message:", message);
+
+ // Clear streaming state
+ streamingStateRef.current = {
+ backendMessageId: null,
+ frontendMessageIndex: null,
+ };
},
onError: (error) => {
Logger.error("An error occurred:", error);
+ // Clear streaming state on error
+ streamingStateRef.current = {
+ backendMessageId: null,
+ frontendMessageIndex: null,
+ };
},
});
+ // Listen for streaming chunks from backend
+ useEventListener(
+ props.host as HTMLElementNotDerivedFromRef,
+ MarimoIncomingMessageEvent.TYPE,
+ (e) => {
+ const message = e.detail.message;
+ if (
+ typeof message === "object" &&
+ message !== null &&
+ "type" in message &&
+ message.type === "stream_chunk"
+ ) {
+ const chunkMessage = message as {
+ type: string;
+ message_id: string;
+ content: string;
+ is_final: boolean;
+ };
+
+ // Initialize streaming state on first chunk if not already set
+ if (streamingStateRef.current.backendMessageId === null) {
+ // Find the last assistant message (which should be the placeholder we created)
+ setMessages((prev) => {
+ const updated = [...prev];
+ // Find the last assistant message
+ for (let i = updated.length - 1; i >= 0; i--) {
+ if (updated[i].role === "assistant") {
+ streamingStateRef.current = {
+ backendMessageId: chunkMessage.message_id,
+ frontendMessageIndex: i,
+ };
+ break;
+ }
+ }
+ return updated;
+ });
+ }
+
+ // Only process chunks for the current streaming message
+ const frontendIndex = streamingStateRef.current.frontendMessageIndex;
+ if (
+ streamingStateRef.current.backendMessageId ===
+ chunkMessage.message_id &&
+ frontendIndex !== null
+ ) {
+ setMessages((prev) => {
+ const updated = [...prev];
+ const index = frontendIndex;
+
+ // Update the message at the tracked index
+ if (index < updated.length) {
+ const messageToUpdate = updated[index];
+ if (messageToUpdate.role === "assistant") {
+ updated[index] = {
+ ...messageToUpdate,
+ parts: [{ type: "text", text: chunkMessage.content }],
+ };
+ }
+ }
+
+ return updated;
+ });
+
+ // Clear streaming state when final chunk arrives
+ if (chunkMessage.is_final) {
+ streamingStateRef.current = {
+ backendMessageId: null,
+ frontendMessageIndex: null,
+ };
+ }
+ }
+ }
+ },
+ );
+
const isLoading = status === "submitted" || status === "streaming";
const handleDelete = (id: string) => {
@@ -283,13 +412,14 @@ export const Chatbot: React.FC<Props> = (props) => {
<Button
variant="text"
size="icon"
+ disabled={messages.length === 0}
onClick={() => {
setMessages([]);
props.setValue([]);
props.delete_chat_history({});
}}
>
- <Trash2Icon className="h-3 w-3" />
+ <RotateCwIcon className="h-3 w-3" />
</Button>
</div>
<div
diff --git a/marimo/_ai/llm/_impl.py b/marimo/_ai/llm/_impl.py
index a1459a9378..d3e77ffae6 100644
--- a/marimo/_ai/llm/_impl.py
+++ b/marimo/_ai/llm/_impl.py
@@ -2,7 +2,10 @@
from __future__ import annotations
import os
-from typing import Any, Callable, Optional, cast
+from typing import TYPE_CHECKING, Any, Callable, Optional, cast
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
from marimo._ai._convert import (
convert_to_anthropic_messages,
@@ -55,6 +58,10 @@ class openai(ChatModel):
If not provided, the API key will be retrieved
from the OPENAI_API_KEY environment variable or the user's config.
base_url: The base URL to use
+ stream: Whether to stream the response token-by-token as it's generated.
+ When True, the response appears progressively (like ChatGPT), creating
+ a more responsive user experience. When False, the complete response
+ is returned all at once. Defaults to False for backward compatibility.
"""
def __init__(
@@ -64,11 +71,13 @@ class openai(ChatModel):
system_message: str = DEFAULT_SYSTEM_MESSAGE,
api_key: Optional[str] = None,
base_url: Optional[str] = None,
+ stream: bool = False,
):
self.model = model
self.system_message = system_message
self.api_key = api_key
self.base_url = base_url
+ self.stream = stream
@property
def _require_api_key(self) -> str:
@@ -96,6 +105,21 @@ class openai(ChatModel):
"set OPENAI_API_KEY as an environment variable"
)
+ def _stream_response(self, response: Any) -> Generator[str, None, None]:
+ """Helper method for streaming - separate to avoid mixing yield/return."""
+ accumulated = ""
+ chunk_count = 0
+ for chunk in response:
+ chunk_count += 1
+ if chunk.choices and len(chunk.choices) > 0:
+ delta = chunk.choices[0].delta
+ if delta.content:
+ accumulated += delta.content
+ yield accumulated
+ # Always yield final accumulated result to ensure complete response
+ # This handles cases where the last chunk has no content
+ yield accumulated
+
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
) -> object:
@@ -135,6 +159,7 @@ class openai(ChatModel):
[ChatMessage(role="system", content=self.system_message)]
+ messages
)
+
response = client.chat.completions.create(
model=self.model,
messages=openai_messages,
@@ -143,12 +168,17 @@ class openai(ChatModel):
top_p=config.top_p,
frequency_penalty=config.frequency_penalty,
presence_penalty=config.presence_penalty,
- stream=False,
+ stream=self.stream,
)
- choice = response.choices[0]
- content = choice.message.content
- return content or ""
+ if self.stream:
+ # Return generator for streaming
+ return self._stream_response(response)
+ else:
+ # Non-streaming response - return string directly
+ choice = cast(Any, response).choices[0]
+ content = choice.message.content
+ return content or ""
class anthropic(ChatModel):
@@ -164,6 +194,10 @@ class anthropic(ChatModel):
from the ANTHROPIC_API_KEY environment variable
or the user's config.
base_url: The base URL to use
+ stream: Whether to stream the response token-by-token as it's generated.
+ When True, the response appears progressively (like ChatGPT), creating
+ a more responsive user experience. When False, the complete response
+ is returned all at once. Defaults to False for backward compatibility.
"""
def __init__(
@@ -173,11 +207,13 @@ class anthropic(ChatModel):
system_message: str = DEFAULT_SYSTEM_MESSAGE,
api_key: Optional[str] = None,
base_url: Optional[str] = None,
+ stream: bool = False,
):
self.model = model
self.system_message = system_message
self.api_key = api_key
self.base_url = base_url
+ self.stream = stream
@property
def _require_api_key(self) -> str:
@@ -205,6 +241,19 @@ class anthropic(ChatModel):
"set ANTHROPIC_API_KEY as an environment variable"
)
+ def _stream_response(
+ self, client: Any, params: Any
+ ) -> Generator[str, None, None]:
+ """Helper method for streaming - separate to avoid mixing yield/return."""
+ accumulated = ""
+ with client.messages.stream(**params) as stream:
+ for text in stream.text_stream:
+ accumulated += text
+ yield accumulated
+ # Yield final accumulated result to ensure complete response
+ if accumulated:
+ yield accumulated
+
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
) -> object:
@@ -224,7 +273,7 @@ class anthropic(ChatModel):
"system": self.system_message,
"max_tokens": config.max_tokens or 4096,
"messages": anthropic_messages,
- "stream": False,
+ "stream": self.stream,
}
if config.top_p is not None:
params["top_p"] = config.top_p
@@ -233,15 +282,23 @@ class anthropic(ChatModel):
if config.temperature is not None:
params["temperature"] = config.temperature
- response = client.messages.create(**params)
-
- content = response.content
- if len(content) > 0:
- if content[0].type == "text":
- return content[0].text
- elif content[0].type == "tool_use":
- return content
- return ""
+ if self.stream:
+ # Return generator for streaming
+ return self._stream_response(client, params)
+ else:
+ # Non-streaming response - return directly
+ response = client.messages.create(**params)
+
+ content = response.content
+ if len(content) > 0:
+ if content[0].type == "text":
+ return content[0].text
+ elif content[0].type == "tool_use":
+ return content
+ else:
+ return ""
+ else:
+ return ""
class google(ChatModel):
@@ -256,6 +313,10 @@ class google(ChatModel):
If not provided, the API key will be retrieved
from the GOOGLE_AI_API_KEY environment variable
or the user's config.
+ stream: Whether to stream the response token-by-token as it's generated.
+ When True, the response appears progressively (like ChatGPT), creating
+ a more responsive user experience. When False, the complete response
+ is returned all at once. Defaults to False for backward compatibility.
"""
def __init__(
@@ -264,10 +325,12 @@ class google(ChatModel):
*,
system_message: str = DEFAULT_SYSTEM_MESSAGE,
api_key: Optional[str] = None,
+ stream: bool = False,
):
self.model = model
self.system_message = system_message
self.api_key = api_key
+ self.stream = stream
@property
def _require_api_key(self) -> str:
@@ -306,22 +369,51 @@ class google(ChatModel):
client = genai.Client(api_key=self._require_api_key)
google_messages = convert_to_google_messages(messages)
- response = client.models.generate_content(
+
+ # Build config once to avoid duplication
+ generation_config = {
+ "system_instruction": self.system_message,
+ "max_output_tokens": config.max_tokens,
+ "temperature": config.temperature,
+ "top_p": config.top_p,
+ "top_k": config.top_k,
+ "frequency_penalty": config.frequency_penalty,
+ "presence_penalty": config.presence_penalty,
+ }
+
+ if self.stream:
+ # Return generator for streaming
+ return self._stream_response(
+ client, google_messages, generation_config
+ )
+ else:
+ # Non-streaming response - return directly
+ response = client.models.generate_content(
+ model=self.model,
+ contents=google_messages,
+ config=cast(Any, generation_config),
+ )
+
+ content = response.text
+ return content or ""
+
+ def _stream_response(
+ self, client: Any, google_messages: Any, generation_config: Any
+ ) -> Generator[str, None, None]:
+ """Helper method for streaming - separate to avoid mixing yield/return."""
+ accumulated = ""
+ response = client.models.generate_content_stream(
model=self.model,
contents=google_messages,
- config={
- "system_instruction": self.system_message,
- "max_output_tokens": config.max_tokens,
- "temperature": config.temperature,
- "top_p": config.top_p,
- "top_k": config.top_k,
- "frequency_penalty": config.frequency_penalty,
- "presence_penalty": config.presence_penalty,
- },
+ config=generation_config,
)
-
- content = response.text
- return content or ""
+ for chunk in response:
+ if chunk.text:
+ accumulated += chunk.text
+ yield accumulated
+ # Yield final accumulated result to ensure complete response
+ if accumulated:
+ yield accumulated
class groq(ChatModel):
@@ -336,6 +428,10 @@ class groq(ChatModel):
If not provided, the API key will be retrieved
from the GROQ_API_KEY environment variable or the user's config.
base_url: The base URL to use
+ stream: Whether to stream the response token-by-token as it's generated.
+ When True, the response appears progressively (like ChatGPT), creating
+ a more responsive user experience. When False, the complete response
+ is returned all at once. Defaults to False for backward compatibility.
"""
def __init__(
@@ -345,11 +441,13 @@ class groq(ChatModel):
system_message: str = DEFAULT_SYSTEM_MESSAGE,
api_key: Optional[str] = None,
base_url: Optional[str] = None,
+ stream: bool = False,
):
self.model = model
self.system_message = system_message
self.api_key = api_key
self.base_url = base_url
+ self.stream = stream
@property
def _require_api_key(self) -> str:
@@ -378,6 +476,31 @@ class groq(ChatModel):
"set GROQ_API_KEY as an environment variable"
)
+ def _stream_response(
+ self, client: Any, groq_messages: Any, config: ChatModelConfig
+ ) -> Generator[str, None, None]:
+ """Helper method for streaming - separate to avoid mixing yield/return."""
+ stream = client.chat.completions.create(
+ model=self.model,
+ messages=groq_messages,
+ max_tokens=config.max_tokens,
+ temperature=config.temperature,
+ top_p=config.top_p,
+ stop=None,
+ stream=True,
+ )
+
+ accumulated = ""
+ for chunk in stream:
+ if chunk.choices and len(chunk.choices) > 0:
+ delta = chunk.choices[0].delta
+ if delta.content:
+ accumulated += delta.content
+ yield accumulated
+ # Yield final accumulated result to ensure complete response
+ if accumulated:
+ yield accumulated
+
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
) -> object:
@@ -392,19 +515,25 @@ class groq(ChatModel):
[ChatMessage(role="system", content=self.system_message)]
+ messages
)
- response = client.chat.completions.create(
- model=self.model,
- messages=groq_messages,
- max_tokens=config.max_tokens,
- temperature=config.temperature,
- top_p=config.top_p,
- stop=None,
- stream=False,
- )
- choice = response.choices[0]
- content = choice.message.content
- return content or ""
+ if self.stream:
+ # Return generator for streaming
+ return self._stream_response(client, groq_messages, config)
+ else:
+ # Non-streaming response - return directly
+ response = client.chat.completions.create(
+ model=self.model,
+ messages=groq_messages,
+ max_tokens=config.max_tokens,
+ temperature=config.temperature,
+ top_p=config.top_p,
+ stop=None,
+ stream=False,
+ )
+
+ choice = response.choices[0]
+ content = choice.message.content
+ return content or ""
class bedrock(ChatModel):
@@ -421,6 +550,10 @@ class bedrock(ChatModel):
Dict with keys: "aws_access_key_id" and "aws_secret_access_key"
If not provided, credentials will be retrieved from the environment
or the AWS configuration files.
+ stream: Whether to stream the response token-by-token as it's generated.
+ When True, the response appears progressively (like ChatGPT), creating
+ a more responsive user experience. When False, the complete response
+ is returned all at once. Defaults to False for backward compatibility.
"""
def __init__(
@@ -432,6 +565,7 @@ class bedrock(ChatModel):
profile_name: Optional[str] = None,
aws_access_key_id: Optional[str] = None,
aws_secret_access_key: Optional[str] = None,
+ stream: bool = False,
):
if not model.startswith("bedrock/"):
model = f"bedrock/{model}"
@@ -441,6 +575,7 @@ class bedrock(ChatModel):
self.profile_name = profile_name
self.aws_access_key_id = aws_access_key_id
self.aws_secret_access_key = aws_secret_access_key
+ self.stream = stream
def _setup_credentials(self) -> None:
# Use profile name if provided, otherwise use API key
@@ -452,6 +587,37 @@ class bedrock(ChatModel):
else:
pass # Use default credential chain
+ def _stream_response(
+ self, messages: list[ChatMessage], config: ChatModelConfig
+ ) -> Generator[str, None, None]:
+ """Helper method for streaming - separate to avoid mixing yield/return."""
+ from litellm import completion as litellm_completion
+
+ response = litellm_completion(
+ model=self.model,
+ messages=convert_to_openai_messages(
+ [ChatMessage(role="system", content=self.system_message)]
+ + messages
+ ),
+ max_tokens=config.max_tokens,
+ temperature=config.temperature,
+ top_p=config.top_p,
+ frequency_penalty=config.frequency_penalty,
+ presence_penalty=config.presence_penalty,
+ stream=True,
+ )
+
+ accumulated = ""
+ for chunk in response:
+ if chunk.choices and len(chunk.choices) > 0:
+ delta = chunk.choices[0].delta
+ if delta.content:
+ accumulated += delta.content
+ yield accumulated
+ # Yield final accumulated result to ensure complete response
+ if accumulated:
+ yield accumulated
+
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
) -> object:
@@ -466,22 +632,29 @@ class bedrock(ChatModel):
self._setup_credentials()
try:
- # Make API call
- response = litellm_completion(
- model=self.model,
- messages=convert_to_openai_messages(
- [ChatMessage(role="system", content=self.system_message)]
- + messages
- ),
- max_tokens=config.max_tokens,
- temperature=config.temperature,
- top_p=config.top_p,
- frequency_penalty=config.frequency_penalty,
- presence_penalty=config.presence_penalty,
- stream=False,
- )
-
- return response.choices[0].message.content
+ if self.stream:
+ # Return generator for streaming
+ return self._stream_response(messages, config)
+ else:
+ # Non-streaming response - return directly
+ response = litellm_completion(
+ model=self.model,
+ messages=convert_to_openai_messages(
+ [
+ ChatMessage(
+ role="system", content=self.system_message
+ )
+ ]
+ + messages
+ ),
+ max_tokens=config.max_tokens,
+ temperature=config.temperature,
+ top_p=config.top_p,
+ frequency_penalty=config.frequency_penalty,
+ presence_penalty=config.presence_penalty,
+ stream=False,
+ )
+ return response.choices[0].message.content
except Exception as e:
# Handle common AWS exceptions with helpful messages
diff --git a/marimo/_plugins/ui/_impl/chat/chat.py b/marimo/_plugins/ui/_impl/chat/chat.py
index 140bfcf83c..3646e65b42 100644
--- a/marimo/_plugins/ui/_impl/chat/chat.py
+++ b/marimo/_plugins/ui/_impl/chat/chat.py
@@ -2,6 +2,7 @@
from __future__ import annotations
import inspect
+import uuid
from dataclasses import dataclass
from typing import Any, Callable, Final, Optional, Union, cast
@@ -20,7 +21,7 @@ from marimo._runtime.functions import EmptyArgs, Function
from marimo._runtime.requests import SetUIElementValueRequest
DEFAULT_CONFIG = ChatModelConfigDict(
- max_tokens=100,
+ max_tokens=4096,
temperature=0.5,
top_p=1,
top_k=40,
@@ -77,16 +78,23 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):
return await my_async_function(messages)
```
+ Regular (sync) generators for streaming:
```python
- async def my_rag_model(messages):
- for response in my_async_iterator(messages):
- yield response
+ def my_streaming_model(messages, config):
+ for chunk in process_stream():
+ yield chunk # Each yield updates the UI
+ ```
+
+ Async generators for streaming with async operations:
+ ```python
+ async def my_async_streaming_model(messages, config):
+ async for chunk in async_process_stream():
+ yield chunk # Each yield updates the UI
```
- The last value yielded by the async generator is treated as the model
- response. ui.chat does not yet support streaming responses to the frontend.
- Please file a GitHub issue if this is important to you:
- https://github.com/marimo-team/marimo/issues
+ The last value yielded by the generator is treated as the model
+ response. Streaming responses are automatically streamed to the frontend
+ as they are generated.
Using a built-in model:
```python
@@ -206,6 +214,56 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):
del self._chat_history[index]
self._value = self._chat_history
+ async def _handle_streaming_response(self, response: Any) -> str:
+ """Handle streaming from both sync and async generators."""
+ message_id = str(uuid.uuid4())
+ latest_response = None
+ accumulated_text = ""
+
+ # Use async for if it's an async generator, otherwise regular for
+ if inspect.isasyncgen(response):
+ async for latest_response in response: # noqa: B007
+ accumulated_text = str(latest_response)
+ self._send_message(
+ {
+ "type": "stream_chunk",
+ "message_id": message_id,
+ "content": accumulated_text,
+ "is_final": False,
+ },
+ buffers=None,
+ )
+ else:
+ for latest_response in response: # noqa: B007
+ accumulated_text = str(latest_response)
+ self._send_message(
+ {
+ "type": "stream_chunk",
+ "message_id": message_id,
+ "content": accumulated_text,
+ "is_final": False,
+ },
+ buffers=None,
+ )
+
+ # Send final message to indicate streaming is complete
+ if latest_response is not None:
+ self._send_message(
+ {
+ "type": "stream_chunk",
+ "message_id": message_id,
+ "content": accumulated_text,
+ "is_final": True,
+ },
+ buffers=None,
+ )
+
+ return (
+ str(latest_response)
+ if latest_response is not None
+ else accumulated_text
+ )
+
async def _send_prompt(self, args: SendMessageRequest) -> str:
messages = args.messages
@@ -223,20 +281,11 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):
if inspect.isawaitable(response):
response = await response
- elif inspect.isasyncgen(response):
- # We support functions that stream the response with an async
- # generator; each yielded value is the latest representation of the
- # response, and the last value is the full value
- latest_response = None
- async for latest_response in response: # noqa: B007
- # TODO(akshayka, mscolnick): Stream response to frontend
- # once bidirectional communication is implemented.
- #
- # RPCs don't yet support bidirectional communication, so we
- # just ignore all the initial responses; ideally we'd stream
- # the response back to the frontend.
- pass
- response = latest_response
+ elif inspect.isasyncgen(response) or inspect.isgenerator(response):
+ # We support functions that stream the response with generators
+ # (both sync and async); each yielded value is the latest
+ # representation of the response, and the last value is the full value
+ response = await self._handle_streaming_response(response)
response_message = ChatMessage(role="assistant", content=response)
self._chat_history = messages + [response_message]
diff --git a/pixi.lock b/pixi.lock
index 70d9afb0de..48f4c828c2 100644
--- a/pixi.lock
+++ b/pixi.lock
@@ -106,25 +106,17 @@ environments:
- conda: https://conda.anaconda.org/conda-forge/noarch/zipp-3.22.0-pyhd8ed1ab_0.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/zlib-1.3.1-hb9d3cd8_2.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/zstandard-0.23.0-py312h66e93f0_2.conda
- - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/8f/d7/9322c609343d929e75e7e5e6255e614fcc67572cfd083959cdef3b7aad79/docutils-0.21.2-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/c0/5a/9cac0c82afec3d09ccd97c8b6502d48f165f9124db81b4bcb90b4af974ee/jedi-0.19.2-py2.py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/7b/43/59fdc2f6267959b71dd23ce0bd8d4aeaf55566aa435a5d00f53d53c7eb24/jiter-0.11.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- pypi: https://files.pythonhosted.org/packages/a3/48/f97f0920c20bc522c34f1eaec4c62d7a03c95c37da2dec858f65c8cb4325/loro-1.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- pypi: https://files.pythonhosted.org/packages/51/3f/afe76f8e2246ffbc867440cbcf90525264df0e658f8a5ca1f872b3f6192a/markdown-3.8-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/87/51/0fa83662b036bdac504192e8067798b6d0ef912eec2af897361e60357808/msgspec_m-0.19.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
- pypi: https://files.pythonhosted.org/packages/50/3b/0e2c535c3e6970cfc5763b67f6cc31accaab35a7aa3e322fb6a12830450f/narwhals-2.6.0-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/8c/74/6bfc3adc81f6c2cea4439f2a734c40e3a420703bbcdc539890096a732bbd/openai-2.7.1-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/c6/ac/dac4a63f978e4dcb3c6d3a78c4d8e0192a113d288502a1216950c41b1027/parso-0.8.4-py2.py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/bf/b9/b0eb3f3cbcb734d930fdf839431606844a825b23eaf9a6ab371edac8162c/psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- - pypi: https://files.pythonhosted.org/packages/bd/1f/73c53fcbfb0b5a78f91176df41945ca466e71e9d9d836e5c522abda39ee7/pydantic-2.11.10-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/f9/41/4b043778cf9c4285d59742281a769eac371b9e47e35f98ad321349cc5d61/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- pypi: https://files.pythonhosted.org/packages/a7/d1/c54e608505776ce4e7966d03358ae635cfd51dff1da6ee421c090dbc797b/pymdown_extensions-10.15-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/8b/0c/9d30a4ebeb6db2b25a841afbb80f6ef9a854fc3b41be131d249a977b4959/starlette-0.46.2-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/b1/4b/4cef6ce21a2aaca9d852a6e84ef4f135d99fcd74fa75105e2fc0c8308acd/uvicorn-0.34.2-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/14/8f/aa61f528fba38578ec553c145857a181384c72b98156f858ca5c8e82d9d3/websockets-15.0.1-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- pypi: ./
@@ -211,25 +203,17 @@ environments:
- conda: https://conda.anaconda.org/conda-forge/noarch/zipp-3.22.0-pyhd8ed1ab_0.conda
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/zlib-1.3.1-h8359307_2.conda
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/zstandard-0.23.0-py312hea69d52_2.conda
- - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/8f/d7/9322c609343d929e75e7e5e6255e614fcc67572cfd083959cdef3b7aad79/docutils-0.21.2-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/c0/5a/9cac0c82afec3d09ccd97c8b6502d48f165f9124db81b4bcb90b4af974ee/jedi-0.19.2-py2.py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/f7/29/6c7de6b5d6e511d9e736312c0c9bfcee8f9b6bef68182a08b1d78767e627/jiter-0.11.1-cp312-cp312-macosx_11_0_arm64.whl
- pypi: https://files.pythonhosted.org/packages/32/ee/7e042a13e93f09c24618dc804f01849735d81740f8b74827124ff7fc8396/loro-1.5.0-cp312-cp312-macosx_11_0_arm64.whl
- pypi: https://files.pythonhosted.org/packages/3f/08/83871f3c50fc983b88547c196d11cf8c3340e37c32d2e9d6152abe2c61f7/Markdown-3.7-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/4c/b7/a93d524907162cb6150179eb47fac885bbbad025c82151b69ad1d62cda4d/msgspec_m-0.19.2-cp312-cp312-macosx_11_0_arm64.whl
- pypi: https://files.pythonhosted.org/packages/7d/81/1cf7a468ef2f8e88266d5c3e5ab026a045aa76150e6640bfe9c5450a8c11/narwhals-2.3.0-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/8c/74/6bfc3adc81f6c2cea4439f2a734c40e3a420703bbcdc539890096a732bbd/openai-2.7.1-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/c6/ac/dac4a63f978e4dcb3c6d3a78c4d8e0192a113d288502a1216950c41b1027/parso-0.8.4-py2.py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/04/8b/30f930733afe425e3cbfc0e1468a30a18942350c1a8816acfade80c005c4/psutil-7.0.0-cp36-abi3-macosx_11_0_arm64.whl
- - pypi: https://files.pythonhosted.org/packages/bd/1f/73c53fcbfb0b5a78f91176df41945ca466e71e9d9d836e5c522abda39ee7/pydantic-2.11.10-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/a1/02/6224312aacb3c8ecbaa959897af57181fb6cf3a3d7917fd44d0f2917e6f2/pydantic_core-2.33.2-cp312-cp312-macosx_11_0_arm64.whl
- pypi: https://files.pythonhosted.org/packages/e4/06/43084e6cbd4b3bc0e80f6be743b2e79fbc6eed8de9ad8c629939fa55d972/pymdown_extensions-10.16.1-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/a0/4b/528ccf7a982216885a1ff4908e886b8fb5f19862d1962f56a3fce2435a70/starlette-0.46.1-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl
- - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/61/14/33a3a1352cfa71812a3a21e8c9bfb83f60b0011f5e36f2b1399d51928209/uvicorn-0.34.0-py3-none-any.whl
- pypi: https://files.pythonhosted.org/packages/3d/69/1a681dd6f02180916f116894181eab8b2e25b31e484c5d0eae637ec01f7c/websockets-15.0.1-cp312-cp312-macosx_11_0_arm64.whl
- pypi: ./
@@ -255,13 +239,6 @@ packages:
purls: []
size: 23621
timestamp: 1650670423406
-- pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl
- name: annotated-types
- version: 0.7.0
- sha256: 1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53
- requires_dist:
- - typing-extensions>=4.0.0 ; python_full_version < '3.9'
- requires_python: '>=3.8'
- conda: https://conda.anaconda.org/conda-forge/noarch/anyio-4.9.0-pyh29332c3_0.conda
sha256: b28e0f78bb0c7962630001e63af25a89224ff504e135a02e50d4d80b6155d386
md5: 9749a2c77a7c40d432ea0927662d7e52
@@ -450,11 +427,6 @@ packages:
- pkg:pypi/distlib?source=hash-mapping
size: 274151
timestamp: 1733238487461
-- pypi: https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl
- name: distro
- version: 1.9.0
- sha256: 7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2
- requires_python: '>=3.6'
- pypi: https://files.pythonhosted.org/packages/8f/d7/9322c609343d929e75e7e5e6255e614fcc67572cfd083959cdef3b7aad79/docutils-0.21.2-py3-none-any.whl
name: docutils
version: 0.21.2
@@ -793,16 +765,6 @@ packages:
- pkg:pypi/jeepney?source=hash-mapping
size: 40015
timestamp: 1740828380668
-- pypi: https://files.pythonhosted.org/packages/7b/43/59fdc2f6267959b71dd23ce0bd8d4aeaf55566aa435a5d00f53d53c7eb24/jiter-0.11.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- name: jiter
- version: 0.11.1
- sha256: eb68e7bf65c990531ad8715e57d50195daf7c8e6f1509e617b4e692af1108939
- requires_python: '>=3.9'
-- pypi: https://files.pythonhosted.org/packages/f7/29/6c7de6b5d6e511d9e736312c0c9bfcee8f9b6bef68182a08b1d78767e627/jiter-0.11.1-cp312-cp312-macosx_11_0_arm64.whl
- name: jiter
- version: 0.11.1
- sha256: ecd4da91b5415f183a6be8f7158d127bdd9e6a3174138293c0d48d6ea2f2009d
- requires_python: '>=3.9'
- conda: https://conda.anaconda.org/conda-forge/noarch/keyring-25.6.0-pyh534df25_0.conda
sha256: c8b436fa9853bf8b836c96afbb7684a04955b80b37f5d5285fd836b6a8566cc5
md5: d2c0c5bda93c249f877c7fceea9e63af
@@ -1139,8 +1101,8 @@ packages:
timestamp: 1727801725384
- pypi: ./
name: marimo
- version: 0.17.7
- sha256: cad6192c0a7ae8eff42fc50f18bd870864e322892608c70ca274c95b3308b632
+ version: 0.17.8
+ sha256: 45bc4e2f2d1b8607176ff73aa7bc6a442a4bd44498b7d9b2b17052470215b7bf
requires_dist:
- click>=8.0,<9
- jedi>=0.18.0
@@ -1160,7 +1122,6 @@ packages:
- narwhals>=2.0.0
- packaging
- msgspec-m>=0.19.2
- - openai>=2.7.1
- duckdb>=1.0.0 ; extra == 'sql'
- polars[pyarrow]>=1.9.0 ; extra == 'sql'
- sqlglot[rs]>=26.2.0 ; extra == 'sql'
@@ -1430,28 +1391,6 @@ packages:
purls: []
size: 15490642
timestamp: 1737401388520
-- pypi: https://files.pythonhosted.org/packages/8c/74/6bfc3adc81f6c2cea4439f2a734c40e3a420703bbcdc539890096a732bbd/openai-2.7.1-py3-none-any.whl
- name: openai
- version: 2.7.1
- sha256: 2f2530354d94c59c614645a4662b9dab0a5b881c5cd767a8587398feac0c9021
- requires_dist:
- - anyio>=3.5.0,<5
- - distro>=1.7.0,<2
- - httpx>=0.23.0,<1
- - jiter>=0.10.0,<1
- - pydantic>=1.9.0,<3
- - sniffio
- - tqdm>4
- - typing-extensions>=4.11,<5
- - aiohttp ; extra == 'aiohttp'
- - httpx-aiohttp>=0.1.9 ; extra == 'aiohttp'
- - numpy>=1 ; extra == 'datalib'
- - pandas-stubs>=1.1.0.11 ; extra == 'datalib'
- - pandas>=1.2.3 ; extra == 'datalib'
- - websockets>=13,<16 ; extra == 'realtime'
- - numpy>=2.0.2 ; extra == 'voice-helpers'
- - sounddevice>=0.5.1 ; extra == 'voice-helpers'
- requires_python: '>=3.8'
- conda: https://conda.anaconda.org/conda-forge/linux-64/openssl-3.5.0-h7b32b05_1.conda
sha256: b4491077c494dbf0b5eaa6d87738c22f2154e9277e5293175ec187634bd808a0
md5: de356753cfdbffcde5bb1e86e3aa6cd0
@@ -1704,32 +1643,6 @@ packages:
- pkg:pypi/pycparser?source=hash-mapping
size: 110100
timestamp: 1733195786147
-- pypi: https://files.pythonhosted.org/packages/bd/1f/73c53fcbfb0b5a78f91176df41945ca466e71e9d9d836e5c522abda39ee7/pydantic-2.11.10-py3-none-any.whl
- name: pydantic
- version: 2.11.10
- sha256: 802a655709d49bd004c31e865ef37da30b540786a46bfce02333e0e24b5fe29a
- requires_dist:
- - annotated-types>=0.6.0
- - pydantic-core==2.33.2
- - typing-extensions>=4.12.2
- - typing-inspection>=0.4.0
- - email-validator>=2.0.0 ; extra == 'email'
- - tzdata ; python_full_version >= '3.9' and sys_platform == 'win32' and extra == 'timezone'
- requires_python: '>=3.9'
-- pypi: https://files.pythonhosted.org/packages/a1/02/6224312aacb3c8ecbaa959897af57181fb6cf3a3d7917fd44d0f2917e6f2/pydantic_core-2.33.2-cp312-cp312-macosx_11_0_arm64.whl
- name: pydantic-core
- version: 2.33.2
- sha256: 3c6db6e52c6d70aa0d00d45cdb9b40f0433b96380071ea80b09277dba021ddf7
- requires_dist:
- - typing-extensions>=4.6.0,!=4.7.0
- requires_python: '>=3.9'
-- pypi: https://files.pythonhosted.org/packages/f9/41/4b043778cf9c4285d59742281a769eac371b9e47e35f98ad321349cc5d61/pydantic_core-2.33.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- name: pydantic-core
- version: 2.33.2
- sha256: 8f57a69461af2a5fa6e6bbd7a5f60d3b7e6cebb687f55106933188e79ad155c1
- requires_dist:
- - typing-extensions>=4.6.0,!=4.7.0
- requires_python: '>=3.9'
- conda: https://conda.anaconda.org/conda-forge/noarch/pygments-2.19.1-pyhd8ed1ab_0.conda
sha256: 28a3e3161390a9d23bc02b4419448f8d27679d9e2c250e29849e37749c8de86b
md5: 232fb4577b6687b2d503ef8e254270c9
@@ -2014,22 +1927,6 @@ packages:
- pkg:pypi/tomlkit?source=hash-mapping
size: 37372
timestamp: 1733230836889
-- pypi: https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl
- name: tqdm
- version: 4.67.1
- sha256: 26445eca388f82e72884e0d580d5464cd801a3ea01e63e5601bdff9ba6a48de2
- requires_dist:
- - colorama ; sys_platform == 'win32'
- - pytest>=6 ; extra == 'dev'
- - pytest-cov ; extra == 'dev'
- - pytest-timeout ; extra == 'dev'
- - pytest-asyncio>=0.24 ; extra == 'dev'
- - nbval ; extra == 'dev'
- - requests ; extra == 'discord'
- - slack-sdk ; extra == 'slack'
- - requests ; extra == 'telegram'
- - ipywidgets>=6 ; extra == 'notebook'
- requires_python: '>=3.7'
- conda: https://conda.anaconda.org/conda-forge/noarch/trove-classifiers-2025.5.9.12-pyhd8ed1ab_0.conda
sha256: 455b7b0dc0cf7e4a6fcc41455b4fd7f646b3b842e6dc0d894438366827d7d9b2
md5: 764db08a8d868de9e377d88277c75d83
@@ -2041,13 +1938,6 @@ packages:
- pkg:pypi/trove-classifiers?source=hash-mapping
size: 19516
timestamp: 1746817031708
-- pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl
- name: typing-inspection
- version: 0.4.2
- sha256: 4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7
- requires_dist:
- - typing-extensions>=4.12.0
- requires_python: '>=3.9'
- conda: https://conda.anaconda.org/conda-forge/noarch/typing_extensions-4.13.2-pyh29332c3_0.conda
sha256: a8aaf351e6461de0d5d47e4911257e25eec2fa409d71f3b643bb2f748bde1c08
md5: 83fc6ae00127671e301c9f44254c31b8
diff --git a/tests/_plugins/ui/_impl/chat/test_chat.py b/tests/_plugins/ui/_impl/chat/test_chat.py
index c7b729427b..0af644fae2 100644
--- a/tests/_plugins/ui/_impl/chat/test_chat.py
+++ b/tests/_plugins/ui/_impl/chat/test_chat.py
@@ -139,6 +139,134 @@ async def test_chat_send_prompt_async_generator():
assert chat._chat_history[1].content == "2"
+async def test_chat_streaming_sends_messages():
+ """Test that streaming async generators send messages via _send_message"""
+ sent_messages = []
+
+ async def mock_streaming_model(
+ messages: list[ChatMessage], config: ChatModelConfig
+ ) -> AsyncIterator[str]:
+ del config, messages
+ # Simulate streaming response
+ accumulated = ""
+ for word in ["Hello", "world", "!"]:
+ accumulated += word + " "
+ yield accumulated.strip()
+
+ chat = ui.chat(mock_streaming_model)
+
+ # Mock _send_message to capture calls
+ original_send_message = chat._send_message
+
+ def capture_send_message(message: dict[str, object], buffers): # noqa: ARG001
+ sent_messages.append(message)
+ # Don't actually send to avoid needing kernel context
+
+ chat._send_message = capture_send_message
+
+ request = SendMessageRequest(
+ messages=[ChatMessage(role="user", content="Test")],
+ config=ChatModelConfig(),
+ )
+
+ response: str = await chat._send_prompt(request)
+
+ # Verify final response
+ assert response == "Hello world !"
+
+ # Verify streaming messages were sent
+ # Should have sent chunks for each yield, plus final message
+ assert len(sent_messages) >= 3
+
+ # Check that messages have streaming structure
+ for msg in sent_messages[:-1]: # All but last
+ assert msg["type"] == "stream_chunk"
+ assert "message_id" in msg
+ assert "content" in msg
+ assert not msg["is_final"]
+
+ # Last message should be final
+ assert sent_messages[-1]["type"] == "stream_chunk"
+ assert sent_messages[-1]["is_final"]
+ assert sent_messages[-1]["content"] == "Hello world !"
+
+
+async def test_chat_sync_generator_streaming():
+ """Test that sync generators also work for streaming"""
+ sent_messages = []
+
+ def mock_streaming_model(
+ messages: list[ChatMessage], config: ChatModelConfig
+ ):
+ del config, messages
+ # Simulate streaming response with sync generator
+ accumulated = ""
+ for word in ["Hello", "world", "!"]:
+ accumulated += word + " "
+ yield accumulated.strip()
+
+ chat = ui.chat(mock_streaming_model)
+
+ def capture_send_message(message: dict[str, object], buffers): # noqa: ARG001
+ sent_messages.append(message)
+
+ chat._send_message = capture_send_message
+
+ request = SendMessageRequest(
+ messages=[ChatMessage(role="user", content="Test")],
+ config=ChatModelConfig(),
+ )
+
+ response: str = await chat._send_prompt(request)
+
+ # Verify final response
+ assert response == "Hello world !"
+
+ # Verify streaming messages were sent
+ assert len(sent_messages) >= 3
+
+ # Check that messages have streaming structure
+ for msg in sent_messages[:-1]: # All but last
+ assert msg["type"] == "stream_chunk"
+ assert "message_id" in msg
+ assert "content" in msg
+ assert not msg["is_final"]
+
+ # Last message should be final
+ assert sent_messages[-1]["type"] == "stream_chunk"
+ assert sent_messages[-1]["is_final"]
+ assert sent_messages[-1]["content"] == "Hello world !"
+
+
+async def test_chat_streaming_complete_response():
+ """Test that streaming returns complete response even with empty final chunks"""
+
+ def mock_streaming_model_with_empty_final(
+ messages: list[ChatMessage], config: ChatModelConfig
+ ):
+ del config, messages
+ # Simulate OpenAI-style streaming where last chunk might be empty
+ # This tests the bug where we might not yield the final accumulated value
+ yield "Hello "
+ yield "Hello world"
+ yield "Hello world!"
+ # Simulate final chunk with no content (like finish_reason only)
+ # Generator continues but no more content - still need complete result
+
+ chat = ui.chat(mock_streaming_model_with_empty_final)
+
+ request = SendMessageRequest(
+ messages=[ChatMessage(role="user", content="Test")],
+ config=ChatModelConfig(),
+ )
+
+ response: str = await chat._send_prompt(request)
+
+ # Verify we got the complete final response
+ assert response == "Hello world!"
+ assert chat._chat_history[-1].content == "Hello world!"
+
+
def test_chat_get_history():
def mock_model(
messages: list[ChatMessage], config: ChatModelConfig