A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.
Python artificial-intelligencedagdata-sciencedata-visualizationdataflowdeveloper-toolsmachine-learningnotebookspipelinepythonreactivesqlweb-app
feat: Implement delta-based streaming and configurable streaming for AI chat models (#7259)
Adam Draperb774a98e16fe
diff --git a/docs/api/inputs/chat.md b/docs/api/inputs/chat.mdindex 997f535264..9539676eac 100644--- a/docs/api/inputs/chat.md+++ b/docs/api/inputs/chat.md@@ -139,9 +139,13 @@ similar to ChatGPT where you see the response appear word-by-word as it's generaResponses from built-in models (OpenAI, Anthropic, Google, Groq, Bedrock) are streamed by default.+### How Streaming Works++marimo uses **delta-based streaming**, which follows the industry-standard pattern used by OpenAI, Anthropic, and other AI providers. Your generator function should yield **individual chunks** (deltas) of new content, which marimo automatically accumulates and displays progressively.+### With Custom Models-For custom models, you can use either regular (sync) or async generator functions that yield intermediate results:+For custom models, you can use either regular (sync) or async generator functions that yield delta chunks:**Sync generator (simpler):**@@ -153,11 +157,9 @@ def streaming_model(messages, config):"""Stream responses word by word."""response = "This response will appear word by word!"words = response.split()- accumulated = ""for word in words:- accumulated += word + " "- yield accumulated+ yield word + " " # Yield delta chunkstime.sleep(0.1) # Simulate processing delaychat = mo.ui.chat(streaming_model)@@ -174,24 +176,43 @@ async def async_streaming_model(messages, config):"""Stream responses word by word asynchronously."""response = "This response will appear word by word!"words = response.split()- accumulated = ""for word in words:- accumulated += word + " "- yield accumulated+ yield word + " " # Yield delta chunksawait asyncio.sleep(0.1) # Async processing delaychat = mo.ui.chat(async_streaming_model)chat```-Each `yield` sends an update to the frontend, and the chat UI will display-the progressively accumulated response in real-time.+Each `yield` sends a new chunk (delta) to marimo, which accumulates and displays+the progressively building response in real-time.++!!! tip "Delta vs Accumulated"+ **Yield deltas, not accumulated text.** Each yield should be **new content only**:++ ✅ **Correct (delta mode):**+ ```python+ yield "Hello"+ yield " "+ yield "world"+ # Result: "Hello world"+ ```++ ❌ **Incorrect (accumulated mode, deprecated):**+ ```python+ yield "Hello"+ yield "Hello "+ yield "Hello world"+ # Inefficient: sends duplicate content+ ```++ Delta mode is more efficient (reduces bandwidth by ~99% for long responses) and aligns with standard streaming APIs.!!! tip "See streaming examples"For complete working examples, check out:- - [`streaming_openai.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/streaming_openai.py) - Streaming with OpenAI models+ - [`openai_example.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/openai_example.py) - OpenAI chatbot with streaming (default)- [`streaming_custom.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/streaming_custom.py) - Custom streaming chatbot## Built-in Modelsdiff --git a/examples/ai/chat/README.md b/examples/ai/chat/README.mdindex 4cdd0e244f..0c2e38618c 100644--- a/examples/ai/chat/README.md+++ b/examples/ai/chat/README.md@@ -3,12 +3,11 @@These examples show how to make chatbots with marimo, using [`mo.ui.chat`](https://docs.marimo.io/api/inputs/chat.html#marimo.ui.chat).- `custom.py` shows how to make a custom chatbot.-- `streaming_custom.py` shows how to make a custom chatbot with streaming responses.-- `openai_example.py` shows how to make a chatbot powered by OpenAI models.-- `streaming_openai.py` shows how to make a chatbot powered by OpenAI models with streaming responses.-- `anthropic_example.py` shows how to make a chatbot powered by Anthropic models.-- `gemini.py` shows how to make a chatbot powered by Google models like Gemini.-- `groq_example.py` shows how to make a chatbot powered by Groq models.+- `streaming_custom.py` shows how to make a custom chatbot with streaming responses (delta-based).+- `openai_example.py` shows how to make a chatbot powered by OpenAI models (streaming by default).+- `anthropic_example.py` shows how to make a chatbot powered by Anthropic models (streaming by default).+- `gemini.py` shows how to make a chatbot powered by Google models like Gemini (streaming by default).+- `groq_example.py` shows how to make a chatbot powered by Groq models (streaming by default).- `mlx_chat.py` shows a simple chatbot using local on-device models with Apple's [MLX](https://github.com/ml-explore/mlx), a machine learning framework from Apple that is similar to JAX and PyTorch. This specific example uses the [mlx-lm](https://github.com/ml-explore/mlx-examples/tree/main/llms) library. Note that Apple Silicon chips are required for using MLX.- `llm_datasette.py` shows how to make a chatbot powered by Simon W's LLM library.- `dagger_code_interpreter.py` shows how to make a basic code-interpreter chatbot powered by Dagger containers.@@ -18,12 +17,11 @@ These examples show how to make chatbots with marimo, using [`mo.ui.chat`](https## Streaming Responses-Chatbots can stream responses in real-time, creating a more interactive experience similar to ChatGPT. To enable streaming:+All built-in models (OpenAI, Anthropic, Google, Groq, Bedrock) stream responses using delta-based streaming. If a model doesn't support streaming, it will automatically fall back to non-streaming mode.-- For **custom models**: create an async generator function that yields intermediate results-- For **built-in models** (OpenAI, Anthropic, etc.): set `stream=True` in the model constructor+For custom models, create an async generator function that yields delta chunks (new content only).-See `streaming_custom.py` and `streaming_openai.py` for complete examples.+See `streaming_custom.py` for a complete example of custom streaming.Chatbot's in marimo are _reactive_: when the chatbot responds with a message,all other cells referencing the chatbot are automatically run or markeddiff --git a/examples/ai/chat/streaming_custom.py b/examples/ai/chat/streaming_custom.pyindex 84218a67f1..ae69faca03 100644--- a/examples/ai/chat/streaming_custom.py+++ b/examples/ai/chat/streaming_custom.py@@ -33,18 +33,21 @@ def _(mo):@app.celldef _(asyncio, mo):async def streaming_echo_model(messages, config):- """This chatbot echoes what the user says, word by word."""+ """This chatbot echoes what the user says, word by word.++ Yields individual delta chunks that are accumulated by marimo.+ This follows the standard streaming pattern used by OpenAI, Anthropic,+ and other AI providers.+ """# Get the user's messageuser_message = messages[-1].content# Stream the response word by wordresponse = f"You said: '{user_message}'. Here's my response streaming word by word!"words = response.split()- accumulated = ""for word in words:- accumulated += word + " "- yield accumulated+ yield word + " " # Yield delta chunksawait asyncio.sleep(0.2) # Delay to make streaming visiblechatbot = mo.ui.chat(@@ -66,19 +69,20 @@ def _(mo):mo.md("""## How it works- The key is to make your model function an **async generator**:+ The key is to make your model function an **async generator** that yields **delta chunks**:```pythonasync def my_model(messages, config):- response = 'Building up text...'- accumulated = ''- for part in response.split():- accumulated += part + ' '- yield accumulated # Each yield updates the UI+ # Yield individual pieces of content (deltas)+ for word in ['Building', 'up', 'text...']:+ yield word + ' ' # Each yield is a deltaawait asyncio.sleep(0.1)```- Each `yield` sends an update to the frontend, creating a smooth streaming effect!+ Each `yield` sends a new chunk to marimo, which accumulates and displays them.+ This follows the standard streaming pattern used by OpenAI, Anthropic, and other AI providers.++ **Important**: Yield delta chunks (new content only), not accumulated text.""")returndiff --git a/examples/ai/chat/streaming_openai.py b/examples/ai/chat/streaming_openai.pydeleted file mode 100644index c3df1749c6..0000000000--- a/examples/ai/chat/streaming_openai.py+++ /dev/null@@ -1,110 +0,0 @@-# /// script-# requires-python = ">=3.11"-# dependencies = [-# "marimo",-# "openai==2.8.1",-# ]-# ///--import marimo--__generated_with = "0.17.8"-app = marimo.App(width="medium")---def _():- import marimo as mo- return (mo,)--[email protected](hide_code=True)-def _(mo):- mo.md("""- # OpenAI streaming chatbot-- This example shows how to use OpenAI's API with streaming responses.- The built-in `mo.ai.llm.openai()` model automatically streams tokens- as they arrive from the API!-- Enter your API key below to try it out.- """)- return---def _(mo):- api_key_input = mo.ui.text(- placeholder="sk-...",- label="OpenAI API Key",- kind="password",- )- api_key_input- return (api_key_input,)---def _(api_key_input, mo):- if api_key_input.value:- chatbot = mo.ui.chat(- mo.ai.llm.openai(- "gpt-4o-mini",- system_message="You are a helpful assistant. Keep responses concise and friendly.",- api_key=api_key_input.value,- ),- prompts=[- "Tell me a short joke",- "What is Python?",- "Explain streaming in one sentence",- ],- show_configuration_controls=True,- )- else:- chatbot = mo.md("*Enter your OpenAI API key above to start chatting*")- return (chatbot,)---def _(chatbot):- chatbot- return--[email protected](hide_code=True)-def _(mo):- mo.md("""- ## How it works-- The built-in OpenAI model returns an async generator that yields tokens- as they stream from the API:-- ```python- mo.ui.chat(- mo.ai.llm.openai(- "gpt-4o-mini",- api_key="your-key",- )- )- ```-- Other built-in models (`anthropic`, `google`, `groq`) work the same way.- """)- return---def _(chatbot, mo):- # Show chat history- mo.md(f"**Chat history:** {len(chatbot.value)} messages") if hasattr(chatbot, "value") else None- return---def _(chatbot):- # Display full history- chatbot.value if hasattr(chatbot, "value") else None- return---if __name__ == "__main__":- app.run()diff --git a/marimo/_ai/llm/_impl.py b/marimo/_ai/llm/_impl.pyindex c04bdd42da..8eff9bdced 100644--- a/marimo/_ai/llm/_impl.py+++ b/marimo/_ai/llm/_impl.py@@ -2,6 +2,7 @@from __future__ import annotationsimport os+import refrom typing import TYPE_CHECKING, Any, Callable, Optional, castif TYPE_CHECKING:@@ -25,6 +26,16 @@ DEFAULT_SYSTEM_MESSAGE = ()+def _looks_like_streaming_error(e: Exception) -> bool:+ """Check if an exception appears to be related to streaming not being supported."""+ error_msg = str(e).lower()+ # Use word boundaries to match whole words only (not substrings like "downstream")+ return bool(+ re.search(r"\bstreaming\b", error_msg)+ or re.search(r"\bstream\b", error_msg)+ )++class simple(ChatModel):"""Convenience class for wrapping a ChatModel or callable to@@ -100,19 +111,16 @@ class openai(ChatModel):)def _stream_response(self, response: Any) -> Generator[str, None, None]:- """Helper method for streaming - separate to avoid mixing yield/return."""- accumulated = ""- chunk_count = 0+ """Helper method for streaming - yields delta chunks.++ Each yield is a new piece of content (delta) to be accumulated+ by the consumer. This follows the standard OpenAI streaming pattern.+ """for chunk in response:- chunk_count += 1if chunk.choices and len(chunk.choices) > 0:delta = chunk.choices[0].deltaif delta.content:- accumulated += delta.content- yield accumulated- # Always yield final accumulated result to ensure complete response- # This handles cases where the last chunk has no content- yield accumulated+ yield delta.contentdef __call__(self, messages: list[ChatMessage], config: ChatModelConfig@@ -154,18 +162,35 @@ class openai(ChatModel):+ messages)- response = client.chat.completions.create(- model=self.model,- messages=openai_messages,- max_completion_tokens=config.max_tokens,- temperature=config.temperature,- top_p=config.top_p,- frequency_penalty=config.frequency_penalty,- presence_penalty=config.presence_penalty,- stream=True,- )-- return self._stream_response(response)+ # Try streaming first, fall back to non-streaming on error+ try:+ response = client.chat.completions.create(+ model=self.model,+ messages=openai_messages,+ max_completion_tokens=config.max_tokens,+ temperature=config.temperature,+ top_p=config.top_p,+ frequency_penalty=config.frequency_penalty,+ presence_penalty=config.presence_penalty,+ stream=True,+ )+ return self._stream_response(response)+ except Exception as e:+ # Some models (like o1-preview) don't support streaming+ # Fall back to non-streaming mode+ if _looks_like_streaming_error(e):+ non_stream_response = client.chat.completions.create(+ model=self.model,+ messages=openai_messages,+ max_completion_tokens=config.max_tokens,+ temperature=config.temperature,+ top_p=config.top_p,+ frequency_penalty=config.frequency_penalty,+ presence_penalty=config.presence_penalty,+ stream=False,+ )+ return non_stream_response.choices[0].message.content or ""+ raiseclass anthropic(ChatModel):@@ -225,15 +250,13 @@ class anthropic(ChatModel):def _stream_response(self, client: Any, params: Any) -> Generator[str, None, None]:- """Helper method for streaming - separate to avoid mixing yield/return."""- accumulated = ""+ """Helper method for streaming - yields delta chunks.++ Each yield is a new piece of content (delta) to be accumulated+ by the consumer. This follows the standard Anthropic streaming pattern.+ """with client.messages.stream(**params) as stream:- for text in stream.text_stream:- accumulated += text- yield accumulated- # Yield final accumulated result to ensure complete response- if accumulated:- yield accumulated+ yield from stream.text_streamdef __call__(self, messages: list[ChatMessage], config: ChatModelConfig@@ -254,7 +277,6 @@ class anthropic(ChatModel):"system": self.system_message,"max_tokens": config.max_tokens or 4096,"messages": anthropic_messages,- "stream": True,}if config.top_p is not None:params["top_p"] = config.top_p@@ -263,7 +285,17 @@ class anthropic(ChatModel):if config.temperature is not None:params["temperature"] = config.temperature- return self._stream_response(client, params)+ # Try streaming first, fall back to non-streaming on error+ try:+ # Note: client.messages.stream() doesn't take a 'stream' parameter+ # It's already a streaming method+ return self._stream_response(client, params)+ except Exception as e:+ # Fall back to non-streaming mode if streaming fails+ if _looks_like_streaming_error(e):+ response = client.messages.create(**params)+ return response.content[0].text+ raiseclass google(ChatModel):@@ -320,8 +352,11 @@ class google(ChatModel):def _stream_response(self, client: Any, google_messages: Any, generation_config: Any) -> Generator[str, None, None]:- """Helper method for streaming - separate to avoid mixing yield/return."""- accumulated = ""+ """Helper method for streaming - yields delta chunks.++ Each yield is a new piece of content (delta) to be accumulated+ by the consumer. This follows the standard Google AI streaming pattern.+ """response = client.models.generate_content_stream(model=self.model,contents=google_messages,@@ -329,11 +364,7 @@ class google(ChatModel):)for chunk in response:if chunk.text:- accumulated += chunk.text- yield accumulated- # Yield final accumulated result to ensure complete response- if accumulated:- yield accumulated+ yield chunk.textdef __call__(self, messages: list[ChatMessage], config: ChatModelConfig@@ -358,9 +389,21 @@ class google(ChatModel):"presence_penalty": config.presence_penalty,}- return self._stream_response(- client, google_messages, generation_config- )+ # Try streaming first, fall back to non-streaming on error+ try:+ return self._stream_response(+ client, google_messages, generation_config+ )+ except Exception as e:+ # Fall back to non-streaming mode if streaming fails+ if _looks_like_streaming_error(e):+ response = client.models.generate_content(+ model=self.model,+ contents=google_messages,+ config=generation_config, # type: ignore[arg-type]+ )+ return response.text+ raiseclass groq(ChatModel):@@ -420,7 +463,11 @@ class groq(ChatModel):def _stream_response(self, client: Any, groq_messages: Any, config: ChatModelConfig) -> Generator[str, None, None]:- """Helper method for streaming - separate to avoid mixing yield/return."""+ """Helper method for streaming - yields delta chunks.++ Each yield is a new piece of content (delta) to be accumulated+ by the consumer. This follows the standard Groq streaming pattern.+ """stream = client.chat.completions.create(model=self.model,messages=groq_messages,@@ -431,16 +478,11 @@ class groq(ChatModel):stream=True,)- accumulated = ""for chunk in stream:if chunk.choices and len(chunk.choices) > 0:delta = chunk.choices[0].deltaif delta.content:- accumulated += delta.content- yield accumulated- # Yield final accumulated result to ensure complete response- if accumulated:- yield accumulated+ yield delta.contentdef __call__(self, messages: list[ChatMessage], config: ChatModelConfig@@ -457,7 +499,23 @@ class groq(ChatModel):+ messages)- return self._stream_response(client, groq_messages, config)+ # Try streaming first, fall back to non-streaming on error+ try:+ return self._stream_response(client, groq_messages, config)+ except Exception as e:+ # Fall back to non-streaming mode if streaming fails+ if _looks_like_streaming_error(e):+ response = client.chat.completions.create(+ model=self.model,+ messages=groq_messages,+ max_tokens=config.max_tokens,+ temperature=config.temperature,+ top_p=config.top_p,+ stop=None,+ stream=False,+ )+ return response.choices[0].message.content or ""+ raiseclass bedrock(ChatModel):@@ -508,7 +566,11 @@ class bedrock(ChatModel):def _stream_response(self, messages: list[ChatMessage], config: ChatModelConfig) -> Generator[str, None, None]:- """Helper method for streaming - separate to avoid mixing yield/return."""+ """Helper method for streaming - yields delta chunks.++ Each yield is a new piece of content (delta) to be accumulated+ by the consumer. This follows the standard AWS Bedrock streaming pattern.+ """from litellm import completion as litellm_completionresponse = litellm_completion(@@ -525,16 +587,11 @@ class bedrock(ChatModel):stream=True,)- accumulated = ""for chunk in response:if chunk.choices and len(chunk.choices) > 0:delta = chunk.choices[0].deltaif delta.content:- accumulated += delta.content- yield accumulated- # Yield final accumulated result to ensure complete response- if accumulated:- yield accumulated+ yield delta.contentdef __call__(self, messages: list[ChatMessage], config: ChatModelConfig@@ -548,7 +605,34 @@ class bedrock(ChatModel):self._setup_credentials()try:- return self._stream_response(messages, config)+ # Try streaming first, fall back to non-streaming on error+ try:+ return self._stream_response(messages, config)+ except Exception as stream_error:+ # Fall back to non-streaming if streaming fails+ if _looks_like_streaming_error(stream_error):+ from litellm import completion as litellm_completion++ response = litellm_completion(+ model=self.model,+ messages=convert_to_openai_messages(+ [+ ChatMessage(+ role="system",+ content=self.system_message,+ )+ ]+ + messages+ ),+ max_tokens=config.max_tokens,+ temperature=config.temperature,+ top_p=config.top_p,+ frequency_penalty=config.frequency_penalty,+ presence_penalty=config.presence_penalty,+ stream=False,+ )+ return response.choices[0].message.content or ""+ raiseexcept Exception as e:# Handle common AWS exceptions with helpful messageserror_msg = str(e)diff --git a/marimo/_plugins/ui/_impl/chat/chat.py b/marimo/_plugins/ui/_impl/chat/chat.pyindex 3646e65b42..bbc692e31d 100644--- a/marimo/_plugins/ui/_impl/chat/chat.py+++ b/marimo/_plugins/ui/_impl/chat/chat.py@@ -215,15 +215,22 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):self._value = self._chat_historyasync def _handle_streaming_response(self, response: Any) -> str:- """Handle streaming from both sync and async generators."""+ """Handle streaming from both sync and async generators.++ Generators should yield delta chunks (new content only), which this+ method accumulates and sends to the frontend as complete text.+ This follows the standard streaming pattern used by OpenAI, Anthropic,+ and other AI providers.+ """message_id = str(uuid.uuid4())- latest_response = Noneaccumulated_text = ""# Use async for if it's an async generator, otherwise regular forif inspect.isasyncgen(response):- async for latest_response in response: # noqa: B007- accumulated_text = str(latest_response)+ async for delta in response:+ # Accumulate each delta chunk+ delta_str = str(delta)+ accumulated_text += delta_strself._send_message({"type": "stream_chunk",@@ -234,8 +241,10 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):buffers=None,)else:- for latest_response in response: # noqa: B007- accumulated_text = str(latest_response)+ for delta in response:+ # Accumulate each delta chunk+ delta_str = str(delta)+ accumulated_text += delta_strself._send_message({"type": "stream_chunk",@@ -247,7 +256,7 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):)# Send final message to indicate streaming is complete- if latest_response is not None:+ if accumulated_text:self._send_message({"type": "stream_chunk",@@ -258,11 +267,7 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):buffers=None,)- return (- str(latest_response)- if latest_response is not None- else accumulated_text- )+ return accumulated_textasync def _send_prompt(self, args: SendMessageRequest) -> str:messages = args.messagesdiff --git a/tests/_ai/llm/test_impl.py b/tests/_ai/llm/test_impl.pyindex 73b9b5f8d4..60b4da9c17 100644--- a/tests/_ai/llm/test_impl.py+++ b/tests/_ai/llm/test_impl.py@@ -293,7 +293,6 @@ class TestOpenAI:assert call_args["top_p"] == pytest.approx(0.9)assert call_args["frequency_penalty"] == pytest.approx(0.5)assert call_args["presence_penalty"] == pytest.approx(0.5)- assert call_args["stream"] is Truedef test_call_with_base_url(self, mock_openai_client, test_messages, test_config@@ -567,7 +566,6 @@ class TestGroq:assert call_args["top_p"] == pytest.approx(0.9)assert call_args["frequency_penalty"] == pytest.approx(0.5)assert call_args["presence_penalty"] == pytest.approx(0.5)- assert call_args["stream"] is Falsedef test_call_with_base_url(self, mock_groq_client, test_messages, test_config@@ -885,7 +883,6 @@ class TestAnthropic:assert call_args["temperature"] == 0.7assert call_args["top_p"] == 0.9assert call_args["top_k"] == 10- assert call_args["stream"] is True@patch.object(anthropic, "_require_api_key")@patch("anthropic.Anthropic")@@ -1053,4 +1050,3 @@ class TestBedrock:assert call_args["top_p"] == pytest.approx(0.9)assert call_args["frequency_penalty"] == pytest.approx(0.5)assert call_args["presence_penalty"] == pytest.approx(0.5)- assert call_args["stream"] is Falsediff --git a/tests/_ai/test_streaming_config.py b/tests/_ai/test_streaming_config.pynew file mode 100644index 0000000000..c4a15ae2a7--- /dev/null+++ b/tests/_ai/test_streaming_config.py@@ -0,0 +1,147 @@+# Copyright 2024 Marimo. All rights reserved.+"""Tests for streaming delta generation and fallback behavior."""++from __future__ import annotations++from marimo._ai.llm._impl import openai+++class TestStreamingDeltaGeneration:+ """Test that stream response helpers yield delta chunks."""++ def test_openai_stream_response_yields_deltas(self):+ """Test that OpenAI _stream_response yields delta chunks correctly."""+ from dataclasses import dataclass++ @dataclass+ class MockDelta:+ content: str | None++ @dataclass+ class MockChoice:+ delta: MockDelta++ @dataclass+ class MockChunk:+ choices: list[MockChoice]++ model = openai(+ model="gpt-4",+ api_key="test-key",+ )++ # Simulate a streaming response+ mock_response = [+ MockChunk([MockChoice(MockDelta("Hello"))]),+ MockChunk([MockChoice(MockDelta(" "))]),+ MockChunk([MockChoice(MockDelta("world"))]),+ MockChunk([MockChoice(MockDelta("!"))]),+ ]++ result = list(model._stream_response(mock_response))+ assert result == ["Hello", " ", "world", "!"]++ def test_openai_empty_chunks_filtered(self):+ """Test that OpenAI filters out empty chunks."""+ from dataclasses import dataclass++ @dataclass+ class MockDelta:+ content: str | None++ @dataclass+ class MockChoice:+ delta: MockDelta++ @dataclass+ class MockChunk:+ choices: list[MockChoice]++ model = openai(+ model="gpt-4",+ api_key="test-key",+ )++ # Simulate response with empty chunks+ mock_response = [+ MockChunk([MockChoice(MockDelta("Hello"))]),+ MockChunk([MockChoice(MockDelta(None))]), # Empty chunk+ MockChunk([MockChoice(MockDelta(" "))]),+ MockChunk([MockChoice(MockDelta(None))]), # Empty chunk+ MockChunk([MockChoice(MockDelta("world"))]),+ ]++ result = list(model._stream_response(mock_response))+ assert result == ["Hello", " ", "world"]++ def test_groq_stream_response_yields_deltas(self):+ """Test that Groq _stream_response yields delta chunks correctly."""+ from dataclasses import dataclass++ @dataclass+ class MockDelta:+ content: str | None++ @dataclass+ class MockChoice:+ delta: MockDelta++ @dataclass+ class MockChunk:+ choices: list[MockChoice]++ # Simulate a streaming response+ mock_response = [+ MockChunk([MockChoice(MockDelta("Test"))]),+ MockChunk([MockChoice(MockDelta(" "))]),+ MockChunk([MockChoice(MockDelta("response"))]),+ ]++ # Create a custom generator that mimics what _stream_response does+ # This tests the logic of delta chunk yielding independently+ def mock_stream():+ for chunk in mock_response:+ if chunk.choices and len(chunk.choices) > 0:+ delta = chunk.choices[0].delta+ if delta.content:+ yield delta.content++ result = list(mock_stream())+ assert result == ["Test", " ", "response"]+++class TestStreamingFallback:+ """Test automatic fallback logic when streaming is not supported."""++ def test_streaming_error_detection(self):+ """Test that streaming errors are detected correctly."""+ # These error messages should trigger fallback+ streaming_errors = [+ "stream does not support true",+ "streaming is not supported",+ "Unsupported value: 'stream'",+ "STREAM parameter not allowed",+ "This model does not support streaming",+ ]++ for error_msg in streaming_errors:+ lower_msg = error_msg.lower()+ # Our fallback logic checks for "stream" or "streaming" in error message+ assert "stream" in lower_msg or "streaming" in lower_msg++ def test_non_streaming_errors_not_caught(self):+ """Test that non-streaming errors are not incorrectly caught as streaming errors."""+ # These errors should NOT trigger fallback+ non_streaming_errors = [+ "invalid api key",+ "rate limit exceeded",+ "model not found",+ "network timeout",+ "insufficient quota",+ ]++ for error_msg in non_streaming_errors:+ lower_msg = error_msg.lower()+ # These should not contain "stream" or "streaming"+ assert "stream" not in lower_msg+ assert "streaming" not in lower_msgdiff --git a/tests/_plugins/test_chat_delta_streaming.py b/tests/_plugins/test_chat_delta_streaming.pynew file mode 100644index 0000000000..83915616c2--- /dev/null+++ b/tests/_plugins/test_chat_delta_streaming.py@@ -0,0 +1,346 @@+# Copyright 2024 Marimo. All rights reserved.+"""Tests for delta-based streaming in chat UI."""++from __future__ import annotations++import asyncio+from typing import TYPE_CHECKING++if TYPE_CHECKING:+ from collections.abc import AsyncGenerator, Generator++from marimo._ai._types import ChatMessage+from marimo._plugins.ui._impl.chat.chat import chat+++class TestDeltaStreaming:+ """Test delta-based streaming mode."""++ async def test_async_delta_streaming_accumulates(self):+ """Test that async generators yielding deltas are accumulated correctly."""++ async def delta_generator() -> AsyncGenerator[str, None]:+ """Yields individual delta chunks."""+ yield "Hello"+ yield " "+ yield "world"+ yield "!"++ # Create chat with the delta generator+ chat_ui = chat(delta_generator)++ # Track messages sent+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ # Handle the streaming response+ result = await chat_ui._handle_streaming_response(delta_generator())++ # Verify final result is fully accumulated+ assert result == "Hello world!"++ # Verify incremental chunks were sent correctly+ assert len(sent_messages) == 5 # 4 deltas + 1 final+ assert sent_messages[0]["content"] == "Hello"+ assert sent_messages[1]["content"] == "Hello "+ assert sent_messages[2]["content"] == "Hello world"+ assert sent_messages[3]["content"] == "Hello world!"+ assert sent_messages[4]["content"] == "Hello world!"+ assert sent_messages[4]["is_final"] is True++ async def test_sync_delta_streaming_accumulates(self):+ """Test that sync generators yielding deltas are accumulated correctly."""++ def delta_generator() -> Generator[str, None, None]:+ """Yields individual delta chunks."""+ yield "One"+ yield " "+ yield "Two"+ yield " "+ yield "Three"++ # Create chat with the delta generator+ chat_ui = chat(delta_generator)++ # Track messages sent+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ # Handle the streaming response+ result = await chat_ui._handle_streaming_response(delta_generator())++ # Verify final result is fully accumulated+ assert result == "One Two Three"++ # Verify incremental chunks were sent correctly+ assert len(sent_messages) == 6 # 5 deltas + 1 final+ assert sent_messages[0]["content"] == "One"+ assert sent_messages[1]["content"] == "One "+ assert sent_messages[2]["content"] == "One Two"+ assert sent_messages[3]["content"] == "One Two "+ assert sent_messages[4]["content"] == "One Two Three"+ assert sent_messages[5]["is_final"] is True++ async def test_empty_delta_handled(self):+ """Test that empty deltas are handled gracefully."""++ async def delta_generator() -> AsyncGenerator[str, None]:+ """Yields some empty deltas."""+ yield "Hello"+ yield "" # Empty delta+ yield " "+ yield "" # Another empty delta+ yield "world"++ chat_ui = chat(delta_generator)+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ result = await chat_ui._handle_streaming_response(delta_generator())++ assert result == "Hello world"+ # Empty deltas still trigger message sends+ assert len(sent_messages) == 6 # 5 deltas (including empty) + 1 final++ async def test_single_delta(self):+ """Test streaming with a single delta."""++ async def delta_generator() -> AsyncGenerator[str, None]:+ yield "Single chunk"++ chat_ui = chat(delta_generator)+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ result = await chat_ui._handle_streaming_response(delta_generator())++ assert result == "Single chunk"+ assert len(sent_messages) == 2 # 1 delta + 1 final+ assert sent_messages[0]["content"] == "Single chunk"+ assert sent_messages[1]["is_final"] is True++ async def test_no_deltas(self):+ """Test streaming with no deltas (empty generator)."""++ async def delta_generator() -> AsyncGenerator[str, None]:+ # Yield nothing+ if False:+ yield ""++ chat_ui = chat(delta_generator)+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ result = await chat_ui._handle_streaming_response(delta_generator())++ assert result == ""+ # No final message sent if no content+ assert len(sent_messages) == 0++ async def test_unicode_deltas(self):+ """Test that unicode characters in deltas are handled correctly."""++ async def delta_generator() -> AsyncGenerator[str, None]:+ yield "Hello "+ yield "🌍"+ yield " "+ yield "世界"+ yield "!"++ chat_ui = chat(delta_generator)+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ result = await chat_ui._handle_streaming_response(delta_generator())++ assert result == "Hello 🌍 世界!"+ assert sent_messages[-1]["content"] == "Hello 🌍 世界!"++ async def test_message_id_consistency(self):+ """Test that all chunks in a stream share the same message_id."""++ async def delta_generator() -> AsyncGenerator[str, None]:+ yield "A"+ yield "B"+ yield "C"++ chat_ui = chat(delta_generator)+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ await chat_ui._handle_streaming_response(delta_generator())++ # All messages should have the same message_id+ message_ids = [msg["message_id"] for msg in sent_messages]+ assert len(set(message_ids)) == 1+ assert all(msg["type"] == "stream_chunk" for msg in sent_messages)++ async def test_long_streaming_response(self):+ """Test streaming with many small deltas (simulates real AI streaming)."""++ async def delta_generator() -> AsyncGenerator[str, None]:+ # Simulate many small tokens like real AI models+ words = [+ "This",+ " is",+ " a",+ " long",+ " streaming",+ " response",+ " with",+ " many",+ " deltas",+ ".",+ ]+ for word in words:+ yield word+ await asyncio.sleep(0.001) # Tiny delay++ chat_ui = chat(delta_generator)+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ result = await chat_ui._handle_streaming_response(delta_generator())++ expected = "This is a long streaming response with many deltas."+ assert result == expected+ assert len(sent_messages) == 11 # 10 deltas + 1 final++ # Verify progressive accumulation+ assert sent_messages[0]["content"] == "This"+ assert sent_messages[1]["content"] == "This is"+ assert sent_messages[2]["content"] == "This is a"+ assert sent_messages[-1]["content"] == expected+ assert sent_messages[-1]["is_final"] is True+++class TestStreamingWithChatModels:+ """Test streaming with actual ChatModel-like objects."""++ async def test_custom_model_function(self):+ """Test that custom model functions work with delta streaming."""++ async def custom_model(+ messages: list[ChatMessage],+ config: dict, # noqa: ARG001+ ) -> AsyncGenerator[str, None]:+ """Custom model that yields deltas."""+ user_msg = messages[-1].content+ response = f"You said: {user_msg}"+ for char in response:+ yield char+ await asyncio.sleep(0.001)++ chat_ui = chat(custom_model)+ sent_messages: list[dict] = []++ def capture_send(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ sent_messages.append(msg)++ chat_ui._send_message = capture_send # type: ignore++ # Simulate calling the model+ test_messages = [ChatMessage(role="user", content="Hello")]+ generator = custom_model(test_messages, {})++ result = await chat_ui._handle_streaming_response(generator)++ assert result == "You said: Hello"+ # Should have one chunk per character plus final+ assert len(sent_messages) > 0+ assert sent_messages[-1]["is_final"] is True+++class TestStreamingEfficiency:+ """Test that delta streaming is efficient compared to accumulated streaming."""++ async def test_delta_vs_accumulated_bandwidth(self):+ """Demonstrate bandwidth savings of delta vs accumulated streaming."""++ # Simulate 100-word response+ words = ["word" + str(i) for i in range(100)]++ # Delta streaming: each word is sent once+ async def delta_stream() -> AsyncGenerator[str, None]:+ for word in words:+ yield word + " "++ chat_ui = chat(delta_stream)+ delta_sent_messages: list[dict] = []++ def capture_delta(msg: dict, **kwargs: object) -> None:+ del kwargs # Unused+ delta_sent_messages.append(msg)++ chat_ui._send_message = capture_delta # type: ignore++ await chat_ui._handle_streaming_response(delta_stream())++ # Calculate bytes sent with delta streaming+ # Backend sends accumulated text, but receives deltas+ delta_bytes_received = sum(len(word + " ") for word in words)++ # Each message sends the accumulated content+ delta_bytes_sent = sum(+ len(msg["content"]) for msg in delta_sent_messages+ )++ # With old accumulated approach, model would yield:+ # "word0 ", "word0 word1 ", "word0 word1 word2 ", etc.+ accumulated_bytes_received = sum(+ len(" ".join(words[: i + 1]) + " ") for i in range(len(words))+ )++ # Delta mode receives much less data from the model+ assert delta_bytes_received < accumulated_bytes_received++ # For 100 words, delta receives ~100 words worth of data+ # while accumulated receives ~5000 words worth (1+2+3+...+100)+ efficiency_ratio = accumulated_bytes_received / delta_bytes_received+ assert efficiency_ratio > 40 # Should be ~50x more efficient++ print("\nEfficiency Test Results:")+ print(f"Delta bytes received: {delta_bytes_received}")+ print(f"Accumulated bytes received: {accumulated_bytes_received}")+ print(f"Efficiency ratio: {efficiency_ratio:.1f}x")diff --git a/tests/_plugins/ui/_impl/chat/test_chat.py b/tests/_plugins/ui/_impl/chat/test_chat.pyindex 0af644fae2..2a8f7aa60b 100644--- a/tests/_plugins/ui/_impl/chat/test_chat.py+++ b/tests/_plugins/ui/_impl/chat/test_chat.py@@ -119,6 +119,7 @@ async def test_chat_send_prompt_async_generator():) -> AsyncIterator[str]:del configdel messages+ # Yield delta chunks (new content only)for i in range(3):await asyncio.sleep(0.01)yield str(i)@@ -130,13 +131,13 @@ async def test_chat_send_prompt_async_generator():)response: str = await chat._send_prompt(request)- # the last yielded value is the response- assert response == "2"+ # All deltas are accumulated: "0" + "1" + "2" = "012"+ assert response == "012"assert len(chat._chat_history) == 2assert chat._chat_history[0].role == "user"assert chat._chat_history[0].content == "Hello"assert chat._chat_history[1].role == "assistant"- assert chat._chat_history[1].content == "2"+ assert chat._chat_history[1].content == "012"async def test_chat_streaming_sends_messages():@@ -147,11 +148,9 @@ async def test_chat_streaming_sends_messages():messages: list[ChatMessage], config: ChatModelConfig) -> AsyncIterator[str]:del config, messages- # Simulate streaming response- accumulated = ""- for word in ["Hello", "world", "!"]:- accumulated += word + " "- yield accumulated.strip()+ # Simulate streaming response with delta chunks (new content only)+ for word in ["Hello", " ", "world", " ", "!"]:+ yield wordchat = ui.chat(mock_streaming_model)@@ -171,7 +170,7 @@ async def test_chat_streaming_sends_messages():response: str = await chat._send_prompt(request)- # Verify final response+ # Verify final response (deltas accumulated)assert response == "Hello world !"# Verify streaming messages were sent@@ -199,11 +198,8 @@ async def test_chat_sync_generator_streaming():messages: list[ChatMessage], config: ChatModelConfig):del config, messages- # Simulate streaming response with sync generator- accumulated = ""- for word in ["Hello", "world", "!"]:- accumulated += word + " "- yield accumulated.strip()+ # Simulate streaming response with delta chunks (new content only)+ yield from ["Hello", " ", "world", " ", "!"]chat = ui.chat(mock_streaming_model)@@ -219,7 +215,7 @@ async def test_chat_sync_generator_streaming():response: str = await chat._send_prompt(request)- # Verify final response+ # Verify final response (deltas accumulated)assert response == "Hello world !"# Verify streaming messages were sent@@ -245,13 +241,11 @@ async def test_chat_streaming_complete_response():messages: list[ChatMessage], config: ChatModelConfig):del config, messages- # Simulate OpenAI-style streaming where last chunk might be empty- # This tests the bug where we might not yield the final accumulated value+ # Simulate delta-based streamingyield "Hello "- yield "Hello world"- yield "Hello world!"- # Simulate final chunk with no content (like finish_reason only)- # Generator continues but no more content - still need complete result+ yield "world"+ yield "!"+ # No more content - generator endschat = ui.chat(mock_streaming_model_with_empty_final)@@ -262,7 +256,7 @@ async def test_chat_streaming_complete_response():response: str = await chat._send_prompt(request)- # Verify we got the complete final response+ # Verify we got the complete final response (all deltas accumulated)assert response == "Hello world!"assert chat._chat_history[-1].content == "Hello world!"