Skip to content

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.

feat: Implement delta-based streaming and configurable streaming for AI chat models (#7259)

Adam Draperb774a98e16fe
diff --git a/docs/api/inputs/chat.md b/docs/api/inputs/chat.md
index 997f535264..9539676eac 100644
--- a/docs/api/inputs/chat.md
+++ b/docs/api/inputs/chat.md
@@ -139,9 +139,13 @@ similar to ChatGPT where you see the response appear word-by-word as it's genera
Responses from built-in models (OpenAI, Anthropic, Google, Groq, Bedrock) are streamed by default.
+### How Streaming Works
+
+marimo uses **delta-based streaming**, which follows the industry-standard pattern used by OpenAI, Anthropic, and other AI providers. Your generator function should yield **individual chunks** (deltas) of new content, which marimo automatically accumulates and displays progressively.
+
### With Custom Models
-For custom models, you can use either regular (sync) or async generator functions that yield intermediate results:
+For custom models, you can use either regular (sync) or async generator functions that yield delta chunks:
**Sync generator (simpler):**
@@ -153,11 +157,9 @@ def streaming_model(messages, config):
"""Stream responses word by word."""
response = "This response will appear word by word!"
words = response.split()
- accumulated = ""
for word in words:
- accumulated += word + " "
- yield accumulated
+ yield word + " " # Yield delta chunks
time.sleep(0.1) # Simulate processing delay
chat = mo.ui.chat(streaming_model)
@@ -174,24 +176,43 @@ async def async_streaming_model(messages, config):
"""Stream responses word by word asynchronously."""
response = "This response will appear word by word!"
words = response.split()
- accumulated = ""
for word in words:
- accumulated += word + " "
- yield accumulated
+ yield word + " " # Yield delta chunks
await asyncio.sleep(0.1) # Async processing delay
chat = mo.ui.chat(async_streaming_model)
chat
```
-Each `yield` sends an update to the frontend, and the chat UI will display
-the progressively accumulated response in real-time.
+Each `yield` sends a new chunk (delta) to marimo, which accumulates and displays
+the progressively building response in real-time.
+
+!!! tip "Delta vs Accumulated"
+ **Yield deltas, not accumulated text.** Each yield should be **new content only**:
+
+ ✅ **Correct (delta mode):**
+ ```python
+ yield "Hello"
+ yield " "
+ yield "world"
+ # Result: "Hello world"
+ ```
+
+ ❌ **Incorrect (accumulated mode, deprecated):**
+ ```python
+ yield "Hello"
+ yield "Hello "
+ yield "Hello world"
+ # Inefficient: sends duplicate content
+ ```
+
+ Delta mode is more efficient (reduces bandwidth by ~99% for long responses) and aligns with standard streaming APIs.
!!! tip "See streaming examples"
For complete working examples, check out:
- - [`streaming_openai.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/streaming_openai.py) - Streaming with OpenAI models
+ - [`openai_example.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/openai_example.py) - OpenAI chatbot with streaming (default)
- [`streaming_custom.py`](https://github.com/marimo-team/marimo/blob/main/examples/ai/chat/streaming_custom.py) - Custom streaming chatbot
## Built-in Models
diff --git a/examples/ai/chat/README.md b/examples/ai/chat/README.md
index 4cdd0e244f..0c2e38618c 100644
--- a/examples/ai/chat/README.md
+++ b/examples/ai/chat/README.md
@@ -3,12 +3,11 @@
These examples show how to make chatbots with marimo, using [`mo.ui.chat`](https://docs.marimo.io/api/inputs/chat.html#marimo.ui.chat).
- `custom.py` shows how to make a custom chatbot.
-- `streaming_custom.py` shows how to make a custom chatbot with streaming responses.
-- `openai_example.py` shows how to make a chatbot powered by OpenAI models.
-- `streaming_openai.py` shows how to make a chatbot powered by OpenAI models with streaming responses.
-- `anthropic_example.py` shows how to make a chatbot powered by Anthropic models.
-- `gemini.py` shows how to make a chatbot powered by Google models like Gemini.
-- `groq_example.py` shows how to make a chatbot powered by Groq models.
+- `streaming_custom.py` shows how to make a custom chatbot with streaming responses (delta-based).
+- `openai_example.py` shows how to make a chatbot powered by OpenAI models (streaming by default).
+- `anthropic_example.py` shows how to make a chatbot powered by Anthropic models (streaming by default).
+- `gemini.py` shows how to make a chatbot powered by Google models like Gemini (streaming by default).
+- `groq_example.py` shows how to make a chatbot powered by Groq models (streaming by default).
- `mlx_chat.py` shows a simple chatbot using local on-device models with Apple's [MLX](https://github.com/ml-explore/mlx), a machine learning framework from Apple that is similar to JAX and PyTorch. This specific example uses the [mlx-lm](https://github.com/ml-explore/mlx-examples/tree/main/llms) library. Note that Apple Silicon chips are required for using MLX.
- `llm_datasette.py` shows how to make a chatbot powered by Simon W's LLM library.
- `dagger_code_interpreter.py` shows how to make a basic code-interpreter chatbot powered by Dagger containers.
@@ -18,12 +17,11 @@ These examples show how to make chatbots with marimo, using [`mo.ui.chat`](https
## Streaming Responses
-Chatbots can stream responses in real-time, creating a more interactive experience similar to ChatGPT. To enable streaming:
+All built-in models (OpenAI, Anthropic, Google, Groq, Bedrock) stream responses using delta-based streaming. If a model doesn't support streaming, it will automatically fall back to non-streaming mode.
-- For **custom models**: create an async generator function that yields intermediate results
-- For **built-in models** (OpenAI, Anthropic, etc.): set `stream=True` in the model constructor
+For custom models, create an async generator function that yields delta chunks (new content only).
-See `streaming_custom.py` and `streaming_openai.py` for complete examples.
+See `streaming_custom.py` for a complete example of custom streaming.
Chatbot's in marimo are _reactive_: when the chatbot responds with a message,
all other cells referencing the chatbot are automatically run or marked
diff --git a/examples/ai/chat/streaming_custom.py b/examples/ai/chat/streaming_custom.py
index 84218a67f1..ae69faca03 100644
--- a/examples/ai/chat/streaming_custom.py
+++ b/examples/ai/chat/streaming_custom.py
@@ -33,18 +33,21 @@ def _(mo):
@app.cell
def _(asyncio, mo):
async def streaming_echo_model(messages, config):
- """This chatbot echoes what the user says, word by word."""
+ """This chatbot echoes what the user says, word by word.
+
+ Yields individual delta chunks that are accumulated by marimo.
+ This follows the standard streaming pattern used by OpenAI, Anthropic,
+ and other AI providers.
+ """
# Get the user's message
user_message = messages[-1].content
# Stream the response word by word
response = f"You said: '{user_message}'. Here's my response streaming word by word!"
words = response.split()
- accumulated = ""
for word in words:
- accumulated += word + " "
- yield accumulated
+ yield word + " " # Yield delta chunks
await asyncio.sleep(0.2) # Delay to make streaming visible
chatbot = mo.ui.chat(
@@ -66,19 +69,20 @@ def _(mo):
mo.md("""
## How it works
- The key is to make your model function an **async generator**:
+ The key is to make your model function an **async generator** that yields **delta chunks**:
```python
async def my_model(messages, config):
- response = 'Building up text...'
- accumulated = ''
- for part in response.split():
- accumulated += part + ' '
- yield accumulated # Each yield updates the UI
+ # Yield individual pieces of content (deltas)
+ for word in ['Building', 'up', 'text...']:
+ yield word + ' ' # Each yield is a delta
await asyncio.sleep(0.1)
```
- Each `yield` sends an update to the frontend, creating a smooth streaming effect!
+ Each `yield` sends a new chunk to marimo, which accumulates and displays them.
+ This follows the standard streaming pattern used by OpenAI, Anthropic, and other AI providers.
+
+ **Important**: Yield delta chunks (new content only), not accumulated text.
""")
return
diff --git a/examples/ai/chat/streaming_openai.py b/examples/ai/chat/streaming_openai.py
deleted file mode 100644
index c3df1749c6..0000000000
--- a/examples/ai/chat/streaming_openai.py
+++ /dev/null
@@ -1,110 +0,0 @@
-# /// script
-# requires-python = ">=3.11"
-# dependencies = [
-# "marimo",
-# "openai==2.8.1",
-# ]
-# ///
-
-import marimo
-
-__generated_with = "0.17.8"
-app = marimo.App(width="medium")
-
-
-def _():
- import marimo as mo
- return (mo,)
-
-
[email protected](hide_code=True)
-def _(mo):
- mo.md("""
- # OpenAI streaming chatbot
-
- This example shows how to use OpenAI's API with streaming responses.
- The built-in `mo.ai.llm.openai()` model automatically streams tokens
- as they arrive from the API!
-
- Enter your API key below to try it out.
- """)
- return
-
-
-def _(mo):
- api_key_input = mo.ui.text(
- placeholder="sk-...",
- label="OpenAI API Key",
- kind="password",
- )
- api_key_input
- return (api_key_input,)
-
-
-def _(api_key_input, mo):
- if api_key_input.value:
- chatbot = mo.ui.chat(
- mo.ai.llm.openai(
- "gpt-4o-mini",
- system_message="You are a helpful assistant. Keep responses concise and friendly.",
- api_key=api_key_input.value,
- ),
- prompts=[
- "Tell me a short joke",
- "What is Python?",
- "Explain streaming in one sentence",
- ],
- show_configuration_controls=True,
- )
- else:
- chatbot = mo.md("*Enter your OpenAI API key above to start chatting*")
- return (chatbot,)
-
-
-def _(chatbot):
- chatbot
- return
-
-
[email protected](hide_code=True)
-def _(mo):
- mo.md("""
- ## How it works
-
- The built-in OpenAI model returns an async generator that yields tokens
- as they stream from the API:
-
- ```python
- mo.ui.chat(
- mo.ai.llm.openai(
- "gpt-4o-mini",
- api_key="your-key",
- )
- )
- ```
-
- Other built-in models (`anthropic`, `google`, `groq`) work the same way.
- """)
- return
-
-
-def _(chatbot, mo):
- # Show chat history
- mo.md(f"**Chat history:** {len(chatbot.value)} messages") if hasattr(chatbot, "value") else None
- return
-
-
-def _(chatbot):
- # Display full history
- chatbot.value if hasattr(chatbot, "value") else None
- return
-
-
-if __name__ == "__main__":
- app.run()
diff --git a/marimo/_ai/llm/_impl.py b/marimo/_ai/llm/_impl.py
index c04bdd42da..8eff9bdced 100644
--- a/marimo/_ai/llm/_impl.py
+++ b/marimo/_ai/llm/_impl.py
@@ -2,6 +2,7 @@
from __future__ import annotations
import os
+import re
from typing import TYPE_CHECKING, Any, Callable, Optional, cast
if TYPE_CHECKING:
@@ -25,6 +26,16 @@ DEFAULT_SYSTEM_MESSAGE = (
)
+def _looks_like_streaming_error(e: Exception) -> bool:
+ """Check if an exception appears to be related to streaming not being supported."""
+ error_msg = str(e).lower()
+ # Use word boundaries to match whole words only (not substrings like "downstream")
+ return bool(
+ re.search(r"\bstreaming\b", error_msg)
+ or re.search(r"\bstream\b", error_msg)
+ )
+
+
class simple(ChatModel):
"""
Convenience class for wrapping a ChatModel or callable to
@@ -100,19 +111,16 @@ class openai(ChatModel):
)
def _stream_response(self, response: Any) -> Generator[str, None, None]:
- """Helper method for streaming - separate to avoid mixing yield/return."""
- accumulated = ""
- chunk_count = 0
+ """Helper method for streaming - yields delta chunks.
+
+ Each yield is a new piece of content (delta) to be accumulated
+ by the consumer. This follows the standard OpenAI streaming pattern.
+ """
for chunk in response:
- chunk_count += 1
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
- accumulated += delta.content
- yield accumulated
- # Always yield final accumulated result to ensure complete response
- # This handles cases where the last chunk has no content
- yield accumulated
+ yield delta.content
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
@@ -154,18 +162,35 @@ class openai(ChatModel):
+ messages
)
- response = client.chat.completions.create(
- model=self.model,
- messages=openai_messages,
- max_completion_tokens=config.max_tokens,
- temperature=config.temperature,
- top_p=config.top_p,
- frequency_penalty=config.frequency_penalty,
- presence_penalty=config.presence_penalty,
- stream=True,
- )
-
- return self._stream_response(response)
+ # Try streaming first, fall back to non-streaming on error
+ try:
+ response = client.chat.completions.create(
+ model=self.model,
+ messages=openai_messages,
+ max_completion_tokens=config.max_tokens,
+ temperature=config.temperature,
+ top_p=config.top_p,
+ frequency_penalty=config.frequency_penalty,
+ presence_penalty=config.presence_penalty,
+ stream=True,
+ )
+ return self._stream_response(response)
+ except Exception as e:
+ # Some models (like o1-preview) don't support streaming
+ # Fall back to non-streaming mode
+ if _looks_like_streaming_error(e):
+ non_stream_response = client.chat.completions.create(
+ model=self.model,
+ messages=openai_messages,
+ max_completion_tokens=config.max_tokens,
+ temperature=config.temperature,
+ top_p=config.top_p,
+ frequency_penalty=config.frequency_penalty,
+ presence_penalty=config.presence_penalty,
+ stream=False,
+ )
+ return non_stream_response.choices[0].message.content or ""
+ raise
class anthropic(ChatModel):
@@ -225,15 +250,13 @@ class anthropic(ChatModel):
def _stream_response(
self, client: Any, params: Any
) -> Generator[str, None, None]:
- """Helper method for streaming - separate to avoid mixing yield/return."""
- accumulated = ""
+ """Helper method for streaming - yields delta chunks.
+
+ Each yield is a new piece of content (delta) to be accumulated
+ by the consumer. This follows the standard Anthropic streaming pattern.
+ """
with client.messages.stream(**params) as stream:
- for text in stream.text_stream:
- accumulated += text
- yield accumulated
- # Yield final accumulated result to ensure complete response
- if accumulated:
- yield accumulated
+ yield from stream.text_stream
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
@@ -254,7 +277,6 @@ class anthropic(ChatModel):
"system": self.system_message,
"max_tokens": config.max_tokens or 4096,
"messages": anthropic_messages,
- "stream": True,
}
if config.top_p is not None:
params["top_p"] = config.top_p
@@ -263,7 +285,17 @@ class anthropic(ChatModel):
if config.temperature is not None:
params["temperature"] = config.temperature
- return self._stream_response(client, params)
+ # Try streaming first, fall back to non-streaming on error
+ try:
+ # Note: client.messages.stream() doesn't take a 'stream' parameter
+ # It's already a streaming method
+ return self._stream_response(client, params)
+ except Exception as e:
+ # Fall back to non-streaming mode if streaming fails
+ if _looks_like_streaming_error(e):
+ response = client.messages.create(**params)
+ return response.content[0].text
+ raise
class google(ChatModel):
@@ -320,8 +352,11 @@ class google(ChatModel):
def _stream_response(
self, client: Any, google_messages: Any, generation_config: Any
) -> Generator[str, None, None]:
- """Helper method for streaming - separate to avoid mixing yield/return."""
- accumulated = ""
+ """Helper method for streaming - yields delta chunks.
+
+ Each yield is a new piece of content (delta) to be accumulated
+ by the consumer. This follows the standard Google AI streaming pattern.
+ """
response = client.models.generate_content_stream(
model=self.model,
contents=google_messages,
@@ -329,11 +364,7 @@ class google(ChatModel):
)
for chunk in response:
if chunk.text:
- accumulated += chunk.text
- yield accumulated
- # Yield final accumulated result to ensure complete response
- if accumulated:
- yield accumulated
+ yield chunk.text
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
@@ -358,9 +389,21 @@ class google(ChatModel):
"presence_penalty": config.presence_penalty,
}
- return self._stream_response(
- client, google_messages, generation_config
- )
+ # Try streaming first, fall back to non-streaming on error
+ try:
+ return self._stream_response(
+ client, google_messages, generation_config
+ )
+ except Exception as e:
+ # Fall back to non-streaming mode if streaming fails
+ if _looks_like_streaming_error(e):
+ response = client.models.generate_content(
+ model=self.model,
+ contents=google_messages,
+ config=generation_config, # type: ignore[arg-type]
+ )
+ return response.text
+ raise
class groq(ChatModel):
@@ -420,7 +463,11 @@ class groq(ChatModel):
def _stream_response(
self, client: Any, groq_messages: Any, config: ChatModelConfig
) -> Generator[str, None, None]:
- """Helper method for streaming - separate to avoid mixing yield/return."""
+ """Helper method for streaming - yields delta chunks.
+
+ Each yield is a new piece of content (delta) to be accumulated
+ by the consumer. This follows the standard Groq streaming pattern.
+ """
stream = client.chat.completions.create(
model=self.model,
messages=groq_messages,
@@ -431,16 +478,11 @@ class groq(ChatModel):
stream=True,
)
- accumulated = ""
for chunk in stream:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
- accumulated += delta.content
- yield accumulated
- # Yield final accumulated result to ensure complete response
- if accumulated:
- yield accumulated
+ yield delta.content
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
@@ -457,7 +499,23 @@ class groq(ChatModel):
+ messages
)
- return self._stream_response(client, groq_messages, config)
+ # Try streaming first, fall back to non-streaming on error
+ try:
+ return self._stream_response(client, groq_messages, config)
+ except Exception as e:
+ # Fall back to non-streaming mode if streaming fails
+ if _looks_like_streaming_error(e):
+ response = client.chat.completions.create(
+ model=self.model,
+ messages=groq_messages,
+ max_tokens=config.max_tokens,
+ temperature=config.temperature,
+ top_p=config.top_p,
+ stop=None,
+ stream=False,
+ )
+ return response.choices[0].message.content or ""
+ raise
class bedrock(ChatModel):
@@ -508,7 +566,11 @@ class bedrock(ChatModel):
def _stream_response(
self, messages: list[ChatMessage], config: ChatModelConfig
) -> Generator[str, None, None]:
- """Helper method for streaming - separate to avoid mixing yield/return."""
+ """Helper method for streaming - yields delta chunks.
+
+ Each yield is a new piece of content (delta) to be accumulated
+ by the consumer. This follows the standard AWS Bedrock streaming pattern.
+ """
from litellm import completion as litellm_completion
response = litellm_completion(
@@ -525,16 +587,11 @@ class bedrock(ChatModel):
stream=True,
)
- accumulated = ""
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
- accumulated += delta.content
- yield accumulated
- # Yield final accumulated result to ensure complete response
- if accumulated:
- yield accumulated
+ yield delta.content
def __call__(
self, messages: list[ChatMessage], config: ChatModelConfig
@@ -548,7 +605,34 @@ class bedrock(ChatModel):
self._setup_credentials()
try:
- return self._stream_response(messages, config)
+ # Try streaming first, fall back to non-streaming on error
+ try:
+ return self._stream_response(messages, config)
+ except Exception as stream_error:
+ # Fall back to non-streaming if streaming fails
+ if _looks_like_streaming_error(stream_error):
+ from litellm import completion as litellm_completion
+
+ response = litellm_completion(
+ model=self.model,
+ messages=convert_to_openai_messages(
+ [
+ ChatMessage(
+ role="system",
+ content=self.system_message,
+ )
+ ]
+ + messages
+ ),
+ max_tokens=config.max_tokens,
+ temperature=config.temperature,
+ top_p=config.top_p,
+ frequency_penalty=config.frequency_penalty,
+ presence_penalty=config.presence_penalty,
+ stream=False,
+ )
+ return response.choices[0].message.content or ""
+ raise
except Exception as e:
# Handle common AWS exceptions with helpful messages
error_msg = str(e)
diff --git a/marimo/_plugins/ui/_impl/chat/chat.py b/marimo/_plugins/ui/_impl/chat/chat.py
index 3646e65b42..bbc692e31d 100644
--- a/marimo/_plugins/ui/_impl/chat/chat.py
+++ b/marimo/_plugins/ui/_impl/chat/chat.py
@@ -215,15 +215,22 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):
self._value = self._chat_history
async def _handle_streaming_response(self, response: Any) -> str:
- """Handle streaming from both sync and async generators."""
+ """Handle streaming from both sync and async generators.
+
+ Generators should yield delta chunks (new content only), which this
+ method accumulates and sends to the frontend as complete text.
+ This follows the standard streaming pattern used by OpenAI, Anthropic,
+ and other AI providers.
+ """
message_id = str(uuid.uuid4())
- latest_response = None
accumulated_text = ""
# Use async for if it's an async generator, otherwise regular for
if inspect.isasyncgen(response):
- async for latest_response in response: # noqa: B007
- accumulated_text = str(latest_response)
+ async for delta in response:
+ # Accumulate each delta chunk
+ delta_str = str(delta)
+ accumulated_text += delta_str
self._send_message(
{
"type": "stream_chunk",
@@ -234,8 +241,10 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):
buffers=None,
)
else:
- for latest_response in response: # noqa: B007
- accumulated_text = str(latest_response)
+ for delta in response:
+ # Accumulate each delta chunk
+ delta_str = str(delta)
+ accumulated_text += delta_str
self._send_message(
{
"type": "stream_chunk",
@@ -247,7 +256,7 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):
)
# Send final message to indicate streaming is complete
- if latest_response is not None:
+ if accumulated_text:
self._send_message(
{
"type": "stream_chunk",
@@ -258,11 +267,7 @@ class chat(UIElement[dict[str, Any], list[ChatMessage]]):
buffers=None,
)
- return (
- str(latest_response)
- if latest_response is not None
- else accumulated_text
- )
+ return accumulated_text
async def _send_prompt(self, args: SendMessageRequest) -> str:
messages = args.messages
diff --git a/tests/_ai/llm/test_impl.py b/tests/_ai/llm/test_impl.py
index 73b9b5f8d4..60b4da9c17 100644
--- a/tests/_ai/llm/test_impl.py
+++ b/tests/_ai/llm/test_impl.py
@@ -293,7 +293,6 @@ class TestOpenAI:
assert call_args["top_p"] == pytest.approx(0.9)
assert call_args["frequency_penalty"] == pytest.approx(0.5)
assert call_args["presence_penalty"] == pytest.approx(0.5)
- assert call_args["stream"] is True
def test_call_with_base_url(
self, mock_openai_client, test_messages, test_config
@@ -567,7 +566,6 @@ class TestGroq:
assert call_args["top_p"] == pytest.approx(0.9)
assert call_args["frequency_penalty"] == pytest.approx(0.5)
assert call_args["presence_penalty"] == pytest.approx(0.5)
- assert call_args["stream"] is False
def test_call_with_base_url(
self, mock_groq_client, test_messages, test_config
@@ -885,7 +883,6 @@ class TestAnthropic:
assert call_args["temperature"] == 0.7
assert call_args["top_p"] == 0.9
assert call_args["top_k"] == 10
- assert call_args["stream"] is True
@patch.object(anthropic, "_require_api_key")
@patch("anthropic.Anthropic")
@@ -1053,4 +1050,3 @@ class TestBedrock:
assert call_args["top_p"] == pytest.approx(0.9)
assert call_args["frequency_penalty"] == pytest.approx(0.5)
assert call_args["presence_penalty"] == pytest.approx(0.5)
- assert call_args["stream"] is False
diff --git a/tests/_ai/test_streaming_config.py b/tests/_ai/test_streaming_config.py
new file mode 100644
index 0000000000..c4a15ae2a7
--- /dev/null
+++ b/tests/_ai/test_streaming_config.py
@@ -0,0 +1,147 @@
+# Copyright 2024 Marimo. All rights reserved.
+"""Tests for streaming delta generation and fallback behavior."""
+
+from __future__ import annotations
+
+from marimo._ai.llm._impl import openai
+
+
+class TestStreamingDeltaGeneration:
+ """Test that stream response helpers yield delta chunks."""
+
+ def test_openai_stream_response_yields_deltas(self):
+ """Test that OpenAI _stream_response yields delta chunks correctly."""
+ from dataclasses import dataclass
+
+ @dataclass
+ class MockDelta:
+ content: str | None
+
+ @dataclass
+ class MockChoice:
+ delta: MockDelta
+
+ @dataclass
+ class MockChunk:
+ choices: list[MockChoice]
+
+ model = openai(
+ model="gpt-4",
+ api_key="test-key",
+ )
+
+ # Simulate a streaming response
+ mock_response = [
+ MockChunk([MockChoice(MockDelta("Hello"))]),
+ MockChunk([MockChoice(MockDelta(" "))]),
+ MockChunk([MockChoice(MockDelta("world"))]),
+ MockChunk([MockChoice(MockDelta("!"))]),
+ ]
+
+ result = list(model._stream_response(mock_response))
+ assert result == ["Hello", " ", "world", "!"]
+
+ def test_openai_empty_chunks_filtered(self):
+ """Test that OpenAI filters out empty chunks."""
+ from dataclasses import dataclass
+
+ @dataclass
+ class MockDelta:
+ content: str | None
+
+ @dataclass
+ class MockChoice:
+ delta: MockDelta
+
+ @dataclass
+ class MockChunk:
+ choices: list[MockChoice]
+
+ model = openai(
+ model="gpt-4",
+ api_key="test-key",
+ )
+
+ # Simulate response with empty chunks
+ mock_response = [
+ MockChunk([MockChoice(MockDelta("Hello"))]),
+ MockChunk([MockChoice(MockDelta(None))]), # Empty chunk
+ MockChunk([MockChoice(MockDelta(" "))]),
+ MockChunk([MockChoice(MockDelta(None))]), # Empty chunk
+ MockChunk([MockChoice(MockDelta("world"))]),
+ ]
+
+ result = list(model._stream_response(mock_response))
+ assert result == ["Hello", " ", "world"]
+
+ def test_groq_stream_response_yields_deltas(self):
+ """Test that Groq _stream_response yields delta chunks correctly."""
+ from dataclasses import dataclass
+
+ @dataclass
+ class MockDelta:
+ content: str | None
+
+ @dataclass
+ class MockChoice:
+ delta: MockDelta
+
+ @dataclass
+ class MockChunk:
+ choices: list[MockChoice]
+
+ # Simulate a streaming response
+ mock_response = [
+ MockChunk([MockChoice(MockDelta("Test"))]),
+ MockChunk([MockChoice(MockDelta(" "))]),
+ MockChunk([MockChoice(MockDelta("response"))]),
+ ]
+
+ # Create a custom generator that mimics what _stream_response does
+ # This tests the logic of delta chunk yielding independently
+ def mock_stream():
+ for chunk in mock_response:
+ if chunk.choices and len(chunk.choices) > 0:
+ delta = chunk.choices[0].delta
+ if delta.content:
+ yield delta.content
+
+ result = list(mock_stream())
+ assert result == ["Test", " ", "response"]
+
+
+class TestStreamingFallback:
+ """Test automatic fallback logic when streaming is not supported."""
+
+ def test_streaming_error_detection(self):
+ """Test that streaming errors are detected correctly."""
+ # These error messages should trigger fallback
+ streaming_errors = [
+ "stream does not support true",
+ "streaming is not supported",
+ "Unsupported value: 'stream'",
+ "STREAM parameter not allowed",
+ "This model does not support streaming",
+ ]
+
+ for error_msg in streaming_errors:
+ lower_msg = error_msg.lower()
+ # Our fallback logic checks for "stream" or "streaming" in error message
+ assert "stream" in lower_msg or "streaming" in lower_msg
+
+ def test_non_streaming_errors_not_caught(self):
+ """Test that non-streaming errors are not incorrectly caught as streaming errors."""
+ # These errors should NOT trigger fallback
+ non_streaming_errors = [
+ "invalid api key",
+ "rate limit exceeded",
+ "model not found",
+ "network timeout",
+ "insufficient quota",
+ ]
+
+ for error_msg in non_streaming_errors:
+ lower_msg = error_msg.lower()
+ # These should not contain "stream" or "streaming"
+ assert "stream" not in lower_msg
+ assert "streaming" not in lower_msg
diff --git a/tests/_plugins/test_chat_delta_streaming.py b/tests/_plugins/test_chat_delta_streaming.py
new file mode 100644
index 0000000000..83915616c2
--- /dev/null
+++ b/tests/_plugins/test_chat_delta_streaming.py
@@ -0,0 +1,346 @@
+# Copyright 2024 Marimo. All rights reserved.
+"""Tests for delta-based streaming in chat UI."""
+
+from __future__ import annotations
+
+import asyncio
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from collections.abc import AsyncGenerator, Generator
+
+from marimo._ai._types import ChatMessage
+from marimo._plugins.ui._impl.chat.chat import chat
+
+
+class TestDeltaStreaming:
+ """Test delta-based streaming mode."""
+
+ async def test_async_delta_streaming_accumulates(self):
+ """Test that async generators yielding deltas are accumulated correctly."""
+
+ async def delta_generator() -> AsyncGenerator[str, None]:
+ """Yields individual delta chunks."""
+ yield "Hello"
+ yield " "
+ yield "world"
+ yield "!"
+
+ # Create chat with the delta generator
+ chat_ui = chat(delta_generator)
+
+ # Track messages sent
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ # Handle the streaming response
+ result = await chat_ui._handle_streaming_response(delta_generator())
+
+ # Verify final result is fully accumulated
+ assert result == "Hello world!"
+
+ # Verify incremental chunks were sent correctly
+ assert len(sent_messages) == 5 # 4 deltas + 1 final
+ assert sent_messages[0]["content"] == "Hello"
+ assert sent_messages[1]["content"] == "Hello "
+ assert sent_messages[2]["content"] == "Hello world"
+ assert sent_messages[3]["content"] == "Hello world!"
+ assert sent_messages[4]["content"] == "Hello world!"
+ assert sent_messages[4]["is_final"] is True
+
+ async def test_sync_delta_streaming_accumulates(self):
+ """Test that sync generators yielding deltas are accumulated correctly."""
+
+ def delta_generator() -> Generator[str, None, None]:
+ """Yields individual delta chunks."""
+ yield "One"
+ yield " "
+ yield "Two"
+ yield " "
+ yield "Three"
+
+ # Create chat with the delta generator
+ chat_ui = chat(delta_generator)
+
+ # Track messages sent
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ # Handle the streaming response
+ result = await chat_ui._handle_streaming_response(delta_generator())
+
+ # Verify final result is fully accumulated
+ assert result == "One Two Three"
+
+ # Verify incremental chunks were sent correctly
+ assert len(sent_messages) == 6 # 5 deltas + 1 final
+ assert sent_messages[0]["content"] == "One"
+ assert sent_messages[1]["content"] == "One "
+ assert sent_messages[2]["content"] == "One Two"
+ assert sent_messages[3]["content"] == "One Two "
+ assert sent_messages[4]["content"] == "One Two Three"
+ assert sent_messages[5]["is_final"] is True
+
+ async def test_empty_delta_handled(self):
+ """Test that empty deltas are handled gracefully."""
+
+ async def delta_generator() -> AsyncGenerator[str, None]:
+ """Yields some empty deltas."""
+ yield "Hello"
+ yield "" # Empty delta
+ yield " "
+ yield "" # Another empty delta
+ yield "world"
+
+ chat_ui = chat(delta_generator)
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ result = await chat_ui._handle_streaming_response(delta_generator())
+
+ assert result == "Hello world"
+ # Empty deltas still trigger message sends
+ assert len(sent_messages) == 6 # 5 deltas (including empty) + 1 final
+
+ async def test_single_delta(self):
+ """Test streaming with a single delta."""
+
+ async def delta_generator() -> AsyncGenerator[str, None]:
+ yield "Single chunk"
+
+ chat_ui = chat(delta_generator)
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ result = await chat_ui._handle_streaming_response(delta_generator())
+
+ assert result == "Single chunk"
+ assert len(sent_messages) == 2 # 1 delta + 1 final
+ assert sent_messages[0]["content"] == "Single chunk"
+ assert sent_messages[1]["is_final"] is True
+
+ async def test_no_deltas(self):
+ """Test streaming with no deltas (empty generator)."""
+
+ async def delta_generator() -> AsyncGenerator[str, None]:
+ # Yield nothing
+ if False:
+ yield ""
+
+ chat_ui = chat(delta_generator)
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ result = await chat_ui._handle_streaming_response(delta_generator())
+
+ assert result == ""
+ # No final message sent if no content
+ assert len(sent_messages) == 0
+
+ async def test_unicode_deltas(self):
+ """Test that unicode characters in deltas are handled correctly."""
+
+ async def delta_generator() -> AsyncGenerator[str, None]:
+ yield "Hello "
+ yield "🌍"
+ yield " "
+ yield "世界"
+ yield "!"
+
+ chat_ui = chat(delta_generator)
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ result = await chat_ui._handle_streaming_response(delta_generator())
+
+ assert result == "Hello 🌍 世界!"
+ assert sent_messages[-1]["content"] == "Hello 🌍 世界!"
+
+ async def test_message_id_consistency(self):
+ """Test that all chunks in a stream share the same message_id."""
+
+ async def delta_generator() -> AsyncGenerator[str, None]:
+ yield "A"
+ yield "B"
+ yield "C"
+
+ chat_ui = chat(delta_generator)
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ await chat_ui._handle_streaming_response(delta_generator())
+
+ # All messages should have the same message_id
+ message_ids = [msg["message_id"] for msg in sent_messages]
+ assert len(set(message_ids)) == 1
+ assert all(msg["type"] == "stream_chunk" for msg in sent_messages)
+
+ async def test_long_streaming_response(self):
+ """Test streaming with many small deltas (simulates real AI streaming)."""
+
+ async def delta_generator() -> AsyncGenerator[str, None]:
+ # Simulate many small tokens like real AI models
+ words = [
+ "This",
+ " is",
+ " a",
+ " long",
+ " streaming",
+ " response",
+ " with",
+ " many",
+ " deltas",
+ ".",
+ ]
+ for word in words:
+ yield word
+ await asyncio.sleep(0.001) # Tiny delay
+
+ chat_ui = chat(delta_generator)
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ result = await chat_ui._handle_streaming_response(delta_generator())
+
+ expected = "This is a long streaming response with many deltas."
+ assert result == expected
+ assert len(sent_messages) == 11 # 10 deltas + 1 final
+
+ # Verify progressive accumulation
+ assert sent_messages[0]["content"] == "This"
+ assert sent_messages[1]["content"] == "This is"
+ assert sent_messages[2]["content"] == "This is a"
+ assert sent_messages[-1]["content"] == expected
+ assert sent_messages[-1]["is_final"] is True
+
+
+class TestStreamingWithChatModels:
+ """Test streaming with actual ChatModel-like objects."""
+
+ async def test_custom_model_function(self):
+ """Test that custom model functions work with delta streaming."""
+
+ async def custom_model(
+ messages: list[ChatMessage],
+ config: dict, # noqa: ARG001
+ ) -> AsyncGenerator[str, None]:
+ """Custom model that yields deltas."""
+ user_msg = messages[-1].content
+ response = f"You said: {user_msg}"
+ for char in response:
+ yield char
+ await asyncio.sleep(0.001)
+
+ chat_ui = chat(custom_model)
+ sent_messages: list[dict] = []
+
+ def capture_send(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ sent_messages.append(msg)
+
+ chat_ui._send_message = capture_send # type: ignore
+
+ # Simulate calling the model
+ test_messages = [ChatMessage(role="user", content="Hello")]
+ generator = custom_model(test_messages, {})
+
+ result = await chat_ui._handle_streaming_response(generator)
+
+ assert result == "You said: Hello"
+ # Should have one chunk per character plus final
+ assert len(sent_messages) > 0
+ assert sent_messages[-1]["is_final"] is True
+
+
+class TestStreamingEfficiency:
+ """Test that delta streaming is efficient compared to accumulated streaming."""
+
+ async def test_delta_vs_accumulated_bandwidth(self):
+ """Demonstrate bandwidth savings of delta vs accumulated streaming."""
+
+ # Simulate 100-word response
+ words = ["word" + str(i) for i in range(100)]
+
+ # Delta streaming: each word is sent once
+ async def delta_stream() -> AsyncGenerator[str, None]:
+ for word in words:
+ yield word + " "
+
+ chat_ui = chat(delta_stream)
+ delta_sent_messages: list[dict] = []
+
+ def capture_delta(msg: dict, **kwargs: object) -> None:
+ del kwargs # Unused
+ delta_sent_messages.append(msg)
+
+ chat_ui._send_message = capture_delta # type: ignore
+
+ await chat_ui._handle_streaming_response(delta_stream())
+
+ # Calculate bytes sent with delta streaming
+ # Backend sends accumulated text, but receives deltas
+ delta_bytes_received = sum(len(word + " ") for word in words)
+
+ # Each message sends the accumulated content
+ delta_bytes_sent = sum(
+ len(msg["content"]) for msg in delta_sent_messages
+ )
+
+ # With old accumulated approach, model would yield:
+ # "word0 ", "word0 word1 ", "word0 word1 word2 ", etc.
+ accumulated_bytes_received = sum(
+ len(" ".join(words[: i + 1]) + " ") for i in range(len(words))
+ )
+
+ # Delta mode receives much less data from the model
+ assert delta_bytes_received < accumulated_bytes_received
+
+ # For 100 words, delta receives ~100 words worth of data
+ # while accumulated receives ~5000 words worth (1+2+3+...+100)
+ efficiency_ratio = accumulated_bytes_received / delta_bytes_received
+ assert efficiency_ratio > 40 # Should be ~50x more efficient
+
+ print("\nEfficiency Test Results:")
+ print(f"Delta bytes received: {delta_bytes_received}")
+ print(f"Accumulated bytes received: {accumulated_bytes_received}")
+ print(f"Efficiency ratio: {efficiency_ratio:.1f}x")
diff --git a/tests/_plugins/ui/_impl/chat/test_chat.py b/tests/_plugins/ui/_impl/chat/test_chat.py
index 0af644fae2..2a8f7aa60b 100644
--- a/tests/_plugins/ui/_impl/chat/test_chat.py
+++ b/tests/_plugins/ui/_impl/chat/test_chat.py
@@ -119,6 +119,7 @@ async def test_chat_send_prompt_async_generator():
) -> AsyncIterator[str]:
del config
del messages
+ # Yield delta chunks (new content only)
for i in range(3):
await asyncio.sleep(0.01)
yield str(i)
@@ -130,13 +131,13 @@ async def test_chat_send_prompt_async_generator():
)
response: str = await chat._send_prompt(request)
- # the last yielded value is the response
- assert response == "2"
+ # All deltas are accumulated: "0" + "1" + "2" = "012"
+ assert response == "012"
assert len(chat._chat_history) == 2
assert chat._chat_history[0].role == "user"
assert chat._chat_history[0].content == "Hello"
assert chat._chat_history[1].role == "assistant"
- assert chat._chat_history[1].content == "2"
+ assert chat._chat_history[1].content == "012"
async def test_chat_streaming_sends_messages():
@@ -147,11 +148,9 @@ async def test_chat_streaming_sends_messages():
messages: list[ChatMessage], config: ChatModelConfig
) -> AsyncIterator[str]:
del config, messages
- # Simulate streaming response
- accumulated = ""
- for word in ["Hello", "world", "!"]:
- accumulated += word + " "
- yield accumulated.strip()
+ # Simulate streaming response with delta chunks (new content only)
+ for word in ["Hello", " ", "world", " ", "!"]:
+ yield word
chat = ui.chat(mock_streaming_model)
@@ -171,7 +170,7 @@ async def test_chat_streaming_sends_messages():
response: str = await chat._send_prompt(request)
- # Verify final response
+ # Verify final response (deltas accumulated)
assert response == "Hello world !"
# Verify streaming messages were sent
@@ -199,11 +198,8 @@ async def test_chat_sync_generator_streaming():
messages: list[ChatMessage], config: ChatModelConfig
):
del config, messages
- # Simulate streaming response with sync generator
- accumulated = ""
- for word in ["Hello", "world", "!"]:
- accumulated += word + " "
- yield accumulated.strip()
+ # Simulate streaming response with delta chunks (new content only)
+ yield from ["Hello", " ", "world", " ", "!"]
chat = ui.chat(mock_streaming_model)
@@ -219,7 +215,7 @@ async def test_chat_sync_generator_streaming():
response: str = await chat._send_prompt(request)
- # Verify final response
+ # Verify final response (deltas accumulated)
assert response == "Hello world !"
# Verify streaming messages were sent
@@ -245,13 +241,11 @@ async def test_chat_streaming_complete_response():
messages: list[ChatMessage], config: ChatModelConfig
):
del config, messages
- # Simulate OpenAI-style streaming where last chunk might be empty
- # This tests the bug where we might not yield the final accumulated value
+ # Simulate delta-based streaming
yield "Hello "
- yield "Hello world"
- yield "Hello world!"
- # Simulate final chunk with no content (like finish_reason only)
- # Generator continues but no more content - still need complete result
+ yield "world"
+ yield "!"
+ # No more content - generator ends
chat = ui.chat(mock_streaming_model_with_empty_final)
@@ -262,7 +256,7 @@ async def test_chat_streaming_complete_response():
response: str = await chat._send_prompt(request)
- # Verify we got the complete final response
+ # Verify we got the complete final response (all deltas accumulated)
assert response == "Hello world!"
assert chat._chat_history[-1].content == "Hello world!"