Skip to content

armgit / burr

Public Imported from GitHub
0

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

Adds documentation for streaming actions

elijahbenizzy0ed5b1b58a34
diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst
index e117f0b2..361aeaa2 100644
--- a/docs/concepts/index.rst
+++ b/docs/concepts/index.rst
@@ -15,5 +15,6 @@ Overview of the concepts -- read these to get a mental model for how Burr works.
transitions
hooks
tracking
+ streaming-actions
additional-visibility
planned-capabilities
diff --git a/docs/concepts/streaming-actions.rst b/docs/concepts/streaming-actions.rst
new file mode 100644
index 00000000..e8de5d88
--- /dev/null
+++ b/docs/concepts/streaming-actions.rst
@@ -0,0 +1,91 @@
+=================
+Streaming Actions
+=================
+
+.. _streaming_actions:
+
+Actions can be implemented as streaming results. This enables a lower time-to-first-token and a more interactive
+interface in the case of AI applications or streaming in of metrics in a model-training application. Broadly,
+this is a tool to enable quicker user interaction in longer running actions that require user focus.
+
+Streaming actions can be implemented as a class or a function, just like actions. However, they have a few additional rules:
+
+#. They give intermediate results to the user as they are generated in the form of a generator
+#. They are responsible for determining the relationship between the intermediate results and the final result
+#. They have a ``return`` statement that handles result collection and state update
+
+If you're not familiar with ``return`` statements in generators, you can read about them `here <https://www.python.org/dev/peps/pep-0380/>`_.
+The high-level idea is that the ``return`` statement is used to raise a ``StopIteration`` exception with a value that is returned to the caller.
+The framework uses this value to update the state of the action and to collect the final result.
+
+A class-based streaming action might look like this:
+
+.. code-block:: python
+
+ from burr.core.action import StreamingAction
+
+ class StreamingChatCall(StreamingAction):
+ def stream_run(self, state: State, **run_kwargs) -> Generator[dict, None, dict]:
+ client = openai.Client()
+ response = client.chat.completions.create(
+ model='gpt-3.5-turbo',
+ messages=[{
+ 'role': 'user',
+ 'content': state["prompt"]
+ }],
+ temperature=0,
+ stream=True,
+ )
+ buffer = []
+ for chunk in response:
+ delta = chunk.choices[0].delta.content
+ buffer.append(delta)
+ yield {'response': delta}
+ full_response = ''.join(buffer)
+ return {'response': full_response}
+
+ @property
+ def reads(self) -> list[str]:
+ return ["prompt"]
+
+ @property
+ def writes(self) -> list[str]:
+ return ["response"]
+
+ def update(self, result: dict, state: State) -> State:
+ return state.append(response=result["response"])
+
+The logic is split between ``stream_run``, which is responsible for generating the intermediate results and
+joining them into the final result, and update, which is responsible for collecting the final result and
+updating the state. The ``return`` statement in ``stream_run`` is used to return the final result to the framework,
+which is passed to ``update``.
+
+The function-based equivalent would look very similar:
+
+.. code-block:: python
+
+ from burr.core.action import streaming_action
+
+ @streaming_action(reads=["prompt"], writes=["prompt"])
+ def streaming_chat_call(state: State, **run_kwargs) -> Generator[dict, None, Tuple[dict, State]]:
+ client = openai.Client()
+ response = client.chat.completions.create(
+ model='gpt-3.5-turbo',
+ messages=[{
+ 'role': 'user',
+ 'content': state["prompt"]
+ }],
+ temperature=0,
+ stream=True,
+ )
+ buffer = []
+ for chunk in response:
+ delta = chunk.choices[0].delta.content
+ buffer.append(delta)
+ yield {'response': delta}
+ full_response = ''.join(buffer)
+ return {'response': full_response}, state.append(response=full_response)
+
+As you can see above, we're doing the same thing, with a bit of syntactic sugar to combine the ``update`` and ``return`` statements.
+
+Currently ``Async`` streaming actions are not supported. We will be adding shortly -- stay tuned! Follow the `issue on github <https://github.com/DAGWorks-Inc/burr/issues/64>`_ for more information.
diff --git a/docs/reference/actions.rst b/docs/reference/actions.rst
index 2ca668e5..8afeea46 100644
--- a/docs/reference/actions.rst
+++ b/docs/reference/actions.rst
@@ -31,3 +31,12 @@ Actions
.. autoclass:: burr.core.action.Reducer
:members:
+
+.. autoclass:: burr.core.action.StreamingResultContainer
+ :members:
+
+.. autoclass:: burr.core.action.streaming_action
+ :members:
+
+.. autoclass:: burr.core.action.StreamingAction
+ :members: