Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.
Typed State
This example goes over how to use the typed state features in Burr with pydantic.
It will cover the following concepts:
- Why you might want to use typing for your state in the first place
- IDE setup to make use of typing.
- Defining/altering state at the action level
- Defining types for streaming contexts
- Wiring that through to a FastAPI app
This README will contain snippets + link out to the code.
This adapts the instructor + youtube example. This takes in a youtube video, creates a transcript, and uses OpenAI to generate a social media post based on that transcript.
Why type state?
Burr originally came without typing -- typing is an additional (optional) plugin that requires some complexity. So, why use typing at all? And why use Pydantic? Lots of good reasons:
Typing provides guard-rails
Typing state ensures you have guarentees about what the data in your state is, and what shape it takes. This makes it easier to work with/manipulate.
Typing makes development easier
IDEs have integrations for handling type annotations, enabling you to avoid the cognitive burden of tracking types in your head. You get auto-completion on typed classes, and errors if you attempt to access or assign to a field that does not exist.
Typing makes downstream integration easier
Multiple tools use types (especially with pydantic) to make interacting with data easier. In this example we use instructor, as well as FastAPI to leverage pydantic models that Burr also uses.
Typing provides a form of documentation
Type-annotation in python allows you to read your code and get some sense of what it is actually doing. This can be a warm introduction to python for those who came from the world of java initially (the authors of this library included), and make reasoning about a complex codebase simpler.
Setting up your IDE
VSCode (or an editor with a similar interface) is generally optimal for this. It has pluggable typing (e.g. pylance), which handles generics cleanly. Note: pycharm is often behind on typing support. See issues like this and this.
While it will still work in pycharm, you will not get some of the better auto-completion capabilities until those issues are resolved.
Defining typed state at the application level
The code for this is in application.py.
First, define a pydantic model -- make it as nested/recursive as you want. This will represent your entire state. In this case, we're going to have a transcript of a youtube video that was given by the user, as well as the social media post. The high-level is here -- the rest is in the code:
class ApplicationState(BaseModel):
# Make these have defaults as they are only set in actions
transcript: Optional[str] = Field(
description="The full transcript of the YouTube video.", default=None
)
post: Optional[SocialMediaPost] = Field(
description="The generated social media post.", default=None
)
Note that this should exactly model your state -- we need to make things optional, as there are points in time when running your application where we have not populated all fields in the application state. For our example the transcript/post will not have been assigned.
Next, we add it in to the application object, both the initial value(s) and the typing system. The typing system is what's responsible for managing the schema:
app = (
ApplicationBuilder()
.with_actions(
...
)
.with_transitions(
...
)
.with_entrypoint(...)
.with_typing(PydanticTypingSystem(ApplicationState))
.with_state(ApplicationState())
.build()
)
That ensures that application and the application's state object are parameterized on the state type.
To get the typed state object you defined, you can use .data on any state object returned by the application:
# just from the application
print(app.state.data.transcript)
# after execution
_, _, state = app.run(halt_after=..., inputs=...)
print(state.data.transcript)
Defining/altering typed state at the action level
The code for this is in application.py.
In addition to defining state centrally, we can define it at an action level.
The code is straightforward, but the API is slightly different from standard Burr. Rather than using the immutable state-based API, we in-place mutate pydantic models in the action function. Don't worry, the state object you are modifying is still immutable, you're just modifying a copy and returning it.
In this case, we use @action.pydantic, which tells that we're using typed state and which fields to read/write to from state. It derives your typed state class(es) from the function annotations, although you can also pass it the pydantic classes as arguments to the decorator if you prefer.
Note that the reads/writes have to be a subset of your defined state object -- fields that exist in your pydantic model.
This is because the action will pull all its data from state -- it listens to specific sub-fields.
In this case we use the global ApplicationState object as described above, although it can use a subset/compatible set of fields (or, if you elect not to use centralized state, it just has to be compatible with upstream/downstream versions).
Under the hood, burr will subset the state class so it only has the relevant fields (the reads/write) fields.
@action.pydantic(reads=["transcript"], writes=["post"])
def generate_post(state: ApplicationState, llm_client) -> ApplicationState:
"""Use the Instructor LLM client to generate `SocialMediaPost` from the YouTube transcript."""
# read the transcript from state
transcript = state.transcript
response = llm_client.chat.completions.create(
model="gpt-4o-mini",
response_model=SocialMediaPost,
messages=[...]
)
# mutate in place
state.post = response
# return the state
return state
Typed state for streaming actions
The code for this is in application.py.
For streaming actions, not only can we type the input/output state, but we can also type the intermediate result.
In this case, we just use the SocialMediaPost as we did in the application state. Instructor will be streaming that in as it gets created.
@streaming_action.pydantic currently requires you to pass in all the pydantic models as classes, although we will be adding the option to derive from the function signature.
We first call out to OpenAI, then we stream through
@streaming_action.pydantic(
reads=["transcript"],
writes=["post"],
state_input_type=ApplicationState,
state_output_type=ApplicationState,
stream_type=SocialMediaPost,
)
def generate_post_streaming(
state: ApplicationState, llm_client
) -> Generator[Tuple[SocialMediaPost, Optional[ApplicationState]], None, None]:
"""Streams a post as it's getting created. This allows for interacting data on the UI side of partial
results, using instructor's streaming capabilities for partial responses:
https://python.useinstructor.com/concepts/partial/
:param state: input state -- of the shape `ApplicationState`
:param llm_client: the LLM client, we will bind this in the application
:yield: a tuple of the post and the state -- state will be non-null when it's done
"""
transcript = state.transcript
response = llm_client.chat.completions.create_partial(
model="gpt-4o-mini",
response_model=SocialMediaPost,
messages=[...],
stream=True,
)
for post in response:
yield post, None
state.post = post
yield post, state
When we call out to the application we built, we have to add a type-hint to get typing to work
in the IDE (see the line streaming_container: StreamingResultContainer[...]), but we still have the same benefits as the non-streaming approach.
app = build_streaming_application(...) # builder left out for now
_, streaming_container = app.stream_result(
halt_after=["generate_post"],
inputs={"youtube_url": "https://www.youtube.com/watch?v=hqutVJyd3TI"},
)
# annotate to make type-completion easier
streaming_container: StreamingResultContainer[ApplicationState, SocialMediaPost]
# post is of type SocialMediaPost
for post in streaming_container:
obj = post.model_dump()
console.clear()
console.print(obj)
FastAPI integration
The code for this is in server.py.
To integrate this with FastAPI is easy, and gets easier with the types cascading through.
Non-streaming
For the non-streaming case, we declare an endpoint that returns the entire state. Note you may want a subset, but for now this is simple as it matches the pydantic models we defined above.
@app.get("/social_media_post", response_model=SocialMediaPost)
def social_media_post(youtube_url: str = DEFAULT_YOUTUBE_URL) -> SocialMediaPost:
_, _, state = burr_app.run(halt_after=["generate_post"], inputs={"youtube_url": youtube_url})
return state.data.post
Streaming
The streaming case involves using FastAPI's StreamingResponse API. We define a generator, which simply yields all intermediate results:
@app.get("/social_media_post_streaming", response_class=StreamingResponse)
def social_media_post_streaming(youtube_url: str = DEFAULT_YOUTUBE_URL) -> StreamingResponse:
"""Creates a completion for the chat message"""
def gen():
_, streaming_container = burr_app_streaming.stream_result(
halt_after=["generate_post"],
inputs={"youtube_url": youtube_url},
) # type: ignore
for post in streaming_container:
obj = post.model_dump()
yield json.dumps(obj)
return StreamingResponse(gen())
Note that StreamingResponse is not typed, but you have access to the types with the post
object, which corresponds to the stream from above!
Async streaming is similar.
You can run server.py with python server.py, which will open up on port 7443. You can use the ./curls.sh command to query the server (it will use a default video, modify to pass your own):
./curls.sh # default, non-streaming
./curls.sh streaming # streaming endpoint
./sucls.sh streaming_async # streaming async endpoint
Note you'll have to have jq installed for this to work.
Caveats + next steps
Some things we'll be building out shortly:
- The ability to derive application level schemas from individual actions
- The ability to automatically generate a FastAPI application from state + Burr
- Configurable validation for state -- guardrails to choose when/when not to validate in pydantic