Skip to content

armgit / burr

Public Imported from GitHub
0

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

Fleshes out web-server example in repository

elijahbenizzydf87b72845fb
diff --git a/docs/examples/web-server.rst b/docs/examples/web-server.rst
new file mode 100644
index 00000000..782c9f81
--- /dev/null
+++ b/docs/examples/web-server.rst
@@ -0,0 +1,6 @@
+========================
+Exposing in a Web Server
+========================
+
+Burr makes it natural to integrate with a web-server such as FastAPI. See the [example in the repository](https://github.com/DAGWorks-Inc/burr/tree/main/examples/web-server)
+for the high-level concepts + some implementation tricks, and a fully-working, runnable demo:
diff --git a/examples/email-assistant/server.py b/examples/email-assistant/server.py
index 2d06320b..1a8ada65 100644
--- a/examples/email-assistant/server.py
+++ b/examples/email-assistant/server.py
@@ -57,6 +57,8 @@ class EmailAssistantState(pydantic.BaseModel):
@functools.lru_cache(maxsize=128)
def _get_application(project_id: str, app_id: str) -> Application:
+ """Utility to get the application. Depending on
+ your use-case you might want to reload from state every time (or consider cache invalidation)"""
app = email_assistant_application.application(app_id=app_id, project=project_id)
return app
diff --git a/examples/web-server/README.md b/examples/web-server/README.md
index 1794a08b..584928e6 100644
--- a/examples/web-server/README.md
+++ b/examples/web-server/README.md
@@ -1,34 +1,223 @@
# Running Burr in a web server
-We're working on fleshing out this demo, but we have an example to get started with.
+Burr is meant to run interactive apps. This means running it as part of a web-service that
+responds to requests, maanges state, and documents its capabilities. The interactive nature of Burr
+(moving in/out of programmatic control) means we want to think carefully about how to expose
+our burr applications to the web.
-You can play around with this example by running the `burr` command after running `pip install`.
-Then navigate to [examples/chatbot](http://localhost:7241/demos/chatbot) in your browser (it will open up).
+In this tutorial we will use the [email assistant example](../email-assistant) as a walk-through.
+Our goal is to expose the email assistant in a web-server that a UI can easily be built on top of.
+While we will not be building the UI here, we will link out to the final product for you to explore.
+
+## Email Assistant
+
+The email assistant is an example of a "human-in-the-loop" generative AI application. This means that
+it requires human assistance at multiple points to build a better product.
+
+### Running the example
+
+If you want to get a sense for how this looks, open the burr UI:
```bash
-pip install burr[start]
+pip install "burr[start]"
burr
```
-You'll see the telemetry on the right and the chatbot on the left.
+Then navigate to the [email assistant](http://localhost:7241/demos/email-assistant),
+
+You can create a new "application" and see it run through, with the telemetry on the right side.
+
+### Conceptual Model
+At a high-level, the email assistant does the following:
+
+1. Accepts an email + instructions to respond
+2. Comes up with a set of clarifying questions (if the LLM deems it required)
+3. Using the answer to those questions, generates a draft
+4. Accepts feedback to that draft and generates another one, repeating until the user is happy
+5. Returns the final draft
+
+Due to the stochastic, often complex nature of LLMs, this has been shown to be one of the most promising
+applications -- a collaboration between humans and AI to quickly build high-quality responses.
+
+### Modeling with Burr
+This is a brief overview, for a more in-depth look at the email assistant, see the [email assistant example](../email-assistant).
+To model our email assistant with burr, we can use the following diagram:
+
+![Modeling](../email-assistant/statemachine.png)
+
+There are three points at which the user can interact:
+1. `process_input`: This is where the user inputs the email and instructions
+2. `clarify_instructions`: The LLM has created a set of clarification questions
+3. `process_feedback`: The user has provided feedback on the draft
+
+(3) repeats until the user is happy with the draft (in our implementation this occurs when the feedback they provide is empty)
+
+Recall that we use the word "application" in Burr to refer to an instance of this process above
+(with persisted state).
+
+You can see the full application in [application.py](../email-assistant/application.py).
+
+## Integrating in a web server
+
+For this example we will use [FastApi](https://fastapi.tiangolo.com/) and [pydantic](https://docs.pydantic.dev/latest/),
+but it should work with any other web-stack that uses python.
+
+### Endpoints
+
+We construct the following endpoints:
+
+1. `POST /create`: This will create a new application and return the ID
+2. `PUT /initialize_draft/{id}/`: This calls out to `process_input`, passing in the email and instructions
+3. `PUT /clarify_instructions/{id}`: This will gives answers back to the LLM
+4. `PUT /process_feedback/{id}`: This will give feedback back to the LLM
+5. `GET /{id}/state`: This will return the current state of the application
+
+The `GET` endpoint allows us to get the current state of the application -- this enables
+the user to reload if they quit the browser/get distracted. Each of these endpoints will return the full state of the application,
+which can be rendered on the frontend. Furthermore, it will indicate the next API endpoint
+we call, which allows the UI to render the appropriate form and
+
+Using FastAPI + Pydantic, this becomes very simple to implement. First, let's add a utility to
+get the `application` object. This will use a cached version or instantiate it:
+
+```python
[email protected]_cache(maxsize=128)
+def _get_application(app_id: str) -> Application:
+ app = email_assistant_application.application(app_id=app_id)
+ return app
+```
+
+All this does is call to our function `application` in `email_assistant` that
+recreates the application. We have not included the `create` function here,
+but it calls out to the same API.
+
+### Data Model
+
+Let's then define a pydantic model to represent the state, and the app object in FastAPI:
+```python
+
+class EmailAssistantState(pydantic.BaseModel):
+ app_id: str
+ email_to_respond: Optional[str]
+ response_instructions: Optional[str]
+ questions: Optional[List[str]]
+ answers: Optional[List[str]]
+ drafts: List[str]
+ feedback_history: List[str]
+ final_draft: Optional[str]
+ # This stores the next step, which tells the frontend which ones to call
+ next_step: Literal["process_input", "clarify_instructions", "process_feedback", None]
+
+ @staticmethod
+ def from_app(app: Application):
+ # implementation left out, call app.state and translate to pydantic model
+ # we can use `app.get_next_action()` to get the next step and return it to the user
+ ...
+```
+
+### Execution
+
+Next, we can run through to the next step, starting from any point:
+
+```python
+def _run_through(project_id: str, app_id: Optional[str], inputs: Dict[str, Any]) -> EmailAssistantState:
+ email_assistant_app = _get_application(project_id, app_id)
+ email_assistant_app.run( # Using this as a side-effect, we'll just get the state aft
+ halt_before=["clarify_instructions", "process_feedback"],
+ halt_after=["final_result"],
+ inputs=inputs,
+ )
+ return EmailAssistantState.from_app(email_assistant_app)
+```
+
+We `halt_before` the steps that require user instructions, and `halt_after`
+the final result. This allows us to get the state after each step.
+
+Finally, we can define our endpoints. For instance:
+
+```python
[email protected]("/provide_feedback/{id}")
+def provide_feedback(project_id: str, app_id: str, feedback: Feedback) -> EmailAssistantState:
+ return _run_through(project_id, app_id, dict(feedback=feedback.feedback))
+```
+
+This represents a simple but powerful architecture. We can continue calling these endpoints
+until we're at a "terminal" state, at which point we can always ask for the state.
+If we decide to add more input steps, we can modify the state machine and add more input steps.
+We are not required to hold state in the app (it is all delegated to Burr's persistence),
+so we can easily load up from any given point, allowing the user to wait for seconds,
+minutes, hours, or even days before continuing.
+
+As the frontend simply renders based on the current state and the next step, it will always
+be correct, and the user can always pick up where they left off. With Burr's telemetry capabilities
+they can debug any state-related issues, ensuring a smooth user experience.
+
+### Persistence
+
+Note that we never called out to databases. It all just magically worked.. This is all because we decouple the persistence
+layer from the web-call. The application will be persisted (to whatever database you want),
+by burr's plugin capabilities -- read more [here](https://burr.dagworks.io/concepts/state-persistence/).
+This greatly reduces the amount you have to think about when developing. As Burr persistence is
+pluggable, you can write to your own database with whichever schema you prefer, customizing
+the schema for your project or using a generic one (state is just a JSON object -- you can easily serialize/deseriealize it).
+
+### Additional concerns
+
+#### Scaling
+
+But [is this webscale](https://www.youtube.com/watch?v=b2F-DItXtZs)? As anything, it depends on how you implement it.
+Two factors determine the scalability of this system:
+
+1. database layer -- can the database support the volume of inputs/outputs?
+2. compute layer -- can the server run fast enough to keep up with the users?
+
+For the database layer, it depends largely on the underlying database, as well as the
+schema you use. That said, Burr makes it easier due to natural partitioning of the data
+into `application_id` and `partition_key` (the latter could be the `user ID`), allowing common
+operations (such as _give me all applications for X user_ and _give me the state of application Y_)
+simple if you index your state table on the application ID and `partition_key`.
+
+For the compute layer, you can simply scale horizontally. The only tricky aspect is ensuring state synchronization
+and locking. As we cached the application object, we could potentially get into a position
+in which the state is out of sync. To solve this, you can either:
+
+1. Use a locking method (either in the database) to ensure that only one server is running a given application at any point
+2. Use sticky sessions/sharding to ensure that a given user always hits the same server
+
+Or possibly both.
+
+#### Async
+
+While we implemented synchronous calls, you can easily make these async by using `async def` and `await` in the appropriate places,
+and using the `arun` method in Burr. Read more about async capabilities in [applications](https://burr.dagworks.io/concepts/state-machine/),
+and [actions](https://burr.dagworks.io/concepts/actions/).
+
+#### Streaming
+
+You can use streaming to send back the stream of the output at any given point. You do this by creating a
+[streaming action](https://burr.dagworks.io/concepts/streaming-actions/). You can then integrate with the
+streaming respose in FastAPI to send back the stream of the output. You can do this with any steps
+(intermediate or final) in your application.
-## General strategy
+#### Authentication/Data access
-The general approach for a read/write app (say, a chatbot) is as follows:
-1. Instantiate a burr application using the builder whenever you need it, loading the state from your persistence layer
-2. Use the burr application object to handle incoming requests
-3. Return the full state result from the burr application object to the client
-4. Render that on the frontend
+While Burr does not operate at the data access layer, this can be easily handles at the application layer.
+Any authentication system will tell you the user ID, which you can look in your DB to determine access
+to your partition key.
-This way you can keep the frontend simple (just render burr's state), and keep the backend simple (delegate
-loading/persistence/logic to burr). Then you can view in telemetry.
+## Wrapup
-## Code
+In this tutorial we showed how to integrate Burr into a web-server. We used FastAPI and Pydantic
+to create a simple but powerful API that allows users to interact with the email assistant, leveraging
+Burr's persistence capabilities to ensure that the user can always pick up where they left off.
-To see the code, you'll want to open the following:
+At a high-level, the real value of representing your application as a state machine (as Burr does)
+is that it all becomes easier to think about. You don't have to conceptually model state persistence,
+dataflow, and the web infrastructure in one piece -- they can all be built separately.
-- [application file](../multi-modal-chatbot/application.py) - The main application file. This defines the Burr logic
-- [fastAPI app](../../tracking/server/examples/chatbot.py) - The server file. This defines the FastAPI app
+In the future we will be automating this process, allowing you to generate a FastAPI app from the Burr application.
-You can see that the server file defines a rounter which gets imported by the main server file -- this
-allows us to reuse this in the demo. We'll be updating this with a self-contained copy/paste example shortly, stay tuned!
+For now though, you can find the resources for the current implementation:
+- [application.py](../email-assistant/application.py)
+- [server.py](../email-assistant/server.py)
+- [ui](../../telemetry/ui/src/examples/EmailAssistant.tsx) -- this uses [ReactQuery](https://tanstack.com/query/latest/docs/framework/react/overview) to call the API and [react](https://react.dev/) to render the state.