Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.
Python aiburrchatbot-frameworkdagsgenerative-aigraphshacktoberfestllmopsllmsmlopspersistent-data-structurestate-machinestate-managementvisibility
Fleshes out web-server example in repository
elijahbenizzydf87b72845fb
diff --git a/docs/examples/web-server.rst b/docs/examples/web-server.rstnew file mode 100644index 00000000..782c9f81--- /dev/null+++ b/docs/examples/web-server.rst@@ -0,0 +1,6 @@+========================+Exposing in a Web Server+========================++Burr makes it natural to integrate with a web-server such as FastAPI. See the [example in the repository](https://github.com/DAGWorks-Inc/burr/tree/main/examples/web-server)+for the high-level concepts + some implementation tricks, and a fully-working, runnable demo:diff --git a/examples/email-assistant/server.py b/examples/email-assistant/server.pyindex 2d06320b..1a8ada65 100644--- a/examples/email-assistant/server.py+++ b/examples/email-assistant/server.py@@ -57,6 +57,8 @@ class EmailAssistantState(pydantic.BaseModel):@functools.lru_cache(maxsize=128)def _get_application(project_id: str, app_id: str) -> Application:+ """Utility to get the application. Depending on+ your use-case you might want to reload from state every time (or consider cache invalidation)"""app = email_assistant_application.application(app_id=app_id, project=project_id)return appdiff --git a/examples/web-server/README.md b/examples/web-server/README.mdindex 1794a08b..584928e6 100644--- a/examples/web-server/README.md+++ b/examples/web-server/README.md@@ -1,34 +1,223 @@# Running Burr in a web server-We're working on fleshing out this demo, but we have an example to get started with.+Burr is meant to run interactive apps. This means running it as part of a web-service that+responds to requests, maanges state, and documents its capabilities. The interactive nature of Burr+(moving in/out of programmatic control) means we want to think carefully about how to expose+our burr applications to the web.-You can play around with this example by running the `burr` command after running `pip install`.-Then navigate to [examples/chatbot](http://localhost:7241/demos/chatbot) in your browser (it will open up).+In this tutorial we will use the [email assistant example](../email-assistant) as a walk-through.+Our goal is to expose the email assistant in a web-server that a UI can easily be built on top of.+While we will not be building the UI here, we will link out to the final product for you to explore.++## Email Assistant++The email assistant is an example of a "human-in-the-loop" generative AI application. This means that+it requires human assistance at multiple points to build a better product.++### Running the example++If you want to get a sense for how this looks, open the burr UI:```bash-pip install burr[start]+pip install "burr[start]"burr```-You'll see the telemetry on the right and the chatbot on the left.+Then navigate to the [email assistant](http://localhost:7241/demos/email-assistant),++You can create a new "application" and see it run through, with the telemetry on the right side.++### Conceptual Model+At a high-level, the email assistant does the following:++1. Accepts an email + instructions to respond+2. Comes up with a set of clarifying questions (if the LLM deems it required)+3. Using the answer to those questions, generates a draft+4. Accepts feedback to that draft and generates another one, repeating until the user is happy+5. Returns the final draft++Due to the stochastic, often complex nature of LLMs, this has been shown to be one of the most promising+applications -- a collaboration between humans and AI to quickly build high-quality responses.++### Modeling with Burr+This is a brief overview, for a more in-depth look at the email assistant, see the [email assistant example](../email-assistant).+To model our email assistant with burr, we can use the following diagram:++++There are three points at which the user can interact:+1. `process_input`: This is where the user inputs the email and instructions+2. `clarify_instructions`: The LLM has created a set of clarification questions+3. `process_feedback`: The user has provided feedback on the draft++(3) repeats until the user is happy with the draft (in our implementation this occurs when the feedback they provide is empty)++Recall that we use the word "application" in Burr to refer to an instance of this process above+(with persisted state).++You can see the full application in [application.py](../email-assistant/application.py).++## Integrating in a web server++For this example we will use [FastApi](https://fastapi.tiangolo.com/) and [pydantic](https://docs.pydantic.dev/latest/),+but it should work with any other web-stack that uses python.++### Endpoints++We construct the following endpoints:++1. `POST /create`: This will create a new application and return the ID+2. `PUT /initialize_draft/{id}/`: This calls out to `process_input`, passing in the email and instructions+3. `PUT /clarify_instructions/{id}`: This will gives answers back to the LLM+4. `PUT /process_feedback/{id}`: This will give feedback back to the LLM+5. `GET /{id}/state`: This will return the current state of the application++The `GET` endpoint allows us to get the current state of the application -- this enables+the user to reload if they quit the browser/get distracted. Each of these endpoints will return the full state of the application,+which can be rendered on the frontend. Furthermore, it will indicate the next API endpoint+we call, which allows the UI to render the appropriate form and++Using FastAPI + Pydantic, this becomes very simple to implement. First, let's add a utility to+get the `application` object. This will use a cached version or instantiate it:++```python[email protected]_cache(maxsize=128)+def _get_application(app_id: str) -> Application:+ app = email_assistant_application.application(app_id=app_id)+ return app+```++All this does is call to our function `application` in `email_assistant` that+recreates the application. We have not included the `create` function here,+but it calls out to the same API.++### Data Model++Let's then define a pydantic model to represent the state, and the app object in FastAPI:+```python++class EmailAssistantState(pydantic.BaseModel):+ app_id: str+ email_to_respond: Optional[str]+ response_instructions: Optional[str]+ questions: Optional[List[str]]+ answers: Optional[List[str]]+ drafts: List[str]+ feedback_history: List[str]+ final_draft: Optional[str]+ # This stores the next step, which tells the frontend which ones to call+ next_step: Literal["process_input", "clarify_instructions", "process_feedback", None]++ @staticmethod+ def from_app(app: Application):+ # implementation left out, call app.state and translate to pydantic model+ # we can use `app.get_next_action()` to get the next step and return it to the user+ ...+```++### Execution++Next, we can run through to the next step, starting from any point:++```python+def _run_through(project_id: str, app_id: Optional[str], inputs: Dict[str, Any]) -> EmailAssistantState:+ email_assistant_app = _get_application(project_id, app_id)+ email_assistant_app.run( # Using this as a side-effect, we'll just get the state aft+ halt_before=["clarify_instructions", "process_feedback"],+ halt_after=["final_result"],+ inputs=inputs,+ )+ return EmailAssistantState.from_app(email_assistant_app)+```++We `halt_before` the steps that require user instructions, and `halt_after`+the final result. This allows us to get the state after each step.++Finally, we can define our endpoints. For instance:++```python[email protected]("/provide_feedback/{id}")+def provide_feedback(project_id: str, app_id: str, feedback: Feedback) -> EmailAssistantState:+ return _run_through(project_id, app_id, dict(feedback=feedback.feedback))+```++This represents a simple but powerful architecture. We can continue calling these endpoints+until we're at a "terminal" state, at which point we can always ask for the state.+If we decide to add more input steps, we can modify the state machine and add more input steps.+We are not required to hold state in the app (it is all delegated to Burr's persistence),+so we can easily load up from any given point, allowing the user to wait for seconds,+minutes, hours, or even days before continuing.++As the frontend simply renders based on the current state and the next step, it will always+be correct, and the user can always pick up where they left off. With Burr's telemetry capabilities+they can debug any state-related issues, ensuring a smooth user experience.++### Persistence++Note that we never called out to databases. It all just magically worked.. This is all because we decouple the persistence+layer from the web-call. The application will be persisted (to whatever database you want),+by burr's plugin capabilities -- read more [here](https://burr.dagworks.io/concepts/state-persistence/).+This greatly reduces the amount you have to think about when developing. As Burr persistence is+pluggable, you can write to your own database with whichever schema you prefer, customizing+the schema for your project or using a generic one (state is just a JSON object -- you can easily serialize/deseriealize it).++### Additional concerns++#### Scaling++But [is this webscale](https://www.youtube.com/watch?v=b2F-DItXtZs)? As anything, it depends on how you implement it.+Two factors determine the scalability of this system:++1. database layer -- can the database support the volume of inputs/outputs?+2. compute layer -- can the server run fast enough to keep up with the users?++For the database layer, it depends largely on the underlying database, as well as the+schema you use. That said, Burr makes it easier due to natural partitioning of the data+into `application_id` and `partition_key` (the latter could be the `user ID`), allowing common+operations (such as _give me all applications for X user_ and _give me the state of application Y_)+simple if you index your state table on the application ID and `partition_key`.++For the compute layer, you can simply scale horizontally. The only tricky aspect is ensuring state synchronization+and locking. As we cached the application object, we could potentially get into a position+in which the state is out of sync. To solve this, you can either:++1. Use a locking method (either in the database) to ensure that only one server is running a given application at any point+2. Use sticky sessions/sharding to ensure that a given user always hits the same server++Or possibly both.++#### Async++While we implemented synchronous calls, you can easily make these async by using `async def` and `await` in the appropriate places,+and using the `arun` method in Burr. Read more about async capabilities in [applications](https://burr.dagworks.io/concepts/state-machine/),+and [actions](https://burr.dagworks.io/concepts/actions/).++#### Streaming++You can use streaming to send back the stream of the output at any given point. You do this by creating a+[streaming action](https://burr.dagworks.io/concepts/streaming-actions/). You can then integrate with the+streaming respose in FastAPI to send back the stream of the output. You can do this with any steps+(intermediate or final) in your application.-## General strategy+#### Authentication/Data access-The general approach for a read/write app (say, a chatbot) is as follows:-1. Instantiate a burr application using the builder whenever you need it, loading the state from your persistence layer-2. Use the burr application object to handle incoming requests-3. Return the full state result from the burr application object to the client-4. Render that on the frontend+While Burr does not operate at the data access layer, this can be easily handles at the application layer.+Any authentication system will tell you the user ID, which you can look in your DB to determine access+to your partition key.-This way you can keep the frontend simple (just render burr's state), and keep the backend simple (delegate-loading/persistence/logic to burr). Then you can view in telemetry.+## Wrapup-## Code+In this tutorial we showed how to integrate Burr into a web-server. We used FastAPI and Pydantic+to create a simple but powerful API that allows users to interact with the email assistant, leveraging+Burr's persistence capabilities to ensure that the user can always pick up where they left off.-To see the code, you'll want to open the following:+At a high-level, the real value of representing your application as a state machine (as Burr does)+is that it all becomes easier to think about. You don't have to conceptually model state persistence,+dataflow, and the web infrastructure in one piece -- they can all be built separately.-- [application file](../multi-modal-chatbot/application.py) - The main application file. This defines the Burr logic-- [fastAPI app](../../tracking/server/examples/chatbot.py) - The server file. This defines the FastAPI app+In the future we will be automating this process, allowing you to generate a FastAPI app from the Burr application.-You can see that the server file defines a rounter which gets imported by the main server file -- this-allows us to reuse this in the demo. We'll be updating this with a self-contained copy/paste example shortly, stay tuned!+For now though, you can find the resources for the current implementation:+- [application.py](../email-assistant/application.py)+- [server.py](../email-assistant/server.py)+- [ui](../../telemetry/ui/src/examples/EmailAssistant.tsx) -- this uses [ReactQuery](https://tanstack.com/query/latest/docs/framework/react/overview) to call the API and [react](https://react.dev/) to render the state.