Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.
Python aiburrchatbot-frameworkdagsgenerative-aigraphshacktoberfestllmopsllmsmlopspersistent-data-structurestate-machinestate-managementvisibility
`example` added an OpenAI api compatible server (#241)
Thierry Jean47478197aca6
diff --git a/README.md b/README.mdindex 57baf9d4..af459fd3 100644--- a/README.md+++ b/README.md@@ -158,9 +158,9 @@ Users who have contributed core functionality, integrations, or examples.- [Stefan Krawczyk](https://github.com/skrawcz)- [Joseph Booth](https://github.com/jombooth)- [Nandani Thakur](https://github.com/NandaniThakur)+- [Thierry Jean](https://github.com/zilto)### Bug hunters/special mentionsUsers who have contributed small docs fixes, design suggestions, and found bugs-- [Thierry Jean](https://github.com/zilto)- [Luke Chadwick](https://github.com/vertis)- [Evans](https://github.com/sudoevans)diff --git a/examples/openai-compatible-agent/README.md b/examples/openai-compatible-agent/README.mdnew file mode 100644index 00000000..7489c7f3--- /dev/null+++ b/examples/openai-compatible-agent/README.md@@ -0,0 +1,34 @@+# Burr with an OpenAI API -compatible server++Connect to your Burr agent using your favorite open-source tooling! This includes:+- Local chat frontends such as [Ollama](https://github.com/ollama/ollama/blob/4ec7445a6f678b6efc773bb9fa886d7c9b075577/docs/openai.md#supported-request-fields), [Jan](https://jan.ai/docs/remote-models/generic-openai), [text-generation-webui](https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#openai-compatible-api)+- Code assistants like [Continue](https://docs.continue.dev/reference/Model%20Providers/openai#openai-compatible-servers--apis), [llm-vscode](https://github.com/huggingface/llm-vscode?tab=readme-ov-file#backend) by HuggingFace+- Other productivity tools like the [llm](https://llm.datasette.io/en/stable/other-models.html#openai-compatible-models) command line tool++## Context++The [OpenAI API](https://platform.openai.com/docs/overview) allows you to send HTTP requests to large language models (LLMs) such as GPT-4. When interacting with ChatGPT, we're using the API endpoint `v1/chat/completions`, but there are many others:++- `v1/embeddings` get an embedding, i.e., a vector representation of the input text+- `v1/audio/transcriptions` uses the Whisper model to convert audio to text+- `v1/audio/speech` convert text to audio+- `v1/images/generations` uses DALL-E to generate images from a text prompt++Other LLM providers (e.g., Cohere, HuggingFace) have their own set of endpoints. But given the influence of OpenAI, many open-source tools include a "OpenAI API-compatible" version. By creating a server that implements endpoints respecting the request and response formats, we can directly interface with them!++## OpenAI API compatible Burr application+This example contains a very simple Burr application (`application.py`) and a FastAPI server to deploy this agent behind the OpenAI `v1/chat/completions` endpoint. After starting the server with `server.py`, you should be able to interact with it from your other tools ([Jan](https://jan.ai/docs) is easy and quick to install across platforms).++++This is great because we can quickly integrate our Burr Agent with high-quality UIs and tools. Simulaneously, you gain Burr's observability, logging, and persistence across your applications.++Most tools save state "in the frontend" because they don't have access to the official OpenAI backend. This means that each of your LLM applications are isolated. By using Burr and persisting state (e.g., chat history) on the backend, this means you can share and resume a conversation between your LLM frontend, CLI, or coding assistant!++## Resources+There are multiple implementations of OpenAI API-compatible servers. Here are some notable examples:++- [cortex](https://github.com/janhq/cortex)+- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python/blob/main/llama_cpp/server/app.py)+- [FastChat](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py)+- [text-generation-webui](https://github.com/oobabooga/text-generation-webui/blob/abe5ddc8833206381c43b002e95788d4cca0893a/extensions/openai/script.py)diff --git a/examples/openai-compatible-agent/__init__.py b/examples/openai-compatible-agent/__init__.pynew file mode 100644index 00000000..e69de29bdiff --git a/examples/openai-compatible-agent/application.py b/examples/openai-compatible-agent/application.pynew file mode 100644index 00000000..403c3c24--- /dev/null+++ b/examples/openai-compatible-agent/application.py@@ -0,0 +1,38 @@+import datetime++import burr.core+from burr.core import Application, State+from burr.core.action import action+++@action(reads=[], writes=[])+def dummy_bot(state: State, user_input: str):+ if "time" in user_input:+ current_time = datetime.datetime.now()+ reply = f"It is currently {current_time}"+ else:+ reply = "🤖 Ask me about the time"++ results = dict(content=reply)+ return results, state.update(**results)+++def build_application() -> Application:+ return (+ burr.core.ApplicationBuilder()+ .with_actions(dummy_bot)+ .with_transitions(("dummy_bot", "dummy_bot"))+ .with_identifiers(app_id="burr-openai")+ .with_entrypoint("dummy_bot")+ .build()+ )+++if __name__ == "__main__":+ app = build_application()+ app.visualize(+ output_file_path="statemachine",+ include_conditions=False,+ view=True,+ format="png",+ )diff --git a/examples/openai-compatible-agent/requirements.txt b/examples/openai-compatible-agent/requirements.txtnew file mode 100644index 00000000..e8071972--- /dev/null+++ b/examples/openai-compatible-agent/requirements.txt@@ -0,0 +1,4 @@+burr[start]+fastapi+openai+uvicorndiff --git a/examples/openai-compatible-agent/server.py b/examples/openai-compatible-agent/server.pynew file mode 100644index 00000000..5cbde794--- /dev/null+++ b/examples/openai-compatible-agent/server.py@@ -0,0 +1,68 @@+import logging+import time+import uuid+from contextlib import asynccontextmanager+from typing import Optional++import application as my_agent+import fastapi+import uvicorn+from openai.types.chat import ChatCompletion, ChatCompletionMessage+from openai.types.chat.chat_completion import Choice++from burr.core import Application++logger = logging.getLogger(__name__)+++burr_app: Optional[Application] = None+++def get_burr_app() -> Application:+ """Retrieve the global Burr app."""+ if burr_app is None:+ raise RuntimeError("Burr app wasn't instantiated.")+ return burr_app+++@asynccontextmanager+async def lifespan(app: fastapi.FastAPI):+ """Instantiate the Burr application on FastAPI startup."""+ global burr_app+ burr_app = my_agent.build_application()+ yield+++app = fastapi.FastAPI(lifespan=lifespan)++[email protected]("/v1/chat/completions")+async def create_chat_completion(+ request: fastapi.Request, burr_app: Application = fastapi.Depends(get_burr_app)+):+ """Creates a completion for the chat message"""+ request_json = await request.json()+ latest_message = request_json["messages"][-1]["content"]++ _, result, _ = burr_app.run(halt_after=["dummy_bot"], inputs={"user_input": latest_message})++ return ChatCompletion(+ id=f"{uuid.uuid4()}",+ created=int(time.time()),+ model="burr-app",+ object="chat.completion",+ choices=[+ Choice(+ index=0,+ message=ChatCompletionMessage(+ role="assistant",+ content=result["content"],+ ),+ finish_reason="stop",+ )+ ],+ )+++if __name__ == "__main__":+ uvicorn.run("server:app", host="127.0.0.1", port=7443, reload=True)diff --git a/examples/openai-compatible-agent/statemachine.png b/examples/openai-compatible-agent/statemachine.pngnew file mode 100644index 00000000..9cfae536Binary files /dev/null and b/examples/openai-compatible-agent/statemachine.png differdiff --git a/examples/validate_examples.py b/examples/validate_examples.pyindex 2d4e7ec7..ffbe8e97 100644--- a/examples/validate_examples.py+++ b/examples/validate_examples.py@@ -13,6 +13,7 @@ REQUIRED_FILES = []FILTERLIST = [+ "openai-compatible-agent","other-examples","conversational-rag",# todo -- remove the following once we've fleshed them out