Enhanced ChatGPT Clone: Features Agents, MCP, Skills, DeepSeek, Anthropic, AWS, OpenAI, Responses API, Azure, Groq, o1, GPT-5, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, Code Interpreter, langchain, DALL-E-3, OpenAPI Actions, Functions, Secure Multi-User Auth, Presets, open-source for self-hosting. Active
TypeScript aianthropicartifactsawsazurechatgptchatgpt-cloneclaudeclonedeepseekgeminigooglegpt-5librechatmcpo1openairesponses-apivisionwebui
📚 docs: Add LiteLLM Proxy - Load balance 100+ LLMs & Spend Tracking ⚖️🤖📈 (#1249)
Ishaan Jaff2bcfb04a7241
diff --git a/README.md b/README.mdindex 127447d382..e77d6b86d7 100644--- a/README.md+++ b/README.md@@ -73,6 +73,7 @@ Keep up with the latest updates by visiting the releases page - [Releases](https* [User Auth System](docs/install/user_auth_system.md)* [Online MongoDB Database](docs/install/mongodb.md)* [Default Language](docs/install/default_language.md)+ * [LiteLLM Proxy: Load Balance LLMs + Spend Tracking](docs/install/litellm.md)</details><details>diff --git a/docs/install/litellm.md b/docs/install/litellm.mdnew file mode 100644index 0000000000..dfae409e00--- /dev/null+++ b/docs/install/litellm.md@@ -0,0 +1,96 @@+# Using LibreChat with LiteLLM Proxy+Use LiteLLM Proxy for:+* Calling 100+ LLMs Huggingface/Bedrock/TogetherAI/etc. in the OpenAI ChatCompletions & Completions format+* Load balancing - between Multiple Models + Deployments of the same model LiteLLM proxy can handle 1k+ requests/second during load tests+* Authentication & Spend Tracking Virtual Keys++## Start LiteLLM Proxy Server+### Pip install litellm+```shell+pip install litellm+```++### Create a config.yaml for litellm proxy+More information on LiteLLM configurations here: https://docs.litellm.ai/docs/simple_proxy#proxy-configs++```yaml+model_list:+ - model_name: gpt-3.5-turbo+ litellm_params:+ model: azure/gpt-turbo-small-eu+ api_base: https://my-endpoint-europe-berri-992.openai.azure.com/+ api_key:+ rpm: 6 # Rate limit for this deployment: in requests per minute (rpm)+ - model_name: gpt-3.5-turbo+ litellm_params:+ model: azure/gpt-turbo-small-ca+ api_base: https://my-endpoint-canada-berri992.openai.azure.com/+ api_key:+ rpm: 6+ - model_name: gpt-3.5-turbo+ litellm_params:+ model: azure/gpt-turbo-large+ api_base: https://openai-france-1234.openai.azure.com/+ api_key:+ rpm: 1440+```++### Start the proxy+```shell+litellm --config /path/to/config.yaml++#INFO: Proxy running on http://0.0.0.0:8000+```++## Use LiteLLM Proxy Server with LibreChat+++#### 1. Clone the repo+```shell+git clone https://github.com/danny-avila/LibreChat.git+```+++#### 2. Modify Librechat's `docker-compose.yml`+```yaml+OPENAI_REVERSE_PROXY=http://host.docker.internal:8000/v1/chat/completions+```++#### 3. Save fake OpenAI key in Librechat's `.env`++Copy Librechat's `.env.example` to `.env` and overwrite the default OPENAI_API_KEY (by default it requires the user to pass a key).+```env+OPENAI_API_KEY=sk-1234+```++#### 4. Run LibreChat:+```shell+docker compose up+```++---++### Why use LiteLLM?++1. **Access to Multiple LLMs**: It allows calling over 100 LLMs from platforms like Huggingface, Bedrock, TogetherAI, etc., using OpenAI's ChatCompletions and Completions format.++2. **Load Balancing**: Capable of handling over 1,000 requests per second during load tests, it balances load across various models and deployments.++3. **Authentication & Spend Tracking**: The server supports virtual keys for authentication and tracks spending.++Key components and features include:++- **Installation**: Easy installation.+- **Testing**: Testing features to route requests to specific models.+- **Server Endpoints**: Offers multiple endpoints for chat completions, completions, embeddings, model lists, and key generation.+- **Supported LLMs**: Supports a wide range of LLMs, including AWS Bedrock, Azure OpenAI, Huggingface, AWS Sagemaker, Anthropic, and more.+- **Proxy Configurations**: Allows setting various parameters like model list, server settings, environment variables, and more.+- **Multiple Models Management**: Configurations can be set up for managing multiple models with fallbacks, cooldowns, retries, and timeouts.+- **Embedding Models Support**: Special configurations for embedding models.+- **Authentication Management**: Features for managing authentication through virtual keys, model upgrades/downgrades, and tracking spend.+- **Custom Configurations**: Supports setting model-specific parameters, caching responses, and custom prompt templates.+- **Debugging Tools**: Options for debugging and logging proxy input/output.+- **Deployment and Performance**: Information on deploying LiteLLM Proxy and its performance metrics.+- **Proxy CLI Arguments**: A wide range of command-line arguments for customization.++Overall, LiteLLM Server offers a comprehensive suite of tools for managing, deploying, and interacting with a variety of LLMs, making it a versatile choice for large-scale AI applications.diff --git a/mkdocs.yml b/mkdocs.ymlindex 211d520e6d..1ce87c5acb 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -93,6 +93,7 @@ nav:- User Auth System: 'install/user_auth_system.md'- Online MongoDB Database: 'install/mongodb.md'- Languages: 'install/default_language.md'+ - LiteLLM Proxy: 'install/litellm.md'- Miscellaneous: 'install/misc.md'- Features:- Plugins: