Skip to content

Automate browser based workflows with AI

Feature: Configuring OpenAI-Compatible (LiteLLM) Models (#2166)

Brandon Beiler3d381a60f02d
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 2b9578c2e..84cfd81da 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -16,6 +16,7 @@
- [Your First Code Contribution](#your-first-code-contribution)
- [Improving The Documentation](#improving-the-documentation)
- [Styleguides](#styleguides)
+ - [Pre Commit Hooks](#pre-commit-hooks)
- [Commit Messages](#commit-messages)
- [Join The Project Team](#join-the-project-team)
- [Attribution](#attribution)
@@ -170,6 +171,18 @@ Updating, improving and correcting the documentation
-->
## Styleguides
+
+### Pre Commit Hooks
+Make sure to run the pre-commit hooks before committing your code.
+This will help you to automatically format your code and catch cicd failures early.
+```bash
+# Make sure `pre-commit` is installed
+pip install pre-commit
+
+# Run pre-commit hooks on files in your commit
+pre-commit run --all-files
+```
+
### Commit Messages
<!-- TODO
diff --git a/README.md b/README.md
index f089f271e..aab39ecf1 100644
--- a/README.md
+++ b/README.md
@@ -307,6 +307,7 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
| Gemini | Coming soon (contributions welcome) |
| Llama 3.2 | Coming soon (contributions welcome) |
| Novita AI | Llama 3.1 (8B, 70B), Llama 3.2 (1B, 3B, 11B Vision) |
+| OpenAI-compatible | Any custom API endpoint that follows OpenAI's API format (via [liteLLM](https://docs.litellm.ai/docs/providers/openai_compatible)) |
#### Environment Variables
| Variable | Description| Type | Sample Value|
@@ -317,8 +318,9 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
| `ENABLE_BEDROCK` | Register AWS Bedrock models. To use AWS Bedrock, you need to make sure your [AWS configurations](https://github.com/boto/boto3?tab=readme-ov-file#using-boto3) are set up correctly first. | Boolean | `true`, `false` |
| `ENABLE_GEMINI` | Register Gemini models| Boolean | `true`, `false` |
| `ENABLE_NOVITA`| Register Novita AI models | Boolean | `true`, `false` |
-| `LLM_KEY` | The name of the model you want to use | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `BEDROCK_AMAZON_NOVA_PRO`, `BEDROCK_AMAZON_NOVA_LITE`|
-| `SECONDARY_LLM_KEY` | The name of the model for mini agents skyvern runs with | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `NOVITA_DEEPSEEK_R1`, `NOVITA_DEEPSEEK_V3`, `NOVITA_LLAMA_3_3_70B`, `NOVITA_LLAMA_3_2_1B`, `NOVITA_LLAMA_3_2_3B`, `NOVITA_LLAMA_3_2_11B_VISION`, `NOVITA_LLAMA_3_1_8B`, `NOVITA_LLAMA_3_1_70B`, `NOVITA_LLAMA_3_1_405B`, `NOVITA_LLAMA_3_8B`, `NOVITA_LLAMA_3_70B`|
+| `ENABLE_OPENAI_COMPATIBLE`| Register a custom OpenAI-compatible API endpoint | Boolean | `true`, `false` |
+| `LLM_KEY` | The name of the model you want to use | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `BEDROCK_AMAZON_NOVA_PRO`, `BEDROCK_AMAZON_NOVA_LITE`, `OPENAI_COMPATIBLE`|
+| `SECONDARY_LLM_KEY` | The name of the model for mini agents skyvern runs with | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `NOVITA_DEEPSEEK_R1`, `NOVITA_DEEPSEEK_V3`, `NOVITA_LLAMA_3_3_70B`, `NOVITA_LLAMA_3_2_1B`, `NOVITA_LLAMA_3_2_3B`, `NOVITA_LLAMA_3_2_11B_VISION`, `NOVITA_LLAMA_3_1_8B`, `NOVITA_LLAMA_3_1_70B`, `NOVITA_LLAMA_3_1_405B`, `NOVITA_LLAMA_3_8B`, `NOVITA_LLAMA_3_70B`, `OPENAI_COMPATIBLE`|
| `OPENAI_API_KEY` | OpenAI API Key | String | `sk-1234567890` |
| `OPENAI_API_BASE` | OpenAI API Base, optional | String | `https://openai.api.base` |
| `OPENAI_ORGANIZATION` | OpenAI Organization ID, optional | String | `your-org-id` |
@@ -328,6 +330,18 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
| `AZURE_API_BASE` | Azure deployment api base url| String | `https://skyvern-deployment.openai.azure.com/`|
| `AZURE_API_VERSION` | Azure API Version| String | `2024-02-01`|
| `GEMINI_API_KEY` | Gemini API Key| String | `your_google_gemini_api_key`|
+| `NOVITA_API_KEY` | Novita AI API Key| String | `your_novita_api_key`|
+| `OPENAI_COMPATIBLE_MODEL_NAME` | Model name for OpenAI-compatible endpoint | String | `yi-34b`, `gpt-3.5-turbo`, `mistral-large`, etc.|
+| `OPENAI_COMPATIBLE_API_KEY` | API key for OpenAI-compatible endpoint | String | `sk-1234567890`|
+| `OPENAI_COMPATIBLE_API_BASE` | Base URL for OpenAI-compatible endpoint | String | `https://api.together.xyz/v1`, `http://localhost:8000/v1`, etc.|
+
+#### Environment Variables (OpenAI-Compatible model - additional config)
+| Variable | Description| Type | Sample Value|
+| -------- | ------- | ------- | ------- |
+| `OPENAI_COMPATIBLE_API_VERSION` | API version for OpenAI-compatible endpoint, optional| String | `2023-05-15`|
+| `OPENAI_COMPATIBLE_MAX_TOKENS` | Maximum tokens for completion, optional| Integer | `4096`, `8192`, etc.|
+| `OPENAI_COMPATIBLE_TEMPERATURE` | Temperature setting, optional| Float | `0.0`, `0.5`, `0.7`, etc.|
+| `OPENAI_COMPATIBLE_SUPPORTS_VISION` | Whether model supports vision, optional| Boolean | `true`, `false`|
# Feature Roadmap
This is our planned roadmap for the next few months. If you have any suggestions or would like to see a feature added, please don't hesitate to reach out to us [via email](mailto:[email protected]) or [discord](https://discord.gg/fG2XXEuQX3).
diff --git a/docs/development.mdx b/docs/development.mdx
index 878300893..646b19622 100644
--- a/docs/development.mdx
+++ b/docs/development.mdx
@@ -96,3 +96,69 @@ Repeat step 3.
</AccordionGroup>
Curious about what changed in a CLI version? [Check out the CLI changelog.](/changelog/command-line)
+
+## Environment Variable Configuration
+
+The simplest way to get started with Skyvern is to set environment variables in a `.env` file at the root of the project. This file is loaded by the application at runtime, and the values are used to configure the system.
+
+### LLM Configuration
+
+Skyvern works with multiple LLM providers. You need to set the following environment variables to configure your LLM provider:
+
+```bash
+# One of the below must be set to true
+ENABLE_OPENAI=true
+# ENABLE_ANTHROPIC=true
+# ENABLE_AZURE=true
+# ENABLE_BEDROCK=true
+# ENABLE_GEMINI=true
+# ENABLE_NOVITA=true
+# ENABLE_OPENAI_COMPATIBLE=true
+
+# Set your LLM provider API key
+OPENAI_API_KEY=sk-xxxxxxxxxxxxx
+
+# Set which model to use
+LLM_KEY=OPENAI_GPT4O
+```
+
+If you're using OpenAI, you'll need to set `ENABLE_OPENAI=true` and provide your `OPENAI_API_KEY`. The `LLM_KEY` specifies which model to use, and should match one of the registered models in the `LLMConfigRegistry`.
+
+#### Using Custom OpenAI-compatible Models
+
+Skyvern can also use any OpenAI-compatible LLM API endpoint. This is useful for connecting to alternative providers that follow the OpenAI API format or to self-hosted models. This feature is implemented using [liteLLM's OpenAI-compatible provider support](https://docs.litellm.ai/docs/providers/openai_compatible).
+
+To enable this:
+
+```bash
+# Enable OpenAI-compatible mode
+ENABLE_OPENAI_COMPATIBLE=true
+
+# Required configuration
+OPENAI_COMPATIBLE_MODEL_NAME=gpt-3.5-turbo # The model name supported by your endpoint (REQUIRED)
+OPENAI_COMPATIBLE_API_KEY=your-api-key # API key for your endpoint (REQUIRED)
+OPENAI_COMPATIBLE_API_BASE=https://your-api-endpoint.com/v1 # Base URL for your endpoint (REQUIRED)
+
+# Optional configuration
+OPENAI_COMPATIBLE_API_VERSION=2023-05-15 # Optional API version
+OPENAI_COMPATIBLE_MAX_TOKENS=4096 # Optional, defaults to LLM_CONFIG_MAX_TOKENS
+OPENAI_COMPATIBLE_TEMPERATURE=0.0 # Optional, defaults to LLM_CONFIG_TEMPERATURE
+OPENAI_COMPATIBLE_SUPPORTS_VISION=false # Optional, defaults to false
+OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX=false # Optional, defaults to false
+OPENAI_COMPATIBLE_MODEL_KEY=OPENAI_COMPATIBLE # Optional custom key to register the model with
+```
+
+**Important Note**: When using this feature, the model name you provide will be prefixed with "openai/" in the liteLLM configuration. This is how liteLLM determines the routing for OpenAI-compatible providers.
+
+This feature allows you to use models from providers like:
+- Together.ai
+- Anyscale
+- Mistral
+- Self-hosted models (e.g., using LM Studio or a local vLLM server)
+- Any API endpoint that follows the OpenAI API format
+
+Once configured, you can set `LLM_KEY=OPENAI_COMPATIBLE` to use this model as your primary LLM.
+
+For more detailed information about OpenAI-compatible providers supported by liteLLM, see their [documentation](https://docs.litellm.ai/docs/providers/openai_compatible).
+
+### Running the Server
diff --git a/setup.sh b/setup.sh
index 8321f730d..607b77fb9 100755
--- a/setup.sh
+++ b/setup.sh
@@ -161,6 +161,39 @@ setup_llm_providers() {
update_or_add_env_var "ENABLE_NOVITA" "false"
fi
+ # OpenAI Compatible Configuration
+ echo "To enable an OpenAI-compatible provider, you must have a model name, API key, and API base URL."
+ read -p "Do you want to enable an OpenAI-compatible provider (y/n)? " enable_openai_compatible
+ if [[ "$enable_openai_compatible" == "y" ]]; then
+ read -p "Enter the model name (e.g., 'yi-34b', 'mistral-large'): " openai_compatible_model_name
+ read -p "Enter your API key: " openai_compatible_api_key
+ read -p "Enter the API base URL (e.g., 'https://api.together.xyz/v1'): " openai_compatible_api_base
+ read -p "Does this model support vision (y/n)? " openai_compatible_vision
+
+ if [ -z "$openai_compatible_model_name" ] || [ -z "$openai_compatible_api_key" ] || [ -z "$openai_compatible_api_base" ]; then
+ echo "Error: All required fields must be populated."
+ echo "OpenAI-compatible provider will not be enabled."
+ else
+ update_or_add_env_var "OPENAI_COMPATIBLE_MODEL_NAME" "$openai_compatible_model_name"
+ update_or_add_env_var "OPENAI_COMPATIBLE_API_KEY" "$openai_compatible_api_key"
+ update_or_add_env_var "OPENAI_COMPATIBLE_API_BASE" "$openai_compatible_api_base"
+
+ # Set vision support
+ if [[ "$openai_compatible_vision" == "y" ]]; then
+ update_or_add_env_var "OPENAI_COMPATIBLE_SUPPORTS_VISION" "true"
+ else
+ update_or_add_env_var "OPENAI_COMPATIBLE_SUPPORTS_VISION" "false"
+ fi
+
+ update_or_add_env_var "ENABLE_OPENAI_COMPATIBLE" "true"
+ model_options+=(
+ "OPENAI_COMPATIBLE"
+ )
+ fi
+ else
+ update_or_add_env_var "ENABLE_OPENAI_COMPATIBLE" "false"
+ fi
+
# Model Selection
if [ ${#model_options[@]} -eq 0 ]; then
echo "No LLM providers enabled. You won't be able to run Skyvern unless you enable at least one provider. You can re-run this script to enable providers or manually update the .env file."
diff --git a/skyvern/cli/commands.py b/skyvern/cli/commands.py
index 328c47dc4..093f758c5 100644
--- a/skyvern/cli/commands.py
+++ b/skyvern/cli/commands.py
@@ -326,6 +326,39 @@ def setup_llm_providers() -> None:
else:
update_or_add_env_var("ENABLE_NOVITA", "false")
+ # OpenAI Compatible Configuration
+ print("To enable an OpenAI-compatible provider, you must have a model name, API key, and API base URL.")
+ enable_openai_compatible = input("Do you want to enable an OpenAI-compatible provider (y/n)? ").lower() == "y"
+ if enable_openai_compatible:
+ openai_compatible_model_name = input("Enter the model name (e.g., 'yi-34b', 'mistral-large'): ")
+ openai_compatible_api_key = input("Enter your API key: ")
+ openai_compatible_api_base = input("Enter the API base URL (e.g., 'https://api.together.xyz/v1'): ")
+ openai_compatible_vision = input("Does this model support vision (y/n)? ").lower() == "y"
+
+ if not all([openai_compatible_model_name, openai_compatible_api_key, openai_compatible_api_base]):
+ print("Error: All required fields must be populated.")
+ print("OpenAI-compatible provider will not be enabled.")
+ else:
+ update_or_add_env_var("OPENAI_COMPATIBLE_MODEL_NAME", openai_compatible_model_name)
+ update_or_add_env_var("OPENAI_COMPATIBLE_API_KEY", openai_compatible_api_key)
+ update_or_add_env_var("OPENAI_COMPATIBLE_API_BASE", openai_compatible_api_base)
+
+ # Set vision support
+ if openai_compatible_vision:
+ update_or_add_env_var("OPENAI_COMPATIBLE_SUPPORTS_VISION", "true")
+ else:
+ update_or_add_env_var("OPENAI_COMPATIBLE_SUPPORTS_VISION", "false")
+
+ # Optional: Ask for API version
+ openai_compatible_api_version = input("Enter API version (optional, press enter to skip): ")
+ if openai_compatible_api_version:
+ update_or_add_env_var("OPENAI_COMPATIBLE_API_VERSION", openai_compatible_api_version)
+
+ update_or_add_env_var("ENABLE_OPENAI_COMPATIBLE", "true")
+ model_options.append("OPENAI_COMPATIBLE")
+ else:
+ update_or_add_env_var("ENABLE_OPENAI_COMPATIBLE", "false")
+
# Model Selection
if not model_options:
print(
diff --git a/skyvern/config.py b/skyvern/config.py
index 5fca961c2..5bb16cbf7 100644
--- a/skyvern/config.py
+++ b/skyvern/config.py
@@ -124,10 +124,24 @@ class Settings(BaseSettings):
ENABLE_BEDROCK: bool = False
ENABLE_GEMINI: bool = False
ENABLE_AZURE_CUA: bool = False
+ ENABLE_OPENAI_COMPATIBLE: bool = False
# OPENAI
OPENAI_API_KEY: str | None = None
# ANTHROPIC
ANTHROPIC_API_KEY: str | None = None
+
+ # OPENAI COMPATIBLE
+ OPENAI_COMPATIBLE_MODEL_NAME: str | None = None
+ OPENAI_COMPATIBLE_API_KEY: str | None = None
+ OPENAI_COMPATIBLE_API_BASE: str | None = None
+ OPENAI_COMPATIBLE_API_VERSION: str | None = None
+ OPENAI_COMPATIBLE_MAX_TOKENS: int | None = None
+ OPENAI_COMPATIBLE_TEMPERATURE: float | None = None
+ OPENAI_COMPATIBLE_SUPPORTS_VISION: bool = False
+ OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX: bool = False
+ OPENAI_COMPATIBLE_MODEL_KEY: str = "OPENAI_COMPATIBLE"
+ OPENAI_COMPATIBLE_REASONING_EFFORT: str | None = None
+
# AZURE
AZURE_DEPLOYMENT: str | None = None
AZURE_API_KEY: str | None = None
diff --git a/skyvern/forge/sdk/api/llm/config_registry.py b/skyvern/forge/sdk/api/llm/config_registry.py
index 6536f6cfb..12935113a 100644
--- a/skyvern/forge/sdk/api/llm/config_registry.py
+++ b/skyvern/forge/sdk/api/llm/config_registry.py
@@ -606,3 +606,45 @@ if settings.ENABLE_NOVITA:
),
),
)
+
+# Add support for dynamically configuring OpenAI-compatible LLM models
+# Based on liteLLM's support for OpenAI-compatible APIs
+# See documentation: https://docs.litellm.ai/docs/providers/openai_compatible
+if settings.ENABLE_OPENAI_COMPATIBLE:
+ # Check for required model name
+ model_key = settings.OPENAI_COMPATIBLE_MODEL_KEY
+ model_name = settings.OPENAI_COMPATIBLE_MODEL_NAME
+
+ if not model_name:
+ raise InvalidLLMConfigError(
+ "OPENAI_COMPATIBLE_MODEL_NAME is required but not set. OpenAI-compatible model will not be registered."
+ )
+ else:
+ # Required environment variables to check
+ required_env_vars = ["OPENAI_COMPATIBLE_API_KEY", "OPENAI_COMPATIBLE_MODEL_NAME", "OPENAI_COMPATIBLE_API_BASE"]
+
+ # Configure litellm parameters - note the "openai/" prefix required for liteLLM routing
+ litellm_params = LiteLLMParams(
+ api_key=settings.OPENAI_COMPATIBLE_API_KEY,
+ api_base=settings.OPENAI_COMPATIBLE_API_BASE,
+ api_version=settings.OPENAI_COMPATIBLE_API_VERSION,
+ model_info={"model_name": f"openai/{model_name}"},
+ )
+
+ # Configure LLMConfig
+ LLMConfigRegistry.register_config(
+ model_key,
+ LLMConfig(
+ f"openai/{model_name}", # Add openai/ prefix for liteLLM
+ required_env_vars,
+ supports_vision=settings.OPENAI_COMPATIBLE_SUPPORTS_VISION,
+ add_assistant_prefix=settings.OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX,
+ max_completion_tokens=settings.OPENAI_COMPATIBLE_MAX_TOKENS or settings.LLM_CONFIG_MAX_TOKENS,
+ temperature=settings.OPENAI_COMPATIBLE_TEMPERATURE
+ if settings.OPENAI_COMPATIBLE_TEMPERATURE is not None
+ else settings.LLM_CONFIG_TEMPERATURE,
+ litellm_params=litellm_params,
+ reasoning_effort=settings.OPENAI_COMPATIBLE_REASONING_EFFORT,
+ ),
+ )
+ LOG.info(f"Registered OpenAI-compatible model with key {model_key}", model_name=model_name)