Automate browser based workflows with AI
Python aiapiautomationbrowserbrowser-automationcomputergptllmplaywrightpowerautomatepuppeteerpythonrpaseleniumvisionworkflow
Feature: Configuring OpenAI-Compatible (LiteLLM) Models (#2166)
Brandon Beiler3d381a60f02d
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 2b9578c2e..84cfd81da 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -16,6 +16,7 @@- [Your First Code Contribution](#your-first-code-contribution)- [Improving The Documentation](#improving-the-documentation)- [Styleguides](#styleguides)+ - [Pre Commit Hooks](#pre-commit-hooks)- [Commit Messages](#commit-messages)- [Join The Project Team](#join-the-project-team)- [Attribution](#attribution)@@ -170,6 +171,18 @@ Updating, improving and correcting the documentation-->## Styleguides++### Pre Commit Hooks+Make sure to run the pre-commit hooks before committing your code.+This will help you to automatically format your code and catch cicd failures early.+```bash+# Make sure `pre-commit` is installed+pip install pre-commit++# Run pre-commit hooks on files in your commit+pre-commit run --all-files+```+### Commit Messages<!-- TODOdiff --git a/README.md b/README.mdindex f089f271e..aab39ecf1 100644--- a/README.md+++ b/README.md@@ -307,6 +307,7 @@ More extensive documentation can be found on our [docs page](https://docs.skyver| Gemini | Coming soon (contributions welcome) || Llama 3.2 | Coming soon (contributions welcome) || Novita AI | Llama 3.1 (8B, 70B), Llama 3.2 (1B, 3B, 11B Vision) |+| OpenAI-compatible | Any custom API endpoint that follows OpenAI's API format (via [liteLLM](https://docs.litellm.ai/docs/providers/openai_compatible)) |#### Environment Variables| Variable | Description| Type | Sample Value|@@ -317,8 +318,9 @@ More extensive documentation can be found on our [docs page](https://docs.skyver| `ENABLE_BEDROCK` | Register AWS Bedrock models. To use AWS Bedrock, you need to make sure your [AWS configurations](https://github.com/boto/boto3?tab=readme-ov-file#using-boto3) are set up correctly first. | Boolean | `true`, `false` || `ENABLE_GEMINI` | Register Gemini models| Boolean | `true`, `false` || `ENABLE_NOVITA`| Register Novita AI models | Boolean | `true`, `false` |-| `LLM_KEY` | The name of the model you want to use | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `BEDROCK_AMAZON_NOVA_PRO`, `BEDROCK_AMAZON_NOVA_LITE`|-| `SECONDARY_LLM_KEY` | The name of the model for mini agents skyvern runs with | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `NOVITA_DEEPSEEK_R1`, `NOVITA_DEEPSEEK_V3`, `NOVITA_LLAMA_3_3_70B`, `NOVITA_LLAMA_3_2_1B`, `NOVITA_LLAMA_3_2_3B`, `NOVITA_LLAMA_3_2_11B_VISION`, `NOVITA_LLAMA_3_1_8B`, `NOVITA_LLAMA_3_1_70B`, `NOVITA_LLAMA_3_1_405B`, `NOVITA_LLAMA_3_8B`, `NOVITA_LLAMA_3_70B`|+| `ENABLE_OPENAI_COMPATIBLE`| Register a custom OpenAI-compatible API endpoint | Boolean | `true`, `false` |+| `LLM_KEY` | The name of the model you want to use | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `BEDROCK_AMAZON_NOVA_PRO`, `BEDROCK_AMAZON_NOVA_LITE`, `OPENAI_COMPATIBLE`|+| `SECONDARY_LLM_KEY` | The name of the model for mini agents skyvern runs with | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `NOVITA_DEEPSEEK_R1`, `NOVITA_DEEPSEEK_V3`, `NOVITA_LLAMA_3_3_70B`, `NOVITA_LLAMA_3_2_1B`, `NOVITA_LLAMA_3_2_3B`, `NOVITA_LLAMA_3_2_11B_VISION`, `NOVITA_LLAMA_3_1_8B`, `NOVITA_LLAMA_3_1_70B`, `NOVITA_LLAMA_3_1_405B`, `NOVITA_LLAMA_3_8B`, `NOVITA_LLAMA_3_70B`, `OPENAI_COMPATIBLE`|| `OPENAI_API_KEY` | OpenAI API Key | String | `sk-1234567890` || `OPENAI_API_BASE` | OpenAI API Base, optional | String | `https://openai.api.base` || `OPENAI_ORGANIZATION` | OpenAI Organization ID, optional | String | `your-org-id` |@@ -328,6 +330,18 @@ More extensive documentation can be found on our [docs page](https://docs.skyver| `AZURE_API_BASE` | Azure deployment api base url| String | `https://skyvern-deployment.openai.azure.com/`|| `AZURE_API_VERSION` | Azure API Version| String | `2024-02-01`|| `GEMINI_API_KEY` | Gemini API Key| String | `your_google_gemini_api_key`|+| `NOVITA_API_KEY` | Novita AI API Key| String | `your_novita_api_key`|+| `OPENAI_COMPATIBLE_MODEL_NAME` | Model name for OpenAI-compatible endpoint | String | `yi-34b`, `gpt-3.5-turbo`, `mistral-large`, etc.|+| `OPENAI_COMPATIBLE_API_KEY` | API key for OpenAI-compatible endpoint | String | `sk-1234567890`|+| `OPENAI_COMPATIBLE_API_BASE` | Base URL for OpenAI-compatible endpoint | String | `https://api.together.xyz/v1`, `http://localhost:8000/v1`, etc.|++#### Environment Variables (OpenAI-Compatible model - additional config)+| Variable | Description| Type | Sample Value|+| -------- | ------- | ------- | ------- |+| `OPENAI_COMPATIBLE_API_VERSION` | API version for OpenAI-compatible endpoint, optional| String | `2023-05-15`|+| `OPENAI_COMPATIBLE_MAX_TOKENS` | Maximum tokens for completion, optional| Integer | `4096`, `8192`, etc.|+| `OPENAI_COMPATIBLE_TEMPERATURE` | Temperature setting, optional| Float | `0.0`, `0.5`, `0.7`, etc.|+| `OPENAI_COMPATIBLE_SUPPORTS_VISION` | Whether model supports vision, optional| Boolean | `true`, `false`|# Feature RoadmapThis is our planned roadmap for the next few months. If you have any suggestions or would like to see a feature added, please don't hesitate to reach out to us [via email](mailto:[email protected]) or [discord](https://discord.gg/fG2XXEuQX3).diff --git a/docs/development.mdx b/docs/development.mdxindex 878300893..646b19622 100644--- a/docs/development.mdx+++ b/docs/development.mdx@@ -96,3 +96,69 @@ Repeat step 3.</AccordionGroup>Curious about what changed in a CLI version? [Check out the CLI changelog.](/changelog/command-line)++## Environment Variable Configuration++The simplest way to get started with Skyvern is to set environment variables in a `.env` file at the root of the project. This file is loaded by the application at runtime, and the values are used to configure the system.++### LLM Configuration++Skyvern works with multiple LLM providers. You need to set the following environment variables to configure your LLM provider:++```bash+# One of the below must be set to true+ENABLE_OPENAI=true+# ENABLE_ANTHROPIC=true+# ENABLE_AZURE=true+# ENABLE_BEDROCK=true+# ENABLE_GEMINI=true+# ENABLE_NOVITA=true+# ENABLE_OPENAI_COMPATIBLE=true++# Set your LLM provider API key+OPENAI_API_KEY=sk-xxxxxxxxxxxxx++# Set which model to use+LLM_KEY=OPENAI_GPT4O+```++If you're using OpenAI, you'll need to set `ENABLE_OPENAI=true` and provide your `OPENAI_API_KEY`. The `LLM_KEY` specifies which model to use, and should match one of the registered models in the `LLMConfigRegistry`.++#### Using Custom OpenAI-compatible Models++Skyvern can also use any OpenAI-compatible LLM API endpoint. This is useful for connecting to alternative providers that follow the OpenAI API format or to self-hosted models. This feature is implemented using [liteLLM's OpenAI-compatible provider support](https://docs.litellm.ai/docs/providers/openai_compatible).++To enable this:++```bash+# Enable OpenAI-compatible mode+ENABLE_OPENAI_COMPATIBLE=true++# Required configuration+OPENAI_COMPATIBLE_MODEL_NAME=gpt-3.5-turbo # The model name supported by your endpoint (REQUIRED)+OPENAI_COMPATIBLE_API_KEY=your-api-key # API key for your endpoint (REQUIRED)+OPENAI_COMPATIBLE_API_BASE=https://your-api-endpoint.com/v1 # Base URL for your endpoint (REQUIRED)++# Optional configuration+OPENAI_COMPATIBLE_API_VERSION=2023-05-15 # Optional API version+OPENAI_COMPATIBLE_MAX_TOKENS=4096 # Optional, defaults to LLM_CONFIG_MAX_TOKENS+OPENAI_COMPATIBLE_TEMPERATURE=0.0 # Optional, defaults to LLM_CONFIG_TEMPERATURE+OPENAI_COMPATIBLE_SUPPORTS_VISION=false # Optional, defaults to false+OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX=false # Optional, defaults to false+OPENAI_COMPATIBLE_MODEL_KEY=OPENAI_COMPATIBLE # Optional custom key to register the model with+```++**Important Note**: When using this feature, the model name you provide will be prefixed with "openai/" in the liteLLM configuration. This is how liteLLM determines the routing for OpenAI-compatible providers.++This feature allows you to use models from providers like:+- Together.ai+- Anyscale+- Mistral+- Self-hosted models (e.g., using LM Studio or a local vLLM server)+- Any API endpoint that follows the OpenAI API format++Once configured, you can set `LLM_KEY=OPENAI_COMPATIBLE` to use this model as your primary LLM.++For more detailed information about OpenAI-compatible providers supported by liteLLM, see their [documentation](https://docs.litellm.ai/docs/providers/openai_compatible).++### Running the Serverdiff --git a/setup.sh b/setup.shindex 8321f730d..607b77fb9 100755--- a/setup.sh+++ b/setup.sh@@ -161,6 +161,39 @@ setup_llm_providers() {update_or_add_env_var "ENABLE_NOVITA" "false"fi+ # OpenAI Compatible Configuration+ echo "To enable an OpenAI-compatible provider, you must have a model name, API key, and API base URL."+ read -p "Do you want to enable an OpenAI-compatible provider (y/n)? " enable_openai_compatible+ if [[ "$enable_openai_compatible" == "y" ]]; then+ read -p "Enter the model name (e.g., 'yi-34b', 'mistral-large'): " openai_compatible_model_name+ read -p "Enter your API key: " openai_compatible_api_key+ read -p "Enter the API base URL (e.g., 'https://api.together.xyz/v1'): " openai_compatible_api_base+ read -p "Does this model support vision (y/n)? " openai_compatible_vision++ if [ -z "$openai_compatible_model_name" ] || [ -z "$openai_compatible_api_key" ] || [ -z "$openai_compatible_api_base" ]; then+ echo "Error: All required fields must be populated."+ echo "OpenAI-compatible provider will not be enabled."+ else+ update_or_add_env_var "OPENAI_COMPATIBLE_MODEL_NAME" "$openai_compatible_model_name"+ update_or_add_env_var "OPENAI_COMPATIBLE_API_KEY" "$openai_compatible_api_key"+ update_or_add_env_var "OPENAI_COMPATIBLE_API_BASE" "$openai_compatible_api_base"++ # Set vision support+ if [[ "$openai_compatible_vision" == "y" ]]; then+ update_or_add_env_var "OPENAI_COMPATIBLE_SUPPORTS_VISION" "true"+ else+ update_or_add_env_var "OPENAI_COMPATIBLE_SUPPORTS_VISION" "false"+ fi++ update_or_add_env_var "ENABLE_OPENAI_COMPATIBLE" "true"+ model_options+=(+ "OPENAI_COMPATIBLE"+ )+ fi+ else+ update_or_add_env_var "ENABLE_OPENAI_COMPATIBLE" "false"+ fi+# Model Selectionif [ ${#model_options[@]} -eq 0 ]; thenecho "No LLM providers enabled. You won't be able to run Skyvern unless you enable at least one provider. You can re-run this script to enable providers or manually update the .env file."diff --git a/skyvern/cli/commands.py b/skyvern/cli/commands.pyindex 328c47dc4..093f758c5 100644--- a/skyvern/cli/commands.py+++ b/skyvern/cli/commands.py@@ -326,6 +326,39 @@ def setup_llm_providers() -> None:else:update_or_add_env_var("ENABLE_NOVITA", "false")+ # OpenAI Compatible Configuration+ print("To enable an OpenAI-compatible provider, you must have a model name, API key, and API base URL.")+ enable_openai_compatible = input("Do you want to enable an OpenAI-compatible provider (y/n)? ").lower() == "y"+ if enable_openai_compatible:+ openai_compatible_model_name = input("Enter the model name (e.g., 'yi-34b', 'mistral-large'): ")+ openai_compatible_api_key = input("Enter your API key: ")+ openai_compatible_api_base = input("Enter the API base URL (e.g., 'https://api.together.xyz/v1'): ")+ openai_compatible_vision = input("Does this model support vision (y/n)? ").lower() == "y"++ if not all([openai_compatible_model_name, openai_compatible_api_key, openai_compatible_api_base]):+ print("Error: All required fields must be populated.")+ print("OpenAI-compatible provider will not be enabled.")+ else:+ update_or_add_env_var("OPENAI_COMPATIBLE_MODEL_NAME", openai_compatible_model_name)+ update_or_add_env_var("OPENAI_COMPATIBLE_API_KEY", openai_compatible_api_key)+ update_or_add_env_var("OPENAI_COMPATIBLE_API_BASE", openai_compatible_api_base)++ # Set vision support+ if openai_compatible_vision:+ update_or_add_env_var("OPENAI_COMPATIBLE_SUPPORTS_VISION", "true")+ else:+ update_or_add_env_var("OPENAI_COMPATIBLE_SUPPORTS_VISION", "false")++ # Optional: Ask for API version+ openai_compatible_api_version = input("Enter API version (optional, press enter to skip): ")+ if openai_compatible_api_version:+ update_or_add_env_var("OPENAI_COMPATIBLE_API_VERSION", openai_compatible_api_version)++ update_or_add_env_var("ENABLE_OPENAI_COMPATIBLE", "true")+ model_options.append("OPENAI_COMPATIBLE")+ else:+ update_or_add_env_var("ENABLE_OPENAI_COMPATIBLE", "false")+# Model Selectionif not model_options:print(diff --git a/skyvern/config.py b/skyvern/config.pyindex 5fca961c2..5bb16cbf7 100644--- a/skyvern/config.py+++ b/skyvern/config.py@@ -124,10 +124,24 @@ class Settings(BaseSettings):ENABLE_BEDROCK: bool = FalseENABLE_GEMINI: bool = FalseENABLE_AZURE_CUA: bool = False+ ENABLE_OPENAI_COMPATIBLE: bool = False# OPENAIOPENAI_API_KEY: str | None = None# ANTHROPICANTHROPIC_API_KEY: str | None = None++ # OPENAI COMPATIBLE+ OPENAI_COMPATIBLE_MODEL_NAME: str | None = None+ OPENAI_COMPATIBLE_API_KEY: str | None = None+ OPENAI_COMPATIBLE_API_BASE: str | None = None+ OPENAI_COMPATIBLE_API_VERSION: str | None = None+ OPENAI_COMPATIBLE_MAX_TOKENS: int | None = None+ OPENAI_COMPATIBLE_TEMPERATURE: float | None = None+ OPENAI_COMPATIBLE_SUPPORTS_VISION: bool = False+ OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX: bool = False+ OPENAI_COMPATIBLE_MODEL_KEY: str = "OPENAI_COMPATIBLE"+ OPENAI_COMPATIBLE_REASONING_EFFORT: str | None = None+# AZUREAZURE_DEPLOYMENT: str | None = NoneAZURE_API_KEY: str | None = Nonediff --git a/skyvern/forge/sdk/api/llm/config_registry.py b/skyvern/forge/sdk/api/llm/config_registry.pyindex 6536f6cfb..12935113a 100644--- a/skyvern/forge/sdk/api/llm/config_registry.py+++ b/skyvern/forge/sdk/api/llm/config_registry.py@@ -606,3 +606,45 @@ if settings.ENABLE_NOVITA:),),)++# Add support for dynamically configuring OpenAI-compatible LLM models+# Based on liteLLM's support for OpenAI-compatible APIs+# See documentation: https://docs.litellm.ai/docs/providers/openai_compatible+if settings.ENABLE_OPENAI_COMPATIBLE:+ # Check for required model name+ model_key = settings.OPENAI_COMPATIBLE_MODEL_KEY+ model_name = settings.OPENAI_COMPATIBLE_MODEL_NAME++ if not model_name:+ raise InvalidLLMConfigError(+ "OPENAI_COMPATIBLE_MODEL_NAME is required but not set. OpenAI-compatible model will not be registered."+ )+ else:+ # Required environment variables to check+ required_env_vars = ["OPENAI_COMPATIBLE_API_KEY", "OPENAI_COMPATIBLE_MODEL_NAME", "OPENAI_COMPATIBLE_API_BASE"]++ # Configure litellm parameters - note the "openai/" prefix required for liteLLM routing+ litellm_params = LiteLLMParams(+ api_key=settings.OPENAI_COMPATIBLE_API_KEY,+ api_base=settings.OPENAI_COMPATIBLE_API_BASE,+ api_version=settings.OPENAI_COMPATIBLE_API_VERSION,+ model_info={"model_name": f"openai/{model_name}"},+ )++ # Configure LLMConfig+ LLMConfigRegistry.register_config(+ model_key,+ LLMConfig(+ f"openai/{model_name}", # Add openai/ prefix for liteLLM+ required_env_vars,+ supports_vision=settings.OPENAI_COMPATIBLE_SUPPORTS_VISION,+ add_assistant_prefix=settings.OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX,+ max_completion_tokens=settings.OPENAI_COMPATIBLE_MAX_TOKENS or settings.LLM_CONFIG_MAX_TOKENS,+ temperature=settings.OPENAI_COMPATIBLE_TEMPERATURE+ if settings.OPENAI_COMPATIBLE_TEMPERATURE is not None+ else settings.LLM_CONFIG_TEMPERATURE,+ litellm_params=litellm_params,+ reasoning_effort=settings.OPENAI_COMPATIBLE_REASONING_EFFORT,+ ),+ )+ LOG.info(f"Registered OpenAI-compatible model with key {model_key}", model_name=model_name)