Skip to content

Self-improving AI Employees for everyone

feat(llm): add named OpenAI-compatible endpoints

trohitg316d90cf249b
diff --git a/docs-internal/node-logic-flows/README.md b/docs-internal/node-logic-flows/README.md
index 460abaf3..24e5a7db 100644
--- a/docs-internal/node-logic-flows/README.md
+++ b/docs-internal/node-logic-flows/README.md
@@ -52,6 +52,7 @@ match its doc here, and each doc must still describe what the code does.
- [Mistral Chat Model (`mistralChatModel`)](./ai_chat_models/mistralChatModel.md)
- [Ollama Chat Model (`ollamaChatModel`)](./ai_chat_models/ollamaChatModel.md)
- [OpenAI Chat Model (`openaiChatModel`)](./ai_chat_models/openaiChatModel.md)
+- [OpenAI-compatible Chat Model (`openaiCompatibleChatModel`)](./ai_chat_models/openaiCompatibleChatModel.md)
- [OpenRouter Chat Model (`openrouterChatModel`)](./ai_chat_models/openrouterChatModel.md)
- [Sarvam AI Chat Model (`sarvamChatModel`)](./ai_chat_models/sarvamChatModel.md)
diff --git a/docs-internal/node-logic-flows/ai_chat_models/openaiCompatibleChatModel.md b/docs-internal/node-logic-flows/ai_chat_models/openaiCompatibleChatModel.md
new file mode 100644
index 00000000..dac61cc5
--- /dev/null
+++ b/docs-internal/node-logic-flows/ai_chat_models/openaiCompatibleChatModel.md
@@ -0,0 +1,111 @@
+# OpenAI-compatible Chat Model (`openaiCompatibleChatModel`)
+
+| Field | Value |
+|------|-------|
+| **Category** | ai_chat_models |
+| **Backend handler** | [`server/nodes/model/openai_compatible_chat_model/__init__.py`](../../../server/nodes/model/openai_compatible_chat_model/__init__.py) (dispatch via `BaseNode.execute()` -> `@Operation("chat")` in [`server/nodes/model/_base.py`](../../../server/nodes/model/_base.py)) |
+| **AI service** | [`server/services/ai.py::AIService.execute_chat`](../../../server/services/ai.py) |
+| **Tests** | [`server/tests/nodes/test_openai_compatible_node.py`](../../../server/tests/nodes/test_openai_compatible_node.py) |
+| **Skill (if any)** | n/a |
+| **Dual-purpose tool** | no (group `('model',)`) |
+
+## Purpose
+
+Chat with any OpenAI-compatible server the user has saved as a named endpoint under Credentials > OpenAI-compatible: llama.cpp, vLLM, a LiteLLM proxy, a second Ollama host, and so on (RFC-0003 D13). The node's `endpoint` parameter holds the endpoint's provider reference, `openai_compatible:<slug>`, which `detect_ai_provider` returns as the provider. `ChatUnifier` reads the endpoint's key from the `{ref}` credential row and its resolved base URL from `{ref}_proxy`, then calls the server through `OpenAIProvider`. The `ChatModelBase.chat` operation calls `AIService.execute_chat`, as for every chat model.
+
+## Inputs (handles)
+
+| Handle | Connection type | Required | Purpose |
+|--------|-----------------|----------|---------|
+| `input-main` | main | no | Upstream data; not consumed directly |
+
+## Parameters
+
+| Name | Type | Default | Required | displayOptions.show | Description |
+|------|------|---------|----------|---------------------|-------------|
+| `endpoint` | string | `""` | yes | - | The saved endpoint, as its reference `openai_compatible:<slug>`. Options from the `openaiCompatibleEndpoints` loader; rendered first |
+| `prompt` | string | `""` | yes | - | User message |
+| `system_prompt` | string | `""` | no | - | System prompt |
+| `model` | string | `""` (first option) | no | - | Options from the `openaiCompatibleModels` loader: the models stored when the endpoint was saved or refreshed. Reloads when `endpoint` changes. Open-world: the name is not pattern-checked |
+| `temperature` | number\|null | `null` | no | - | 0-2 |
+| `max_tokens` | number\|null | `null` | no | - | 1-200000. Unset: the model's registered max output (a quarter of its context, 512-4096, when only the context is known), else 2048 |
+| `top_p` | number\|null | `1.0` | no | - | |
+| `api_key` | string\|null | `null` (injected) | no | - | Injected from the endpoint's `{ref}` row: the key the user saved, or the placeholder `sk-no-key-required` |
+
+The field is named `endpoint`, not `provider`, on purpose: a `provider` sibling of `model` triggers the parameter panel's stored-key effect, which would copy the endpoint's key into this node's `api_key`.
+
+## Outputs (handles)
+
+| Handle | Shape | Description |
+|--------|-------|-------------|
+| `output-model` | object | Model output; standard envelope payload |
+
+### Output payload
+
+```ts
+{
+ response: string;
+ thinking: string | null;
+ thinking_enabled: boolean;
+ model: string;
+ provider: string; // the endpoint reference, openai_compatible:<slug>
+ finish_reason: string;
+ timestamp: string;
+ input: { prompt: string; system_prompt: string };
+}
+```
+
+Wrapped in `{ success, node_id, node_type, result, execution_time }`.
+
+## Logic Flow
+
+```mermaid
+flowchart TD
+ A[NodeExecutor dispatch -> BaseNode.execute] --> B[ChatModelBase.chat Operation]
+ B --> C[AIService.execute_chat]
+ C --> D{key + prompt?}
+ D -- no --> X[error envelope]
+ D -- yes --> E[detect_ai_provider -> parameters.endpoint<br/>openai_compatible:slug]
+ E --> F[ChatUnifier.chat -> registry.get_provider openai_compatible]
+ F --> G{ref_proxy row?}
+ G -- no --> Y[NodeUserError: endpoint not configured]
+ G -- yes --> H[OpenAIProvider<br/>base_url=resolved URL, key from the ref row]
+ H --> I[provider.chat]
+ I --> J[success envelope]
+ H -- typed SDK error --> Z[NodeUserError]
+```
+
+## Decision Logic
+
+- **Provider routing**: `detect_ai_provider` recognises this node type before the `lmstudio` / `ollama` tokens and returns `parameters["endpoint"]`. With no endpoint chosen it returns the bare `openai_compatible`, which holds no key, so the run stops before any request is sent.
+- **Base URL**: the URL resolved when the endpoint was saved (`services/llm/endpoints.py::resolve_base_url`); nothing is probed at call time. A call whose `{ref}_proxy` row is missing is refused instead of going to the SDK's default, api.openai.com.
+- **Key**: the stored key, or the declared placeholder `sk-no-key-required` (`auth.placeholder_key` in `llm_defaults.json`); never `None`, so the SDK never falls back to `OPENAI_API_KEY`.
+- **Open-world model name**: `openai_compatible` declares `open_world_models: true`, so `is_model_valid_for_provider` accepts any model id.
+- **Answers without a completion**: a 2xx with no `choices` and an error body raises `LLMError(PROTOCOL)`, which names the redacted URL instead of returning an empty response.
+
+## Side Effects
+
+- **Database writes**: none on the chat path. Saving or refreshing the endpoint under Credentials writes the `{ref}` and `{ref}_proxy` rows and registers per-model context and price in `DATA_DIR/local_models.json`.
+- **Broadcasts**: none on the chat path.
+- **External API calls**: `POST {resolved base URL}/chat/completions` via the `openai` SDK.
+- **File I/O**: none.
+- **Subprocess**: none.
+
+## External Dependencies
+
+- **Credentials**: the `OpenAICompatibleCredential` rows `openai_compatible:<slug>` (key, models, per-model params) and `openai_compatible:<slug>_proxy` (resolved URL).
+- **Services**: `services/llm/providers/openai.py` (`OpenAIProvider`); `services/llm/endpoints.py` (URL rooting at save time); `nodes/model/_local_validator.py` (the save path and kind detection); `nodes/model/_option_loaders.py` (the endpoint and model dropdowns).
+- **Python packages**: `openai`.
+- **Environment variables**: none.
+
+## Edge cases & known limits
+
+- **The model list is a snapshot**: it is read when the endpoint is saved or refreshed. A model the server loads later appears only after Refresh in the Credentials panel.
+- **Sizing**: the kind is detected once at save time. An endpoint detected as Ollama or LM Studio is sized by their SDK probes, llama.cpp by `n_ctx` from `/props`; any other server by vLLM's `max_model_len` on the model list, then LiteLLM's model table by model id. Failing all of these the `openai_compatible` defaults apply (8192 context, 2048 output).
+- **Removed endpoint**: a node that still names it has no key row, so the run fails before any request is sent. Pick another endpoint, or add it again.
+- **Error boundary**: typed OpenAI SDK failures become user-safe `NodeUserError` values in `ChatUnifier`, which `BaseNode.execute()` turns into the standard failure envelope.
+
+## Related
+
+- **Peer nodes**: [`ollamaChatModel`](./ollamaChatModel.md) and [`lmstudioChatModel`](./lmstudioChatModel.md), which share the save path; the cloud chat-model docs in this folder.
+- **Architecture docs**: [Native LLM SDK](../../native_llm_sdk.md) and [RFC-0003](../../../RFC-0003-OPENAI-COMPATIBLE-PROVIDER-CONTRACT.md).
diff --git a/server/config/credential_providers.json b/server/config/credential_providers.json
index 4fbc7dbf..5eae9962 100644
--- a/server/config/credential_providers.json
+++ b/server/config/credential_providers.json
@@ -231,6 +231,44 @@
],
"instructions": "Start the LM Studio local server from the Developer tab. API key is not required."
},
+ "openai_compatible": {
+ "_description": "Named OpenAI-compatible endpoints (RFC-0003 D13): llama.cpp, vLLM, a LiteLLM proxy, a second Ollama host, and so on. Each saved endpoint is its own provider reference, openai_compatible:<slug>, with the same two rows Ollama uses. The panel adds and refreshes endpoints through validate_api_key (api_key carries the Base URL, the other fields ride alongside under their own keys, and a refresh adds ref) and removes one through delete_api_key with its reference. `stored` and `endpoints` are filled on every catalogue fetch by OpenAICompatibleCredential.catalogue_extras, so the fields below are an add-endpoint form, never stored values.",
+ "name": "OpenAI-compatible",
+ "category": "ai",
+ "category_label": "AI Providers",
+ "color": "dracula.pink",
+ "kind": "apiKey",
+ "icon_ref": "lucide:Plug",
+ "fields": [
+ {
+ "key": "openai_compatible_url",
+ "label": "Base URL",
+ "type": "string",
+ "placeholder": "http://localhost:8080/v1",
+ "default": "http://localhost:8080",
+ "required": true,
+ "help": "Where the server listens. With or without /v1: saving asks the server which one it answers on."
+ },
+ {
+ "key": "openai_compatible_label",
+ "label": "Label",
+ "type": "text",
+ "placeholder": "e.g. home-llama",
+ "required": false,
+ "help": "How the endpoint appears in provider dropdowns. Defaults to the server's host and port."
+ },
+ {
+ "key": "openai_compatible_api_key",
+ "label": "API key (optional)",
+ "type": "password",
+ "secret": true,
+ "placeholder": "Only if the server requires one",
+ "required": false,
+ "help": "For servers started with an API key, such as vLLM or llama.cpp with --api-key, or a LiteLLM proxy virtual key."
+ }
+ ],
+ "instructions": "Point at any server that speaks the OpenAI API: llama.cpp (llama-server; start it with --jinja for tool calling), vLLM, a LiteLLM proxy, SGLang, LocalAI, or another Ollama or LM Studio host. Each saved endpoint appears in every agent's provider dropdown and in the OpenAI-compatible chat-model node."
+ },
"_cli_base": {
"_abstract": true,
"category": "ai",
diff --git a/server/config/pricing.json b/server/config/pricing.json
index 2c7dd9dc..cb4bb1fe 100644
--- a/server/config/pricing.json
+++ b/server/config/pricing.json
@@ -121,6 +121,10 @@
"lmstudio": {
"_description": "Local inference — same zero-cost convention as ollama.",
"_default": {"input": 0.00, "output": 0.00}
+ },
+ "openai_compatible": {
+ "_description": "Named OpenAI-compatible endpoints (openai_compatible:<slug>). Self-hosted servers cost nothing at the API layer. When saving an endpoint found a model id in the LiteLLM table, PricingService.get_pricing uses that per-model price from the model registry before this _default.",
+ "_default": {"input": 0.00, "output": 0.00}
}
},
diff --git a/server/constants.py b/server/constants.py
index c017e51d..343d4407 100644
--- a/server/constants.py
+++ b/server/constants.py
@@ -26,6 +26,8 @@ AI_CHAT_MODEL_TYPES: FrozenSet[str] = frozenset(
# format; routed through OpenAIProvider with a localhost base_url).
"ollamaChatModel",
"lmstudioChatModel",
+ # A user-named OpenAI-compatible endpoint (RFC-0003 D13).
+ "openaiCompatibleChatModel",
]
)
@@ -466,6 +468,13 @@ def detect_ai_provider(node_type: str, parameters: dict = None) -> str:
return "anthropic"
if "gemini" in nt:
return "gemini"
+ # A named endpoint: the node's ``endpoint`` parameter holds its
+ # provider reference, ``openai_compatible:<slug>``. Without one the
+ # bare id reaches the unifier, which reports "not configured".
+ if "openaicompatible" in nt:
+ from services.llm.config import ENDPOINT_PROVIDER
+
+ return (parameters or {}).get("endpoint") or ENDPOINT_PROVIDER
# Local-server providers — match the LMStudioChatModelNode /
# OllamaChatModelNode plugin types so the runtime path reads the
# correct {provider}_proxy credential and the openai SDK is pointed
diff --git a/server/nodes/agent/_provider.py b/server/nodes/agent/_provider.py
new file mode 100644
index 00000000..bbead7b8
--- /dev/null
+++ b/server/nodes/agent/_provider.py
@@ -0,0 +1,36 @@
+"""The ``provider`` field every LLM agent declares.
+
+A provider reference: a registered provider id, or a named
+OpenAI-compatible endpoint ``openai_compatible:<slug>`` (RFC-0003 D13).
+Options come from the ``aiProviders`` loader (``nodes/model``), so a newly
+saved endpoint appears without a NodeSpec change. The value travels as-is
+through key injection, Temporal payloads and the unifier, which is why it
+is one string rather than a provider plus a separate endpoint field.
+"""
+
+from __future__ import annotations
+
+from typing import Annotated
+
+from pydantic import AfterValidator, Field
+
+
+def _check_provider_ref(value: str) -> str:
+ """Accept a registered provider, or a named endpoint of the endpoint provider."""
+ from services.llm.config import ENDPOINT_PROVIDER, split_provider_ref
+ from services.llm.registry import has_provider
+
+ name, slug = split_provider_ref(value)
+ if not has_provider(name) or (name == ENDPOINT_PROVIDER) != bool(slug):
+ raise ValueError(f"unknown LLM provider {value!r}")
+ return value
+
+
+ProviderRef = Annotated[
+ str,
+ AfterValidator(_check_provider_ref),
+ Field(json_schema_extra={"loadOptionsMethod": "aiProviders"}),
+]
+
+
+__all__ = ["ProviderRef"]
diff --git a/server/nodes/agent/_specialized.py b/server/nodes/agent/_specialized.py
index 1296b758..abf2f4a0 100644
--- a/server/nodes/agent/_specialized.py
+++ b/server/nodes/agent/_specialized.py
@@ -11,13 +11,14 @@ one-file edit.
from __future__ import annotations
-from typing import Any, Literal, Optional
+from typing import Any, Optional
from pydantic import BaseModel, ConfigDict, Field
from services.plugin import ActionNode, NodeContext, Operation, TaskQueue
from ._handles import STD_AGENT_HINTS, std_agent_handles
+from ._provider import ProviderRef
class SpecializedAgentParams(BaseModel):
@@ -35,23 +36,7 @@ class SpecializedAgentParams(BaseModel):
"rows": 4,
},
)
- provider: Literal[
- "openai",
- "anthropic",
- "gemini",
- "openrouter",
- "xai",
- "groq",
- "cerebras",
- "deepseek",
- "kimi",
- "mistral",
- "sarvam",
- # Local-server providers — see ai_agent.Params for the proxy_url
- # rationale. Same fix; same reason.
- "ollama",
- "lmstudio",
- ] = "openai"
+ provider: ProviderRef = "openai"
model: str = Field(
default="",
json_schema_extra={"placeholder": "Select a model..."},
diff --git a/server/nodes/agent/ai_agent/__init__.py b/server/nodes/agent/ai_agent/__init__.py
index 88338d2a..5b0d754c 100644
--- a/server/nodes/agent/ai_agent/__init__.py
+++ b/server/nodes/agent/ai_agent/__init__.py
@@ -8,13 +8,14 @@ streaming + memory persistence + delegation.
from __future__ import annotations
-from typing import Any, Literal, Optional
+from typing import Any, Optional
from pydantic import BaseModel, ConfigDict, Field
from services.plugin import ActionNode, NodeContext, Operation, TaskQueue
from .._handles import STD_AGENT_HINTS, std_agent_handles
+from .._provider import ProviderRef
class AIAgentParams(BaseModel):
@@ -38,26 +39,7 @@ class AIAgentParams(BaseModel):
"rows": 4,
},
)
- provider: Literal[
- "openai",
- "anthropic",
- "gemini",
- "openrouter",
- "xai",
- "groq",
- "cerebras",
- "deepseek",
- "kimi",
- "mistral",
- "sarvam",
- # Local-server providers — agent execution reads
- # ``{provider}_proxy`` to point the native OpenAI client at the
- # user's localhost server. Without these entries the dropdown
- # silently falls back to ``"openai"`` and execute_agent ends
- # up calling api.openai.com instead.
- "ollama",
- "lmstudio",
- ] = "openai"
+ provider: ProviderRef = "openai"
model: str = Field(
default="",
json_schema_extra={"placeholder": "Select a model..."},
diff --git a/server/nodes/agent/chat_agent/__init__.py b/server/nodes/agent/chat_agent/__init__.py
index 8c555643..32bb0b54 100644
--- a/server/nodes/agent/chat_agent/__init__.py
+++ b/server/nodes/agent/chat_agent/__init__.py
@@ -7,13 +7,14 @@ applies a different default system message + skill prompt assembly.
from __future__ import annotations
-from typing import Any, Literal, Optional
+from typing import Any, Optional
from pydantic import BaseModel, ConfigDict, Field
from services.plugin import ActionNode, NodeContext, Operation, TaskQueue
from .._handles import STD_AGENT_HINTS, std_agent_handles
+from .._provider import ProviderRef
class ChatAgentParams(BaseModel):
@@ -25,23 +26,7 @@ class ChatAgentParams(BaseModel):
convention.
"""
- provider: Literal[
- "openai",
- "anthropic",
- "gemini",
- "openrouter",
- "xai",
- "groq",
- "cerebras",
- "deepseek",
- "kimi",
- "mistral",
- "sarvam",
- # Local-server providers — see ai_agent.Params for the proxy_url
- # rationale. Same fix; same reason.
- "ollama",
- "lmstudio",
- ] = "openai"
+ provider: ProviderRef = "openai"
model: str = Field(
default="",
json_schema_extra={"placeholder": "Select a model..."},
diff --git a/server/nodes/model/__init__.py b/server/nodes/model/__init__.py
index edd80913..11f04f92 100644
--- a/server/nodes/model/__init__.py
+++ b/server/nodes/model/__init__.py
@@ -1 +1,14 @@
-"""Plugins for the 'model' palette group. See ../__init__.py for the package layout."""
+"""Plugins for the 'model' palette group. See ../__init__.py for the package layout.
+
+Also owns the LLM provider dropdowns: the agent ``provider`` field and the
+OpenAI-compatible chat-model node load their options from here, because
+this folder owns the credentials those options are built from.
+"""
+
+from services.ws_handler_registry import register_option_loader
+
+from ._option_loaders import load_ai_providers, load_endpoint_models, load_endpoints
+
+register_option_loader("aiProviders", load_ai_providers)
+register_option_loader("openaiCompatibleEndpoints", load_endpoints)
+register_option_loader("openaiCompatibleModels", load_endpoint_models)
diff --git a/server/nodes/model/_credentials.py b/server/nodes/model/_credentials.py
index 8e351b38..5fdafe95 100644
--- a/server/nodes/model/_credentials.py
+++ b/server/nodes/model/_credentials.py
@@ -8,17 +8,19 @@ The native SDK client pulls the key directly from
:mod:`services.auth`; this class is the Credentials-modal + discovery
manifest, not the runtime client.
-Servers the user runs (Ollama, LM Studio) store their Base URL under
-``{provider}_proxy`` and need no key unless the server enforces one;
-without one, the placeholder the vendor documents is sent
-(``auth.placeholder_key`` in llm_defaults.json, resolved by
-:func:`services.llm.config.resolve_credential`). Both save through one
-path, ``_local_validator.save_llm_server`` (RFC-0003 §6).
+Servers the user runs (Ollama, LM Studio, and named OpenAI-compatible
+endpoints) store their Base URL under ``{provider}_proxy`` and need no
+key unless the server enforces one; without one, the placeholder the
+vendor documents is sent (``auth.placeholder_key`` in llm_defaults.json,
+resolved by :func:`services.llm.config.resolve_credential`). All three
+save through one path, ``_local_validator.save_llm_server`` (RFC-0003 §6).
"""
from __future__ import annotations
-from typing import Any, Dict
+import re
+from typing import Any, Dict, Optional
+from urllib.parse import urlsplit
from services.plugin.credential import ApiKeyCredential, ProbeResult
@@ -198,3 +200,98 @@ class LMStudioCredential(_LocalLLM):
display_name = "LM Studio"
icon = "lobehub:lmstudio"
docs_url = "https://lmstudio.ai/docs/local-server"
+
+
+# Mirrors the provider column limit: "openai_compatible:" + slug + "_proxy"
+# must fit in EncryptedAPIKey.provider (max 50 characters).
+_ENDPOINT_SLUG_MAX = 24
+_SLUG_PATTERN = re.compile(r"^[a-z0-9-]+$")
+
+
+def _endpoint_slug(label: str, base_url: str) -> str:
+ """Slug for a new endpoint: from its label, else from the URL's host."""
+ from slugify import slugify
+
+ source = label or urlsplit(base_url).netloc or base_url
+ return slugify(source, max_length=_ENDPOINT_SLUG_MAX, separator="-")
+
+
+def _rejected(message: str) -> Dict[str, Any]:
+ return {"provider": OpenAICompatibleCredential.id, "success": True, "valid": False, "message": message, "models": []}
+
+
+class OpenAICompatibleCredential(_LLMApiKey):
+ """Named OpenAI-compatible endpoints (RFC-0003 D13).
+
+ Any number of servers, each saved under its own provider reference
+ ``openai_compatible:<slug>`` with the same two rows Ollama uses
+ (``{ref}`` for the key, ``{ref}_proxy`` for the resolved URL). The
+ LiteLLM ``model_list`` shape: a label, a base URL, an optional key.
+
+ Saved and refreshed through the standard ``validate_api_key`` message:
+ ``api_key`` carries the Base URL, the other catalogue fields ride
+ alongside under their own keys, and ``ref`` marks a refresh of an
+ existing endpoint. Removed through ``delete_api_key`` with the
+ endpoint's reference, which also clears its URL row.
+ """
+
+ id = "openai_compatible"
+ display_name = "OpenAI-compatible"
+ docs_url = "https://docs.litellm.ai/docs/providers/openai_compatible"
+
+ # Field keys of this provider's entry in config/credential_providers.json.
+ label_field = "openai_compatible_label"
+ key_field = "openai_compatible_api_key"
+
+ @classmethod
+ async def validate(cls, data: Dict[str, Any]) -> Dict[str, Any]:
+ from services.llm.config import endpoint_ref, split_provider_ref
+ from services.llm.endpoints import SERVER_META_KEY, base_url_key
+ from services.plugin.deps import get_auth_service
+
+ from ._local_validator import save_llm_server
+
+ auth = get_auth_service()
+ label = (data.get(cls.label_field) or "").strip()
+ user_key: Optional[str] = (data.get(cls.key_field) or "").strip() or None
+ ref = (data.get("ref") or "").strip()
+
+ if ref:
+ # Refresh. The stored URL is authoritative: the panel only ever
+ # sees the redacted form, which may have lost userinfo.
+ name, slug = split_provider_ref(ref)
+ if name != cls.id or not _SLUG_PATTERN.match(slug):
+ return _rejected(f"Unknown endpoint {ref!r}.")
+ candidate = await auth.get_api_key(base_url_key(ref))
+ if not candidate:
+ return _rejected(f"The endpoint '{slug}' no longer exists.")
+ user_key = user_key or await auth.get_api_key(ref)
+ if not label:
+ meta = (await auth.get_model_params(ref)).get(SERVER_META_KEY) or {}
+ label = meta.get("label") or slug
+ else:
+ candidate = (data.get("api_key") or "").strip()
+ slug = _endpoint_slug(label, candidate)
+ if not slug:
+ return _rejected("Give the endpoint a label with at least one letter or digit.")
+ ref = endpoint_ref(slug)
+ if await auth.get_api_key(ref):
+ return _rejected(
+ f"An endpoint named '{slug}' already exists. Choose another label, "
+ "or refresh the existing endpoint."
+ )
+ label = label or slug
+
+ return await save_llm_server(ref, candidate, user_key, display=label, label=label)
+
+ @classmethod
+ async def catalogue_extras(cls) -> Dict[str, Any]:
+ """``stored`` (any endpoint saved) plus the endpoint list for the panel."""
+ from services.llm.endpoints import list_endpoints
+ from services.plugin.deps import get_auth_service
+
+ endpoints = [
+ {"ref": e.ref, "label": e.label, "base_url": e.base_url, "kind": e.kind, "model_count": len(e.models)}
+ for e in await list_endpoints(get_auth_service())
+ ]
+ return {"stored": bool(endpoints), "endpoints": endpoints}
diff --git a/server/nodes/model/_option_loaders.py b/server/nodes/model/_option_loaders.py
new file mode 100644
index 00000000..a7c18e11
--- /dev/null
+++ b/server/nodes/model/_option_loaders.py
@@ -0,0 +1,57 @@
+"""Dropdown loaders for LLM provider and endpoint selection.
+
+The provider dropdown was a ``Literal`` on each agent's Params. A saved
+endpoint cannot be a literal, so the options come from here instead and a
+new endpoint appears without a NodeSpec change. Imports no LLM SDK: these
+run on every parameter-panel render.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List
+
+
+async def _endpoints():
+ from services.llm.endpoints import list_endpoints
+ from services.plugin.deps import get_auth_service
+
+ return await list_endpoints(get_auth_service())
+
+
+async def load_ai_providers(params: Dict[str, Any] | None = None) -> List[Dict[str, str]]:
+ """Every registered provider, then one option per saved endpoint.
+
+ Ordered as ``llm_defaults.json`` lists the providers, so the order is
+ declared rather than an accident of import order. The bare
+ ``openai_compatible`` id is not an option: it only exists as endpoints.
+ """
+ from services.llm.config import ENDPOINT_PROVIDER, LLM_DEFAULTS
+ from services.llm.registry import all_providers
+
+ blocks = LLM_DEFAULTS.get("providers", {})
+ registered = set(all_providers()) - {ENDPOINT_PROVIDER}
+ ordered = [name for name in blocks if name in registered]
+ ordered += sorted(registered - set(ordered))
+ options = [{"name": blocks.get(name, {}).get("display_name") or name, "value": name} for name in ordered]
+ options += [{"name": f"{e.label} (OpenAI-compatible)", "value": e.ref} for e in await _endpoints()]
+ return options
+
+
+async def load_endpoints(params: Dict[str, Any] | None = None) -> List[Dict[str, str]]:
+ """Saved endpoints only, for the OpenAI-compatible chat-model node."""
+ return [{"name": e.label, "value": e.ref} for e in await _endpoints()]
+
+
+async def load_endpoint_models(params: Dict[str, Any] | None = None) -> List[Dict[str, str]]:
+ """Models the selected endpoint listed when it was saved."""
+ from services.llm.config import ENDPOINT_PROVIDER, split_provider_ref
+ from services.plugin.deps import get_auth_service
+
+ ref = (params or {}).get("endpoint") or ""
+ name, slug = split_provider_ref(ref)
+ if name != ENDPOINT_PROVIDER or not slug:
+ return []
+ return [{"name": model, "value": model} for model in await get_auth_service().get_stored_models(ref)]
+
+
+__all__ = ["load_ai_providers", "load_endpoint_models", "load_endpoints"]
diff --git a/server/nodes/model/openai_compatible_chat_model/__init__.py b/server/nodes/model/openai_compatible_chat_model/__init__.py
new file mode 100644
index 00000000..e7ef3128
--- /dev/null
+++ b/server/nodes/model/openai_compatible_chat_model/__init__.py
@@ -0,0 +1,49 @@
+"""OpenAI-compatible chat model: one of the user's named endpoints.
+
+Pick an endpoint saved under Credentials > OpenAI-compatible (llama.cpp,
+vLLM, a LiteLLM proxy, a second Ollama host, ...) and a model it serves.
+The endpoint's reference, ``openai_compatible:<slug>``, is the provider the
+runtime resolves (``constants.detect_ai_provider``), so its stored key and
+resolved URL are used with no per-node configuration.
+"""
+
+from pydantic import BaseModel, Field
+
+from .._base import ChatModelBase, ChatModelParams
+from .._credentials import OpenAICompatibleCredential
+
+
+class _EndpointField(BaseModel):
+ # Pydantic lays fields out in reversed-MRO order, so declaring the
+ # endpoint on a base listed after ChatModelParams renders it first,
+ # above the prompt and model, where it is chosen.
+ endpoint: str = Field(
+ default="",
+ description="Saved endpoint to send requests to.",
+ json_schema_extra={"loadOptionsMethod": "openaiCompatibleEndpoints"},
+ )
+
+
+class OpenAICompatibleChatModelParams(ChatModelParams, _EndpointField):
+ # The field is ``endpoint``, not ``provider``: a ``provider`` sibling of
+ # ``model`` triggers the parameter panel's stored-key effect, which
+ # would copy the endpoint's key into this node's ``api_key``.
+ model: str = Field(
+ default="",
+ json_schema_extra={
+ "placeholder": "Select a model...",
+ "loadOptionsMethod": "openaiCompatibleModels",
+ "loadOptionsDependsOn": ["endpoint"],
+ },
+ )
+
+
+class OpenAICompatibleChatModelNode(ChatModelBase):
+ type = "openaiCompatibleChatModel"
+ display_name = "OpenAI-compatible"
+ subtitle = "Chat Model"
+ group = ("model",)
+ description = "Any OpenAI-compatible server saved as a named endpoint: llama.cpp, vLLM, a LiteLLM proxy, and more"
+
+ credentials = (OpenAICompatibleCredential,)
+ Params = OpenAICompatibleChatModelParams
diff --git a/server/nodes/model/openai_compatible_chat_model/meta.json b/server/nodes/model/openai_compatible_chat_model/meta.json
new file mode 100644
index 00000000..66f2f1bc
--- /dev/null
+++ b/server/nodes/model/openai_compatible_chat_model/meta.json
@@ -0,0 +1,4 @@
+{
+ "color": "#ff79c6",
+ "icon": "lucide:Plug"
+}
diff --git a/server/routers/websocket.py b/server/routers/websocket.py
index d4c9ab57..11ca1a3b 100644
--- a/server/routers/websocket.py
+++ b/server/routers/websocket.py
@@ -393,6 +393,7 @@ async def handle_get_credential_catalogue(data: Dict[str, Any], websocket: WebSo
client-side credential checks needed.
"""
from services.credential_registry import get_credential_registry
+ from services.plugin.credential import CREDENTIAL_REGISTRY
registry = get_credential_registry()
since = data.get("since")
@@ -449,6 +450,16 @@ async def handle_get_credential_catalogue(data: Dict[str, Any], websocket: WebSo
else:
provider["account_label"] = None
+ # A credential whose state is not one row per provider id (e.g.
+ # several named OpenAI-compatible endpoints) contributes its own
+ # fields, and may replace ``stored``. Declared on the plugin's
+ # Credential class, so this handler names no provider.
+ cred_cls = CREDENTIAL_REGISTRY.get(pid)
+ if cred_cls is not None:
+ extras = await cred_cls.catalogue_extras()
+ if extras:
+ provider.update(extras)
+
return catalogue
diff --git a/server/services/auth.py b/server/services/auth.py
index 5421b60c..ebe6c348 100644
--- a/server/services/auth.py
+++ b/server/services/auth.py
@@ -234,6 +234,25 @@ class AuthService:
logger.error("Failed to list key scopes", provider=provider, error=str(e))
return []
+ async def list_api_key_providers(self, session_id: str = "default") -> List[str]:
+ """List every provider name holding a key in one session.
+
+ The other axis of :meth:`list_key_scopes`. Reads the database, not
+ the lazily populated cache. Used to enumerate named OpenAI-compatible
+ endpoints, whose provider names are ``openai_compatible:<slug>``.
+
+ Args:
+ session_id: Session identifier
+
+ Returns:
+ Provider names, empty on failure
+ """
+ try:
+ return await self.credentials_db.list_api_keys(session_id)
+ except Exception as e:
+ logger.error("Failed to list API key providers", session_id=session_id, error=str(e))
+ return []
+
async def get_stored_models(self, provider: str, session_id: str = "default") -> List[str]:
"""Get stored models for provider.
diff --git a/server/services/llm/endpoints.py b/server/services/llm/endpoints.py
index d2ed0c1c..3920706e 100644
--- a/server/services/llm/endpoints.py
+++ b/server/services/llm/endpoints.py
@@ -25,10 +25,14 @@ pure helpers costs nothing.
from __future__ import annotations
from dataclasses import dataclass, field
-from typing import Any, List, Optional
+from typing import TYPE_CHECKING, Any, List, Optional
from urllib.parse import urlsplit, urlunsplit
from core.logging import get_logger
+from services.llm.config import ENDPOINT_PROVIDER, split_provider_ref
+
+if TYPE_CHECKING:
+ from services.auth import AuthService
logger = get_logger(__name__)
@@ -168,11 +172,60 @@ async def resolve_base_url(candidate: str, *, api_key: str, timeout: float = 10.
)
+async def list_endpoint_refs(auth: "AuthService") -> List[str]:
+ """Every saved named endpoint, as provider references, sorted.
+
+ Enumerated from the credential rows themselves rather than a separate
+ index, so there is nothing that can disagree with them.
+ """
+ prefix = f"{ENDPOINT_PROVIDER}:"
+ return sorted(
+ provider
+ for provider in await auth.list_api_key_providers()
+ if provider.startswith(prefix) and not provider.endswith(BASE_URL_SUFFIX)
+ )
+
+
+@dataclass(frozen=True)
+class SavedEndpoint:
+ """A saved named endpoint, as the credentials panel and dropdowns show it.
+
+ ``base_url`` is the redacted form stored for display; the full URL
+ stays in the encrypted ``{ref}_proxy`` row.
+ """
+
+ ref: str
+ label: str
+ base_url: str
+ kind: str
+ models: List[str]
+
+
+async def list_endpoints(auth: "AuthService") -> List[SavedEndpoint]:
+ """Every saved named endpoint with its display metadata and models."""
+ endpoints: List[SavedEndpoint] = []
+ for ref in await list_endpoint_refs(auth):
+ meta = (await auth.get_model_params(ref)).get(SERVER_META_KEY) or {}
+ endpoints.append(
+ SavedEndpoint(
+ ref=ref,
+ label=meta.get("label") or split_provider_ref(ref)[1],
+ base_url=meta.get("base_url", ""),
+ kind=meta.get("kind", "generic"),
+ models=list(await auth.get_stored_models(ref)),
+ )
+ )
+ return endpoints
+
+
__all__ = [
"BASE_URL_SUFFIX",
"SERVER_META_KEY",
"ResolvedBaseUrl",
+ "SavedEndpoint",
"base_url_key",
+ "list_endpoint_refs",
+ "list_endpoints",
"redact_url",
"resolve_base_url",
]
diff --git a/server/services/llm/providers/_compat.py b/server/services/llm/providers/_compat.py
index 73ff7956..4377c5e6 100644
--- a/server/services/llm/providers/_compat.py
+++ b/server/services/llm/providers/_compat.py
@@ -13,6 +13,10 @@ Phase D (see the ``_COMPAT_PROVIDERS`` comment below); Sarvam followed and
brought the ``supports_model_listing`` flag with it — a provider can be
wire-compatible yet ship no model-list route, which ``OpenAIProvider``
handles generically off that JSON key.
+
+One more registration, ``openai_compatible``, carries no ``base_url`` at
+all: it serves every user-named endpoint, each of which stores its own
+resolved URL (RFC-0003 D13).
"""
from __future__ import annotations
@@ -20,6 +24,7 @@ from __future__ import annotations
from typing import Tuple
from core.logging import get_logger
+from services.llm.config import ENDPOINT_PROVIDER
from services.llm.providers.openai import OpenAIProvider
from services.llm.registry import ProviderSpec, register_provider
@@ -111,4 +116,23 @@ def _register_compat_providers() -> None:
)
+def _register_endpoint_provider() -> None:
+ """Register the one provider behind every named endpoint (RFC-0003 D13).
+
+ Deliberately no ``base_url``: each endpoint's resolved URL lives on its
+ own ``openai_compatible:<slug>_proxy`` row, and the unifier refuses the
+ call when that row is missing rather than let the SDK default to
+ api.openai.com.
+ """
+ register_provider(
+ ProviderSpec(
+ name=ENDPOINT_PROVIDER,
+ factory=OpenAIProvider,
+ sdk_exception_refs=("openai:OpenAIError",),
+ client_kwargs={"provider_name": ENDPOINT_PROVIDER},
+ )
+ )
+
+
_register_compat_providers()
+_register_endpoint_provider()
diff --git a/server/services/plugin/credential.py b/server/services/plugin/credential.py
index 25ececa7..36c6e232 100644
--- a/server/services/plugin/credential.py
+++ b/server/services/plugin/credential.py
@@ -340,6 +340,19 @@ class Credential:
"""
raise NotImplementedError(f"Credential subclass {cls.__name__} must override _probe()")
+ @classmethod
+ async def catalogue_extras(cls) -> Optional[Dict[str, Any]]:
+ """Extra catalogue fields computed from stored state, or ``None``.
+
+ ``handle_get_credential_catalogue`` merges the result into this
+ provider's entry after computing ``stored`` the default way, so an
+ override may also replace ``stored``. For credentials whose state is
+ not one row per provider id, such as a provider holding several
+ named rows. Must not do network IO: the catalogue is fetched on
+ every modal open.
+ """
+ return None
+
class OAuth2Credential(Credential):
"""OAuth 2.0 with refresh-token support.
diff --git a/server/services/pricing.py b/server/services/pricing.py
index ee4d0a65..bfa31b19 100644
--- a/server/services/pricing.py
+++ b/server/services/pricing.py
@@ -122,16 +122,21 @@ class PricingService:
Falls back to '_default' if no match found.
Args:
- provider: Provider name (openai, anthropic, gemini, groq, cerebras, openrouter)
+ provider: Provider reference (openai, anthropic, ..., or a named
+ endpoint ``openai_compatible:<slug>``, which reads the
+ ``openai_compatible`` block)
model: Model name or ID
Returns:
ModelPricing with rates per million tokens
"""
+ from services.llm.config import split_provider_ref
+
provider_lower = provider.lower()
model_lower = model.lower() if model else ""
+ name, endpoint_slug = split_provider_ref(provider_lower)
- provider_pricing = self._llm_registry.get(provider_lower, {})
+ provider_pricing = self._llm_registry.get(name, {})
# Try exact match first
if model_lower in provider_pricing:
@@ -147,6 +152,16 @@ class PricingService:
if model_key != "_default" and model_key in model_lower:
return pricing
+ # A named endpoint's per-model price, recorded when it was saved
+ # (from the LiteLLM table). Scoped to endpoints so every other
+ # provider keeps its curated pricing.json behaviour.
+ if endpoint_slug:
+ from services.model_registry import get_model_registry
+
+ info = get_model_registry().get_model_info(model or "", provider)
+ if info and (info.input_price_per_mtok or info.output_price_per_mtok):
+ return ModelPricing(info.input_price_per_mtok, info.output_price_per_mtok)
+
# Fall back to provider default
default_pricing = provider_pricing.get("_default")
if default_pricing:
diff --git a/server/services/rlm/service.py b/server/services/rlm/service.py
index 34000d79..3dd54a4c 100644
--- a/server/services/rlm/service.py
+++ b/server/services/rlm/service.py
@@ -92,6 +92,18 @@ class RLMService:
provider = parameters.get("provider", "openai")
model = parameters.get("model", "")
+ # RLM builds its own clients from constants.PROVIDER_TO_BACKEND
+ # and never reads a user base URL, so any other provider (a
+ # local server, a named endpoint) would silently be sent to
+ # api.openai.com with that provider's key.
+ from .constants import PROVIDER_TO_BACKEND
+
+ if provider not in PROVIDER_TO_BACKEND:
+ raise ValueError(
+ f"The RLM Agent cannot run on provider {provider!r}. "
+ f"Choose one of: {', '.join(sorted(PROVIDER_TO_BACKEND))}."
+ )
+
if not model or not is_model_valid_for_provider(model, provider):
model = await get_default_model_async(provider, database)
diff --git a/server/services/settings/handlers.py b/server/services/settings/handlers.py
index 7ad21f97..86121f80 100644
--- a/server/services/settings/handlers.py
+++ b/server/services/settings/handlers.py
@@ -178,6 +178,23 @@ async def handle_get_validated_ai_providers(
}
)
+ # Each named endpoint is its own provider, so one can be the global
+ # default (RFC-0003 D13). The bare ``openai_compatible`` id holds no
+ # key, so the loop above already skipped it.
+ from services.llm.endpoints import list_endpoints
+
+ for endpoint in await list_endpoints(auth_service):
+ endpoint_defaults = await database.get_provider_defaults(endpoint.ref)
+ providers.append(
+ {
+ "provider": endpoint.ref,
+ "display_name": f"{endpoint.label} (OpenAI-compatible)",
+ "models": endpoint.models,
+ "popular_models": [],
+ "default_model": (endpoint_defaults or {}).get("default_model", ""),
+ }
+ )
+
user_id = data.get("user_id", "default")
settings = await database.get_user_settings(user_id)
global_provider = settings.get("default_llm_provider") if settings else None
diff --git a/server/tests/credentials/test_openai_compatible_credential.py b/server/tests/credentials/test_openai_compatible_credential.py
new file mode 100644
index 00000000..36320846
--- /dev/null
+++ b/server/tests/credentials/test_openai_compatible_credential.py
@@ -0,0 +1,151 @@
+"""RFC-0003 D13 / AG15: named OpenAI-compatible endpoints as credentials.
+
+Adding and refreshing go through ``validate_api_key``; removing through
+``delete_api_key``; the panel's list rides the catalogue. Storage is the
+real encrypted credentials DB (``auth_service`` fixture) wherever rows
+matter; the save path itself is covered in tests/nodes/test_local_llm_save.py.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+
+from nodes.model import _credentials as creds
+from nodes.model._credentials import OpenAICompatibleCredential
+
+pytestmark = pytest.mark.credentials
+
+REF = "openai_compatible:home-vllm"
+
+
+def auth(auth_service, monkeypatch):
+ monkeypatch.setattr("services.plugin.deps.get_auth_service", lambda: auth_service)
+ return auth_service
+
+
+def save(monkeypatch) -> AsyncMock:
+ fake = AsyncMock(return_value={"success": True, "valid": True})
+ monkeypatch.setattr("nodes.model._local_validator.save_llm_server", fake)
+ return fake
+
+
+async def _store_endpoint(auth, ref: str, url: str, key: str, label: str, models=("m1",)) -> None:
+ await auth.store_api_key(provider=f"{ref}_proxy", api_key=url, models=[])
+ await auth.store_api_key(
+ provider=ref,
+ api_key=key,
+ models=list(models),
+ model_params={"_endpoint": {"label": label, "base_url": url.split("@")[-1], "kind": "generic"}},
+ )
+
+
+class TestAdd:
+ async def test_the_label_becomes_the_slug_of_the_reference(self, auth, save):
+ await OpenAICompatibleCredential.validate(
+ {"api_key": "http://host:8000", "openai_compatible_label": "Home vLLM", "openai_compatible_api_key": "sk-1"}
+ )
+
+ save.assert_awaited_once_with(REF, "http://host:8000", "sk-1", display="Home vLLM", label="Home vLLM")
+
+ async def test_without_a_label_the_slug_comes_from_the_host(self, auth, save):
+ await OpenAICompatibleCredential.validate({"api_key": "http://localhost:8080"})
+
+ ref, url, key = save.await_args.args
+ assert (ref, url, key) == ("openai_compatible:localhost-8080", "http://localhost:8080", None)
+
+ async def test_a_slug_fits_the_provider_column_with_its_url_suffix(self, auth, save):
+ await OpenAICompatibleCredential.validate({"api_key": "http://h", "openai_compatible_label": "a very long endpoint label indeed"})
+
+ ref = save.await_args.args[0]
+ assert len(f"{ref}_proxy") <= 50
+
+ async def test_an_existing_label_is_rejected_not_overwritten(self, auth, save):
+ await _store_endpoint(auth, REF, "http://host:8000/v1", "sk-old", "Home vLLM")
+
+ result = await OpenAICompatibleCredential.validate({"api_key": "http://other", "openai_compatible_label": "home vllm"})
+
+ assert result["valid"] is False and "already exists" in result["message"]
+ save.assert_not_awaited()
+
+ async def test_a_label_with_no_letter_or_digit_is_rejected(self, auth, save):
+ result = await OpenAICompatibleCredential.validate({"api_key": "http://h", "openai_compatible_label": "!!!"})
+
+ assert result["valid"] is False
+ save.assert_not_awaited()
+
+
+class TestRefresh:
+ async def test_uses_the_stored_url_and_key_not_what_the_panel_sent(self, auth, save):
+ await _store_endpoint(auth, REF, "http://user:pw@host:8000/v1", "sk-stored", "Home vLLM")
+
+ await OpenAICompatibleCredential.validate({"api_key": "http://host:8000/v1", "ref": REF})
+
+ save.assert_awaited_once_with(
+ REF, "http://user:pw@host:8000/v1", "sk-stored", display="Home vLLM", label="Home vLLM"
+ )
+
+ @pytest.mark.parametrize("ref", ["openai_compatible:gone", "ollama", "openai_compatible:Bad_Slug"])
+ async def test_an_unknown_or_malformed_reference_is_rejected(self, auth, save, ref):
+ result = await OpenAICompatibleCredential.validate({"api_key": "http://h", "ref": ref})
+
+ assert result["valid"] is False
+ save.assert_not_awaited()
+
+
+class TestCatalogue:
+ async def test_lists_saved_endpoints_and_marks_the_provider_stored(self, auth):
+ await _store_endpoint(auth, REF, "http://host:8000/v1", "sk-1", "Home vLLM", models=("a", "b"))
+ await _store_endpoint(auth, "openai_compatible:lab", "http://lab:8080", "sk-no-key-required", "Lab")
+ await auth.store_api_key(provider="ollama", api_key="ollama", models=["x"])
+
+ extras = await OpenAICompatibleCredential.catalogue_extras()
+
+ assert extras["stored"] is True
+ assert extras["endpoints"] == [
+ {"ref": REF, "label": "Home vLLM", "base_url": "http://host:8000/v1", "kind": "generic", "model_count": 2},
+ {"ref": "openai_compatible:lab", "label": "Lab", "base_url": "http://lab:8080", "kind": "generic", "model_count": 1},
+ ]
+
+ async def test_no_endpoints_means_not_stored(self, auth):
+ assert await OpenAICompatibleCredential.catalogue_extras() == {"stored": False, "endpoints": []}
+
+ async def test_the_catalogue_handler_merges_the_extras(self, auth, monkeypatch):
+ import routers.websocket as ws
+
+ await _store_endpoint(auth, REF, "http://host:8000/v1", "sk-1", "Home vLLM")
+ monkeypatch.setattr(ws.container, "auth_service", lambda: auth)
+
+ catalogue = await ws.handle_get_credential_catalogue({}, SimpleNamespace())
+
+ entry = next(p for p in catalogue["providers"] if p["id"] == "openai_compatible")
+ assert entry["stored"] is True
+ assert [e["ref"] for e in entry["endpoints"]] == [REF]
+
+
+class TestDelete:
+ async def test_deleting_an_endpoint_removes_its_url_row_too(self, auth, monkeypatch):
+ from services.credentials import handlers
+
+ await _store_endpoint(auth, REF, "http://host:8000/v1", "sk-1", "Home vLLM")
+ broadcaster = MagicMock()
+ broadcaster.update_api_key_status = AsyncMock()
+ broadcaster.broadcast_credential_event = AsyncMock()
+ monkeypatch.setattr(handlers.container, "auth_service", lambda: auth)
+ monkeypatch.setattr(handlers, "get_status_broadcaster", lambda: broadcaster)
+
+ await handlers.handle_delete_api_key({"provider": REF}, SimpleNamespace())
+
+ assert await auth.get_api_key(REF) is None
+ assert await auth.get_api_key(f"{REF}_proxy") is None
+ broadcaster.broadcast_credential_event.assert_awaited_once()
+
+
+def test_the_credential_is_registered_for_dispatch():
+ from services.plugin.credential import CREDENTIAL_REGISTRY
+
+ assert CREDENTIAL_REGISTRY["openai_compatible"] is creds.OpenAICompatibleCredential
diff --git a/server/tests/llm/test_plugin_shape.py b/server/tests/llm/test_plugin_shape.py
index 6cda29d8..5ccaa14c 100644
--- a/server/tests/llm/test_plugin_shape.py
+++ b/server/tests/llm/test_plugin_shape.py
@@ -144,19 +144,29 @@ def test_every_provider_declares_sdk_exception_types():
)
-def test_all_registered_providers_are_agent_selectable():
- """Agent surfaces must not silently omit a native provider."""
+async def test_all_registered_providers_are_agent_selectable():
+ """Agent surfaces must not silently omit a native provider.
- from typing import get_args
+ The ``provider`` field is loader-driven (RFC-0003 D13): a saved endpoint
+ cannot be a ``Literal``. Every agent Params points at the ``aiProviders``
+ loader, and the loader offers every registered provider except the bare
+ endpoint id, which is only selectable as a saved endpoint.
+ """
+ from unittest.mock import AsyncMock, patch
from nodes.agent._specialized import SpecializedAgentParams
from nodes.agent.ai_agent import AIAgentParams
from nodes.agent.chat_agent import ChatAgentParams
+ from nodes.model._option_loaders import load_ai_providers
+ from services.llm.config import ENDPOINT_PROVIDER
- expected = set(all_providers())
for params_type in (AIAgentParams, ChatAgentParams, SpecializedAgentParams):
- annotation = params_type.model_fields["provider"].annotation
- assert set(get_args(annotation)) == expected
+ schema = params_type.model_json_schema()["properties"]["provider"]
+ assert schema.get("loadOptionsMethod") == "aiProviders", params_type.__name__
+
+ with patch("nodes.model._option_loaders._endpoints", AsyncMock(return_value=[])):
+ options = await load_ai_providers({})
+ assert {option["value"] for option in options} == set(all_providers()) - {ENDPOINT_PROVIDER}
# ---------------------------------------------------------------------------
diff --git a/server/tests/llm/test_provider_refs.py b/server/tests/llm/test_provider_refs.py
new file mode 100644
index 00000000..7abf1708
--- /dev/null
+++ b/server/tests/llm/test_provider_refs.py
@@ -0,0 +1,196 @@
+"""RFC-0003 D6/D7/D13 at call time: provider references and credentials.
+
+A named endpoint is the provider reference ``openai_compatible:<slug>``.
+Its credential rows are keyed by the full reference; everything read from
+``llm_defaults.json`` resolves through the shared ``openai_compatible``
+block (AG16). No path may hand an SDK an empty key (AG8).
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+import respx
+
+import services.llm # noqa: F401 — populate the registry
+from services.llm.config import (
+ ENDPOINT_PROVIDER,
+ curated_models,
+ endpoint_ref,
+ get_default_model,
+ is_model_valid_for_provider,
+ resolve_credential,
+ split_provider_ref,
+ supports_model_listing,
+)
+from services.llm.protocol import LLMResponse, Message
+from services.llm.registry import ProviderSpec, all_providers
+from services.llm.unifier import ChatUnifier
+from services.plugin import NodeUserError
+
+REF = "openai_compatible:home"
+SERVER_DIR = Path(__file__).resolve().parents[2]
+
+
+def _unifier(get_api_key) -> ChatUnifier:
+ auth = MagicMock()
+ auth.get_api_key = get_api_key
+ return ChatUnifier(defaults={"providers": {}}, auth_service=auth)
+
+
+async def _chat(unifier: ChatUnifier, provider: str, api_key: str = "k"):
+ return await unifier.chat(
+ provider=provider,
+ api_key=api_key,
+ messages=[Message(role="user", content="hi")],
+ model="m",
+ )
+
+
+class TestReferences:
+ def test_split_and_build_round_trip(self):
+ assert split_provider_ref("ollama") == ("ollama", "")
+ assert split_provider_ref(REF) == (ENDPOINT_PROVIDER, "home")
+ assert endpoint_ref("home") == REF
+ assert split_provider_ref("") == ("", "")
+
+
+class TestResolveCredential:
+ def test_a_stored_key_wins(self):
+ assert resolve_credential("ollama", "real-key") == "real-key"
+
+ @pytest.mark.parametrize(
+ ("provider", "placeholder"),
+ [("ollama", "ollama"), ("lmstudio", "lm-studio"), (REF, "sk-no-key-required")],
+ )
+ def test_keyless_servers_get_the_placeholder_their_vendor_documents(self, provider, placeholder):
+ assert resolve_credential(provider, None) == placeholder
+ assert resolve_credential(provider, "") == placeholder
+
+ @pytest.mark.parametrize("provider", ["openai", "anthropic", "deepseek"])
+ def test_no_key_and_no_placeholder_is_an_error_not_none(self, provider):
+ with pytest.raises(ValueError):
+ resolve_credential(provider, None)
+
+
+class TestReferencesReadTheSharedBlock:
+ def test_default_model_is_not_a_cloud_fallback(self):
+ # An unknown provider would get the "gpt-5.2" literal.
+ assert get_default_model(REF) == ""
+
+ def test_listing_is_supported_and_nothing_is_curated(self):
+ assert supports_model_listing(REF) is True
+ assert curated_models(REF) == []
+
+ @pytest.mark.parametrize("provider", [REF, "ollama", "lmstudio", "openrouter", "groq"])
+ def test_open_world_providers_accept_any_model_id(self, provider):
+ assert is_model_valid_for_provider("owner/any-model:7b", provider) is True
+
+ def test_closed_world_providers_still_pattern_check(self):
+ assert is_model_valid_for_provider("gpt-5.6-sol", "openai") is True
+ assert is_model_valid_for_provider("llama3", "openai") is False
+
+
+class TestRegistryFallbacks:
+ @pytest.fixture
+ def registry(self, tmp_path, monkeypatch):
+ import services.model_registry as mr
+
+ monkeypatch.setattr(mr, "_local_models_path", lambda: tmp_path / "local_models.json")
+ svc = mr.ModelRegistryService()
+ svc._load_llm_defaults()
+ return svc
+
+ def test_an_unsized_endpoint_model_gets_the_conservative_defaults(self, registry):
+ assert registry.get_context_length("unknown-model", REF) == 8192
+ assert registry.get_max_output_tokens("unknown-model", REF) == 2048
+ assert registry.get_temperature_range("unknown-model", REF) == (0.0, 2.0)
+
+ def test_a_model_registered_at_save_time_wins(self, registry):
+ registry.register_local_model(REF, "qwen3", {"context_length": 32768})
+
+ assert registry.get_context_length("qwen3", REF) == 32768
+ # Another endpoint serving the same id is a different server.
+ assert registry.get_context_length("qwen3", "openai_compatible:other") == 8192
+
+
+class TestPricing:
+ @pytest.fixture
+ def pricing(self):
+ # tests/conftest.py stubs services.pricing; load the real module.
+ spec = importlib.util.spec_from_file_location("_real_pricing", SERVER_DIR / "services" / "pricing.py")
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module.PricingService()
+
+ def test_an_endpoint_is_free_by_default(self, pricing):
+ price = pricing.get_pricing(REF, "qwen3")
+ assert (price.input_per_mtok, price.output_per_mtok) == (0.0, 0.0)
+
+ def test_an_endpoint_uses_the_price_recorded_when_it_was_saved(self, pricing):
+ info = MagicMock(input_price_per_mtok=3.0, output_price_per_mtok=15.0)
+ registry = MagicMock()
+ registry.get_model_info.return_value = info
+ with patch("services.model_registry.get_model_registry", return_value=registry):
+ price = pricing.get_pricing(REF, "gpt-4o")
+ assert (price.input_per_mtok, price.output_per_mtok) == (3.0, 15.0)
+ registry.get_model_info.assert_called_once_with("gpt-4o", REF)
+
+ def test_other_providers_keep_their_curated_prices(self, pricing):
+ price = pricing.get_pricing("openai", "gpt-6-astra")
+ assert (price.input_per_mtok, price.output_per_mtok) == (10.0, 50.0)
+
+
+class TestUnifierGuards:
+ async def test_no_request_leaves_without_a_key(self, monkeypatch):
+ """AG8: the SDK would read OPENAI_API_KEY and send it to the base URL."""
+ monkeypatch.setenv("OPENAI_API_KEY", "sk-operator-secret")
+ unifier = _unifier(AsyncMock(return_value=None))
+ keyless = {name for name in all_providers() if name in ("ollama", "lmstudio", ENDPOINT_PROVIDER)}
+
+ with respx.mock(assert_all_called=False) as mock:
+ for provider in sorted(set(all_providers()) - keyless):
+ with pytest.raises(NodeUserError, match="No API key"):
+ await _chat(unifier, provider, api_key="")
+ assert mock.calls.call_count == 0
+
+ async def test_an_endpoint_without_its_url_row_is_not_configured(self):
+ factory = MagicMock()
+ spec = ProviderSpec(name=ENDPOINT_PROVIDER, factory=factory, sdk_exception_refs=("openai:OpenAIError",))
+ unifier = _unifier(AsyncMock(return_value=None))
+
+ with patch("services.llm.unifier.get_provider", return_value=spec):
+ with pytest.raises(NodeUserError, match="'home' is not configured"):
+ await _chat(unifier, REF)
+ factory.assert_not_called()
+
+ async def test_an_endpoint_reads_its_own_url_row_and_placeholder(self):
+ looked_up = []
+
+ async def get_api_key(name, session_id="default"):
+ looked_up.append(name)
+ return "http://host:8080/v1" if name == f"{REF}_proxy" else None
+
+ client = MagicMock()
+ client.chat = AsyncMock(return_value=LLMResponse(content="ok"))
+ factory = MagicMock(return_value=client)
+ spec = ProviderSpec(
+ name=ENDPOINT_PROVIDER,
+ factory=factory,
+ sdk_exception_refs=("openai:OpenAIError",),
+ client_kwargs={"provider_name": ENDPOINT_PROVIDER},
+ )
+ unifier = _unifier(get_api_key)
+
+ with patch("services.llm.unifier.get_provider", return_value=spec) as lookup:
+ await _chat(unifier, REF, api_key="")
+
+ lookup.assert_called_once_with(ENDPOINT_PROVIDER)
+ assert f"{REF}_proxy" in looked_up
+ kwargs = factory.call_args.kwargs
+ assert kwargs["proxy_url"] == "http://host:8080/v1"
+ assert kwargs["api_key"] == "sk-no-key-required"
+ assert kwargs["provider_name"] == ENDPOINT_PROVIDER
diff --git a/server/tests/llm/test_provider_self_registration.py b/server/tests/llm/test_provider_self_registration.py
index 6c4a1006..92bd113b 100644
--- a/server/tests/llm/test_provider_self_registration.py
+++ b/server/tests/llm/test_provider_self_registration.py
@@ -36,6 +36,8 @@ EXPECTED_REGISTERED_PROVIDERS = {
# Indic-first; OpenAI-compatible chat endpoint but no model-list route,
# which it declares via `supports_model_listing: false`.
"sarvam",
+ # The one registration behind every user-named endpoint (RFC-0003 D13).
+ "openai_compatible",
}
COMPAT_PROVIDERS = (
"xai",
@@ -186,6 +188,22 @@ def test_every_compat_provider_inherits_shared_response_and_tool_contract(
assert spec.sdk_exception_refs == ("openai:OpenAIError",)
+def test_endpoint_provider_is_registered_once_without_a_base_url():
+ """RFC-0003 AG17: named endpoints share one OpenAIProvider registration.
+
+ It must not pin a ``base_url``: each endpoint's resolved URL lives on its
+ own ``openai_compatible:<slug>_proxy`` row, and a pinned default would be
+ used silently when that row is missing.
+ """
+ from services.llm.providers.openai import OpenAIProvider
+
+ spec = get_provider("openai_compatible")
+ assert spec.factory is OpenAIProvider
+ assert spec.client_kwargs == {"provider_name": "openai_compatible"}
+ assert spec.sdk_exception_refs == ("openai:OpenAIError",)
+ assert list(all_providers()).count("openai_compatible") == 1
+
+
def test_unknown_provider_raises_node_user_error():
"""Unknown provider lookups surface as NodeUserError (user-correctable)."""
from services.plugin import NodeUserError
diff --git a/server/tests/nodes/test_openai_compatible_node.py b/server/tests/nodes/test_openai_compatible_node.py
new file mode 100644
index 00000000..cf31e306
--- /dev/null
+++ b/server/tests/nodes/test_openai_compatible_node.py
@@ -0,0 +1,102 @@
+"""RFC-0003 D13 on the node side: choosing a named endpoint.
+
+Agents carry it in their ``provider`` field; the OpenAI-compatible chat-model
+node in its ``endpoint`` field. Both resolve to the provider reference the
+runtime keys everything by. RLM, which builds its own clients, refuses what
+it cannot route instead of sending it to api.openai.com.
+"""
+
+from __future__ import annotations
+
+from unittest.mock import AsyncMock, patch
+
+import pytest
+from pydantic import ValidationError
+
+import services.llm # noqa: F401 — populate the registry
+from constants import AI_CHAT_MODEL_TYPES, detect_ai_provider
+from nodes.agent.ai_agent import AIAgentParams
+from nodes.model._option_loaders import load_ai_providers, load_endpoint_models, load_endpoints
+from nodes.model.openai_compatible_chat_model import OpenAICompatibleChatModelNode
+from services.llm.endpoints import SavedEndpoint
+
+REF = "openai_compatible:home"
+HOME = SavedEndpoint(ref=REF, label="Home vLLM", base_url="http://host:8000/v1", kind="generic", models=["a", "b"])
+
+
+class TestProviderRef:
+ @pytest.mark.parametrize("value", ["openai", "ollama", REF])
+ def test_registered_providers_and_named_endpoints_are_accepted(self, value):
+ assert AIAgentParams(provider=value).provider == value
+
+ @pytest.mark.parametrize("value", ["openai_compatible", "nope", "ollama:home", ""])
+ def test_a_bare_endpoint_id_or_unknown_provider_is_rejected(self, value):
+ with pytest.raises(ValidationError):
+ AIAgentParams(provider=value)
+
+
+class TestDetectProvider:
+ def test_agents_carry_the_reference_in_their_provider_field(self):
+ assert detect_ai_provider("aiAgent", {"provider": REF}) == REF
+
+ def test_the_chat_model_node_carries_it_in_its_endpoint_field(self):
+ assert "openaiCompatibleChatModel" in AI_CHAT_MODEL_TYPES
+ assert detect_ai_provider("openaiCompatibleChatModel", {"endpoint": REF}) == REF
+
+ def test_no_endpoint_yields_the_bare_id_which_the_unifier_refuses(self):
+ assert detect_ai_provider("openaiCompatibleChatModel", {}) == "openai_compatible"
+
+
+class TestLoaders:
+ async def test_providers_then_endpoints_in_declared_order(self):
+ with patch("nodes.model._option_loaders._endpoints", AsyncMock(return_value=[HOME])):
+ options = await load_ai_providers({})
+
+ values = [o["value"] for o in options]
+ assert values[0] == "openai" # llm_defaults.json order, not import order
+ assert "openai_compatible" not in values
+ assert options[-1] == {"name": "Home vLLM (OpenAI-compatible)", "value": REF}
+
+ async def test_the_node_lists_endpoints_only(self):
+ with patch("nodes.model._option_loaders._endpoints", AsyncMock(return_value=[HOME])):
+ assert await load_endpoints({}) == [{"name": "Home vLLM", "value": REF}]
+
+ async def test_models_come_from_the_selected_endpoint(self, monkeypatch):
+ auth = AsyncMock()
+ auth.get_stored_models = AsyncMock(return_value=["a", "b"])
+ monkeypatch.setattr("services.plugin.deps.get_auth_service", lambda: auth)
+
+ assert await load_endpoint_models({"endpoint": REF}) == [
+ {"name": "a", "value": "a"},
+ {"name": "b", "value": "b"},
+ ]
+ auth.get_stored_models.assert_awaited_once_with(REF)
+ assert await load_endpoint_models({"endpoint": "ollama"}) == []
+ assert await load_endpoint_models({}) == []
+
+
+class TestChatModelNode:
+ def test_the_endpoint_is_chosen_first_and_models_follow_it(self):
+ schema = OpenAICompatibleChatModelNode.Params.model_json_schema()["properties"]
+
+ assert list(schema)[0] == "endpoint"
+ assert schema["endpoint"]["loadOptionsMethod"] == "openaiCompatibleEndpoints"
+ assert schema["model"]["loadOptionsMethod"] == "openaiCompatibleModels"
+ assert schema["model"]["loadOptionsDependsOn"] == ["endpoint"]
+ # A `provider` sibling of `model` would trigger the panel's stored-key effect.
+ assert "provider" not in schema
+
+ def test_it_stays_single_purpose(self):
+ assert OpenAICompatibleChatModelNode.usable_as_tool is False
+ assert "tool" not in OpenAICompatibleChatModelNode.group
+
+
+class TestRlmGuard:
+ @pytest.mark.parametrize("provider", [REF, "ollama"])
+ async def test_rlm_refuses_a_provider_it_cannot_route(self, provider):
+ from services.rlm.service import RLMService
+
+ result = await RLMService(auth=None).execute("node-1", {"provider": provider, "model": "m", "prompt": "hi"})
+
+ assert result["success"] is False
+ assert "cannot run on provider" in result["error"]