Skip to content

Self-improving AI Employees for everyone

docs(rfc): RFC-0003, the OpenAI-compatible provider contract

trohitga5d2a8d04cb6
diff --git a/CLAUDE.md b/CLAUDE.md
index 7cc7bd2e..cdafca5c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -57,6 +57,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[CLI Services Integration](./docs-internal/cli_services_integration.md)** | Guide for integrating CLI-based services (Temporal, etc.) with proper lifecycle management |
| **[Temporal Architecture](./docs-internal/TEMPORAL_ARCHITECTURE.md)** | Distributed workflow execution: activities, FIRST_COMPLETED scheduling, horizontal scaling |
| **[Native LLM SDK](./docs-internal/native_llm_sdk.md)** | Native SDK layer in `services/llm/`: protocol-based providers, config-driven base URLs, and 13 providers (11 cloud plus Ollama / LM Studio; xAI is agent-selectable without a standalone chat-model node). A JSON-driven `supports_model_listing: false` flag covers endpoints that speak the OpenAI chat wire format but ship no `/v1/models` route (Sarvam) — `OpenAIProvider.fetch_models` then serves the curated list and probes the key with a one-token completion. Every provider is native; `tests/llm/test_langchain_removed.py` locks the retired-dependency end state. |
+| **[RFC-0003 — OpenAI-Compatible Provider Contract](./RFC-0003-OPENAI-COMPATIBLE-PROVIDER-CONTRACT.md)** | Draft. "OpenAI-compatible" is a family of divergent shapes with no spec behind it, and the client currently *guesses* per-provider facts it cannot infer. Twelve decisions (D1–D12) with acceptance gates: a configured `base_url` is copied **verbatim** from vendor docs and never rewritten (`/v1` is neither universal nor positionally predictable — DeepSeek is root-mounted, Groq is `/openai/v1`, Sarvam is `/v1` *and* `/v2`); a **user-supplied** URL is resolved by **probing** `GET {base}/models` then `{base}/v1/models` at save time, because a path-less URL is indistinguishable from a deliberately root-mounted server without probing; 401/403 counts as rooted at probe time and as a credential failure at runtime; the credential is resolved by one function and no provider hardcodes a placeholder key (the `api_key="ollama"` overwrite in `OpenAIProvider.__init__` applies to *every* provider with a proxy row); `api_key=None` must never reach the SDK, since it falls back to `OPENAI_API_KEY` and ships the operator's key to a third-party `base_url`; capabilities are declared, never sniffed, and a flag ships only with a consumer; a 2xx with no `choices` and an error body raises `PROTOCOL` naming the URL called. Five revertible phases, `ProviderSpec` gains no field, `_strip_v1_path` survives scoped to the native probe. Also records that `native_llm_sdk.md`'s DeepSeek `/v1` claim is wrong and the path-less JSON entry is correct. |
| **[Event Waiter System](./docs-internal/event_waiter_system.md)** | In-memory asyncio.Future waiter for push-based trigger nodes on the canvas-Run path (WhatsApp, Telegram, Webhook, Chat, Task completion). Deployed triggers ride the Temporal canary path; the Redis-Streams backend was retired in Wave 15.3 |
| **[Credentials Encryption](./docs-internal/credentials_encryption.md)** | Fernet + PBKDF2 encryption pipeline, separate credentials.db, two credential systems (OAuth vs API keys), multi-backend abstraction |
| **[Status Broadcaster](./docs-internal/status_broadcaster.md)** | WebSocket-first communication: StatusBroadcaster singleton, live count via `len(MESSAGE_HANDLERS) + len(get_ws_handlers())`, broadcast message types, Android two-state model |
diff --git a/RFC-0003-OPENAI-COMPATIBLE-PROVIDER-CONTRACT.md b/RFC-0003-OPENAI-COMPATIBLE-PROVIDER-CONTRACT.md
new file mode 100644
index 00000000..64855857
--- /dev/null
+++ b/RFC-0003-OPENAI-COMPATIBLE-PROVIDER-CONTRACT.md
@@ -0,0 +1,354 @@
+# RFC-0003 — OpenAI-Compatible Provider Contract
+
+Status: Draft
+Created: 2026-09-01
+Scope: `server/services/llm/`, `server/nodes/model/`
+Relationship: independent of RFC-0002 (Context/Memory). Touches no agent state.
+
+## How to read this
+
+If you are **adding a provider**, read §9. If you are **implementing**, read §3
+(decisions) then §10 (phases). If you are **reviewing**, §3 and §11 are the
+whole normative surface; everything else is rationale.
+
+Files this RFC changes, by phase:
+
+| File | Phase | Change |
+|---|---|---|
+| `services/llm/protocol.py` | 1 | `LLMError` gains `public_message`; new `PROTOCOL` category |
+| `services/llm/config.py` | 1 | `resolve_credential()`; capability accessors |
+| `services/llm/providers/openai.py` | 2 | drop `api_key="ollama"`; drop `proxy_url` param |
+| `services/llm/providers/anthropic.py` | 2 | drop the duplicate placeholder hardcode |
+| `services/llm/unifier.py` | 2 | collapse proxy/base into one effective URL + `url_source` |
+| `services/llm/endpoints.py` | 3 | **new** — save-time base-URL resolver |
+| `nodes/model/_local_validator.py` | 3 | resolve before persist; drop the `/v1` 404 hint |
+| `config/llm_defaults.json` | 1, 4 | `auth` block; capability flags |
+| `services/llm/providers/openai.py` | 5 | 2xx-with-error-body guard |
+| `docs-internal/native_llm_sdk.md` | 1 | fix the DeepSeek `/v1` claim (§13) |
+
+## 1. Summary
+
+OpenCompany treats "OpenAI-compatible" as one shape. It is a family of
+divergent shapes with **no specification behind it**, and the divergences are
+invisible from a successful response.
+
+There are only two ways to hold a per-provider fact: declare it, or guess it.
+Today the client guesses — that a stored base URL is correctly rooted, that the
+credential belongs in `Authorization: Bearer`, that a provider with a
+`{provider}_proxy` row wants the literal key `"ollama"`, and that HTTP 200 means
+the request was understood. It also validates two providers against a different
+wire surface than the one execution uses, so a credential can read green while
+the runtime is dead.
+
+This RFC makes the endpoint and the credential **resolved facts** — computed
+once, at save time, with evidence — and makes capability variance **declared**,
+extending the `supports_model_listing` precedent the repo already ships. The
+error contract is handled behaviourally, not by declaration (§8.4).
+
+## 2. Problem
+
+### 2.1 The failure chain (normative; other sections cross-reference here)
+
+A stored LM Studio base URL without `/v1`. The panel read **Connected** with
+models discovered. Every run failed with `AI generated empty response`.
+
+| # | Step | Site |
+|---|---|---|
+| 1 | URL persisted verbatim, **before** the probe runs | `_local_validator.py:270-275` |
+| 2 | Probe strips a *trailing* `/v1` and the scheme, then talks to LM Studio's **native WebSocket SDK**. With no `/v1` present the strip is a no-op, so the probe cannot see the defect | `_local_validator.py:124-134`, `:201-203` |
+| 3 | Probe succeeds. `valid=True` broadcast | `_local_validator.py:314-339` |
+| 4 | Runtime reads the same row, for every provider | `unifier.py:296` |
+| 5 | `url = proxy_url or base_url` then `AsyncOpenAI(base_url=url)`, key overwritten with `"ollama"` | `providers/openai.py:45-49` |
+| 6 | SDK concatenates prefix + endpoint, giving `POST /chat/completions` | `openai/_base_client.py:496` |
+| 7 | LM Studio answers **HTTP 200** with an `{"error": ...}` body | vendor behaviour |
+| 8 | SDK raises only on 4xx/5xx. `choices` is absent, so it coerces to `None` | `openai/_models.py` |
+| 9 | Empty content becomes `AI generated empty response` | `services/ai.py` |
+
+Two structural defects compound. **Validation exercises a surface execution
+never touches** — LM Studio serves OpenAI-compat at `/v1/*`, native REST at
+`/api/v1/*`, and a WebSocket SDK at `ws://{host}/{namespace}`, all on one port;
+Ollama likewise serves `/api/*` beside `/v1/*`. A green Fetch proves host
+liveness on the *native* surface and nothing about the prefix the runtime uses.
+And **a 2xx is treated as an understood request**, which for this class of
+server it is not.
+
+### 2.2 Why the endpoint cannot be inferred
+
+`base_url` is a path **prefix**, string-concatenated with the endpoint
+(`_prepare_url`: `base_url.raw_path + endpoint.lstrip("/")`,
+`openai/_base_client.py:496`) — not `urljoin`. The base path is never
+discarded, and the SDK **never inserts `/v1`**; that segment lives in the
+default constant `https://api.openai.com/v1`. Override `base_url` and the whole
+prefix is yours.
+
+`/v1` is neither universal nor positionally predictable:
+
+| Shape | Providers |
+|---|---|
+| Root-mounted, versionless | DeepSeek (`/chat/completions` at root; siblings `/beta`, `/anthropic`) |
+| Namespaced | Groq `/openai/v1`, Fireworks `/inference/v1`, OpenRouter `/api/v1` |
+| Version varies by model | Sarvam `/v1` and `/v2` |
+| `/v1` only | vLLM, Ollama, LM Studio, Mistral, xAI, Kimi, Cerebras, Together |
+| Both root and `/v1` | llama.cpp, LocalAI, LiteLLM |
+| Configurable, may be empty | Jan |
+
+So "append `/v1` when the URL has no path" is wrong for the first three rows,
+and it fails in the opposite direction for path-prefix deployments — vLLM
+`--root-path`, llama.cpp `--api-prefix`, Open WebUI proxying Ollama under
+`/ollama` — where a URL that *has* a path still needs `/v1` appended.
+
+**A path-less base URL is genuinely ambiguous**: indistinguishable from a
+deliberately root-mounted server without probing. No string rule resolves it.
+That is why §6 probes.
+
+`AzureOpenAI` is the shape of the fix: it takes `azure_endpoint` (host, no
+path), builds `f"{endpoint}/openai"` itself, and rejects `base_url` as mutually
+exclusive. Where the prefix is knowable, the library supplies it.
+
+## 3. Decisions
+
+| # | Decision | Locked by |
+|---|---|---|
+| D1 | A configured provider's `base_url` is copied verbatim from vendor docs. The runtime never appends, strips, or rewrites a path segment. | AG1, AG2 |
+| D2 | A user-supplied base URL is **resolved by probing at save time**, once, and the resolved value is persisted. The runtime consumes it verbatim. | AG3, AG4 |
+| D3 | Probe order is `{base}/models` then `{base}/v1/models`. First success wins. 401/403 counts as rooted. | AG3 |
+| D4 | The probe verdict is three-state: `verified`, `rooted-unverified`, `failed`. | AG5 |
+| D5 | On probe failure nothing is written to `{provider}_proxy`. The verdict lands on the credential row. | AG6 |
+| D6 | The credential is resolved by one function. No provider hardcodes a placeholder key. | AG7, AG8 |
+| D7 | No code path hands the SDK an unresolved credential — `api_key=None` makes the SDK read `OPENAI_API_KEY` and ship the operator's OpenAI key to a third party. | AG8 |
+| D8 | A capability is **declared**, never sniffed. A flag ships only with a v1 consumer. | AG9 |
+| D9 | A 2xx chat response with no `choices` and an error-shaped body raises `PROTOCOL`, naming the URL called. | AG10 |
+| D10 | Every provider-call failure logs the effective URL and `url_source`, redacted. Never the key. | AG11 |
+| D11 | `ProviderSpec` gains no field. | AG12 |
+| D12 | `_strip_v1_path` survives, scoped to the native enrichment probe only. | AG2 |
+
+## 4. Goals and non-goals
+
+**Goals.** One connected verdict across panel and palette. A stored URL that
+works keeps working with no operator action. A routing mistake is
+distinguishable from a model failure in the message the user sees. Adding a
+compatible provider is a JSON edit plus one name in a tuple.
+
+**Non-goals.** Not a universal LLM abstraction — `ChatUnifier` stays the single
+facade. Not a provider-registry rewrite. Not Anthropic's or Gemini's native wire
+format; D6 and D7 apply to their credential handling only, because they share
+the defect. Not per-model *discovery* — per-model declaration is permitted via
+the existing `CapabilityConfig` precedence and required only where vendor docs
+document a divergence.
+
+## 5. The base URL contract
+
+A provider's `base_url` in `llm_defaults.json` is an **opaque, vendor-declared,
+verbatim prefix**. It is copied from the vendor's own documentation and is never
+synthesized, completed, or normalized by the runtime.
+
+This is why `deepseek: "https://api.deepseek.com"` is **correct** — DeepSeek
+documents exactly that, with routes at `/chat/completions`. It is not a missing
+`/v1` (§13).
+
+Trailing slashes are irrelevant: `_enforce_trailing_slash`
+(`openai/_base_client.py:414`) normalizes the base on assignment. Defensive
+stripping or appending in our code is dead weight.
+
+## 6. Endpoint resolution
+
+Applies **only** to user-supplied URLs for self-hosted providers (`ollama`,
+`lmstudio`). Configured providers are covered by §5.
+
+```python
+# services/llm/endpoints.py -- save-time only. Async, does IO.
+async def resolve_base_url(candidate: str, *, api_key: str) -> ResolvedBaseUrl:
+ """Probe which prefix actually roots the OpenAI surface.
+
+ A path-less URL is ambiguous: LiteLLM/llama.cpp/Jan serve at root,
+ LM Studio/Ollama/vLLM only under /v1. Probing is the only way to tell.
+
+ GET /models is the one endpoint every compatible server implements and
+ the only side-effect-free one -- /chat/completions loads a model and
+ bills. Where a provider declares model_listing.path: null (Sarvam), the
+ declared fallback is one max_tokens=1 completion; that fallback is
+ itself a declared fact, not a guess.
+ """
+```
+
+Order, first success wins:
+
+| Candidate | Outcome |
+|---|---|
+| `{base}/models` returns 2xx | `verified`, adopt `{base}` |
+| `{base}/models` returns 401/403 | `rooted-unverified`, adopt `{base}` |
+| `{base}/v1/models` returns 2xx | `verified`, adopt `{base}/v1`, `rewritten=True` |
+| `{base}/v1/models` returns 401/403 | `rooted-unverified`, adopt `{base}/v1`, `rewritten=True` |
+| neither | `failed`; nothing persisted to `{provider}_proxy` (D5) |
+
+401/403 means **routing succeeded, credentials did not** — vLLM's `--api-key`
+protects the whole `/v1` prefix. At probe time that is a success signal. At
+runtime the same status is a failure (§8.3). The two rules are deliberately
+different, and both are stated here so neither is mistaken for the other.
+
+This handles path-prefix deployments correctly: `https://host/vllm` probes
+`/vllm/models` then `/vllm/v1/models`. A has-a-path heuristic gets that
+backwards.
+
+`rewritten=True` logs at INFO naming both candidates. An operator who typed one
+thing and got another must be able to find out why.
+
+**Two layers, different rules.** Save-time resolution is async and does IO.
+Call-time consumption is synchronous and pure — it reads the resolved value and
+nothing else. The AG1 tripwire excludes `endpoints.py` for exactly this reason,
+and that exclusion is the module's justification for existing.
+
+## 7. Credential resolution
+
+`OpenAIProvider.__init__` currently does `if proxy_url: kwargs["api_key"] =
+"ollama"` (`providers/openai.py:49`), duplicated at `anthropic.py:41`. That
+hardcodes a dummy key for **any** provider with a proxy row, not just local
+ones, and it is why a cloud proxy cannot work today.
+
+It is replaced by one resolver, in `config.py`:
+
+```python
+def resolve_credential(provider: str, stored: str | None) -> str:
+ """Return the key to send. Never None, never a hardcoded vendor name."""
+```
+
+Providers that require a placeholder declare it:
+
+```json
+"lmstudio": { "auth": { "scheme": "bearer", "placeholder_key": "lm-studio" } }
+```
+
+Ollama's own docs say `api_key='ollama'` is "required but ignored", so the
+placeholder is a **declared vendor fact**, not an invention.
+
+**D7 is a security rule, not hygiene.** `AsyncOpenAI(api_key=None)` falls back
+to `os.environ["OPENAI_API_KEY"]`. Combined with a third-party `base_url`, that
+ships the operator's OpenAI key to another vendor. No path may reach the SDK
+with an unresolved credential; AG8 asserts zero requests are issued in that
+state.
+
+## 8. Capabilities and the error contract
+
+### 8.1 Declared, never sniffed
+
+This extends `supports_model_listing` (`config.py:179-189`) rather than adding
+a parallel mechanism. **A flag ships only with a consumer.** A declaration with
+no reader is how `models_endpoint` and `build_headers` became dead config.
+
+v1 flags and their consumers:
+
+| Flag | Consumer |
+|---|---|
+| `model_listing.path` (null = no route) | existing `fetch_models` branch |
+| `surface.responses` | replaces the `provider_name == "openai"` hardcode in `openai.py` |
+| `auth.scheme`, `auth.placeholder_key` | `resolve_credential` (§7) |
+
+Everything else — tool calling, structured outputs, embeddings, streaming usage
+— is **declared next, not now**. Naming them here without a reader would repeat
+the mistake this section exists to prevent.
+
+### 8.2 The 2xx guard
+
+A 2xx chat response with **no `choices`** and an error-shaped body is a routing
+failure, not an empty completion. It raises `LLMErrorCategory.PROTOCOL` with a
+message naming the URL actually called. This is the single change that would
+have made the original bug self-diagnosing.
+
+### 8.3 Runtime errors
+
+401/403 at runtime is a credential failure and says so, quoting the effective
+URL. Every failure logs `url_source` (one of `proxy`, `llm_defaults`,
+`factory_default`), computed in `_get_or_create_entry` where the override is
+chosen and carried into the exception. URLs pass through `redact_url()` — which
+strips userinfo and the query string — before reaching a log or a user-facing
+message. The key is never logged in any form.
+
+Per the repo's `NodeUserError` contract, a routing or credential failure is
+user-correctable: one WARN line, no traceback.
+
+### 8.4 No declared error block
+
+The four per-provider facts are rooting, auth, capability, and error contract.
+The first three are declared; the **error contract is handled behaviourally** —
+tolerant parsing plus the §8.2 guard. Declaring per-provider error shapes would
+be a table with one entry and no way to keep it honest.
+
+## 9. Adding an OpenAI-compatible provider
+
+1. Copy `base_url` **verbatim** from the vendor's docs. Do not normalize it.
+2. Add a `providers.<name>` block to `llm_defaults.json` with `base_url`, an
+ `auth` block, and any capability flag that differs from the permissive
+ default. Add a `_note` citing the doc URL for each non-default flag.
+3. Add the name to `_COMPAT_PROVIDERS` in `providers/_compat.py`.
+4. If the vendor documents no model-list route, set `model_listing.path: null`.
+5. Nothing else. No provider subclass, no `ProviderSpec` field (D11), no Python.
+
+## 10. Migration
+
+Backward compatible throughout: an existing stored URL that works keeps working
+with no operator action.
+
+| Phase | Change | Rollback |
+|---|---|---|
+| 1 | `LLMError.public_message`, `PROTOCOL`, `resolve_credential`, `auth` blocks. Additive, no reader. | revert alone |
+| 2 | Constructors consume `resolve_credential`; unifier collapses to one effective URL plus `url_source`. Touches `openai.py`, `anthropic.py`, `unifier.py`. | revert alone |
+| 3 | `endpoints.py` plus validator resolves before persist. Gated by `LLM_ENDPOINT_CONTRACT_ENABLED`, shipping **false**; flipped in a separate commit. | flag |
+| 4 | Capability flags with consumers (§8.1). | revert alone |
+| 5 | 2xx guard. Depends on Phase 2 for `url_source`. | revert alone |
+
+Phase 2 preserves behaviour exactly for every currently-working install: a local
+provider with a placeholder row resolves to the same key it sends today. Phase 3
+is where a previously-broken URL starts working.
+
+## 11. Acceptance gates
+
+| # | Assertion | Style |
+|---|---|---|
+| AG1 | No `"/v1"` literal and no `.rstrip("/")` under `services/llm/` or `nodes/model/`, except the AG2 allowlist | AST scan |
+| AG2 | The allowlist is exactly `endpoints.py` (save-time resolver) and `_local_validator._strip_v1_path` (native probe only) | source introspection |
+| AG3 | The resolver probes `{base}/models` then `{base}/v1/models`, adopting on 2xx or 401/403 | async unit |
+| AG4 | Runtime client construction contains no probe and no `await` on a URL | AST |
+| AG5 | All three verdict states render distinctly on both panel and palette | FE + BE unit |
+| AG6 | A failed probe writes nothing to `{provider}_proxy` | async unit |
+| AG7 | No literal `"ollama"` or `"lm-studio"` appears in any provider `__init__` | source scan |
+| AG8 | With `OPENAI_API_KEY` set and no stored key, every provider raises `NodeUserError` and issues **zero** requests | async unit |
+| AG9 | Every flag in `llm_defaults.json` has a reader in `services/llm/` | AST cross-check |
+| AG10 | 2xx with no `choices` and an error body raises `PROTOCOL` naming the URL | unit |
+| AG11 | No log line or `public_message` contains userinfo, a query string, or `Bearer` | unit |
+| AG12 | The `ProviderSpec` field set is unchanged | source introspection |
+
+## 12. What this RFC does not change
+
+`ChatUnifier` stays the single facade. `ProviderSpec` keeps its shape (D11).
+`sdk_exception_refs` stay lazy `"module:ClassName"` strings — no SDK import at
+registration, per `tests/llm/test_lazy_sdk_imports.py`. `OpenRouterProvider`
+keeps its own `__init__` and its hardcoded base; it is correct per vendor docs.
+The dead JSON fields (`models_endpoint`, `api_key_format`, `extra_headers`) are
+**not deleted here** — that is a subtractive change with its own risk, tracked
+separately.
+
+## 13. Compatibility audit of the nine compat providers
+
+All nine `base_url` values in `llm_defaults.json` match their vendor docs and
+need no change. In particular **`deepseek` is path-less and correct**.
+
+The defect is in our documentation: `docs-internal/native_llm_sdk.md:235` states
+DeepSeek's base as `https://api.deepseek.com/v1`, which the vendor does not
+document. Phase 1 fixes the doc, not the config. Any future change that
+"corrects" the JSON toward `/v1` is a regression.
+
+## 14. References
+
+- openai-python `_base_client.py` — `_prepare_url:496`, `_enforce_trailing_slash:414`
+- openai-python `lib/azure.py` — `azure_endpoint` / `base_url` mutual exclusion
+- DeepSeek https://api-docs.deepseek.com/ — base `https://api.deepseek.com`
+- Groq https://console.groq.com/docs/openai — `/openai/v1`
+- OpenRouter https://openrouter.ai/docs/quickstart — `/api/v1`
+- Fireworks https://docs.fireworks.ai/tools-sdks/openai-compatibility — `/inference/v1`
+- Sarvam https://docs.sarvam.ai/api/getting-started/models/open-source — `/v1` and `/v2`
+- LM Studio https://lmstudio.ai/docs/app/api/endpoints/openai
+- Ollama https://docs.ollama.com/api/openai-compatibility
+- vLLM https://docs.vllm.ai/en/latest/cli/serve.html — `--root-path`
+- llama.cpp https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md — `--api-prefix`
+- LiteLLM https://docs.litellm.ai/docs/proxy/user_keys
+- Open WebUI https://docs.openwebui.com/reference/api-endpoints/
diff --git a/docs-internal/native_llm_sdk.md b/docs-internal/native_llm_sdk.md
index 9f48fe5a..466fee89 100644
--- a/docs-internal/native_llm_sdk.md
+++ b/docs-internal/native_llm_sdk.md
@@ -232,8 +232,8 @@ Adding a new OpenAI-compatible provider requires a config entry:
"deepseek": {
"default_model": "deepseek-v4-flash",
"detection_patterns": ["deepseek"],
- "models_endpoint": "https://api.deepseek.com/v1/models",
- "base_url": "https://api.deepseek.com/v1",
+ "models_endpoint": "https://api.deepseek.com/models",
+ "base_url": "https://api.deepseek.com",
"max_output_tokens": { "_default": 8192 },
"context_length": { "_default": 128000 },
"temperature_range": [0.0, 2.0]