Skip to content

Self-improving AI Employees for everyone

feat(gateway): run LLM calls with no own key on the OpenCompany Gateway

trohitg3f32e4e159f0
diff --git a/.env.template b/.env.template
index c5b74b13..4ee232a6 100644
--- a/.env.template
+++ b/.env.template
@@ -333,4 +333,11 @@ WEBSOCKET_API_KEY=
# Default port for AI proxy servers (e.g., Ollama, Claude Code CLI proxy)
AI_PROXY_DEFAULT_PORT=11434
+# OpenCompany Gateway (docs-internal/llm_gateway.md). When enabled, the
+# Connectors panel offers "OpenCompany Gateway": with a gateway key there, LLM
+# calls for a provider this install has no key of its own for go through the
+# gateway, billed to the key owner's monthly allowance.
+OPENCOMPANY_GATEWAY_ENABLED=false
+OPENCOMPANY_GATEWAY_URL=https://api.gw.opencompany.sh
+
diff --git a/CLAUDE.md b/CLAUDE.md
index 229ae60b..70e2ed9e 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -51,7 +51,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[Translate / Transliterate / Detect](./docs-internal/speech_provider_rfc.md#8-the-pattern-generalised-nodestranslate)** | The second application of the multi-vendor pattern, and the one that retired `nodes/sarvam/`. Three nodes (`translateText`, `transliterateText`, `detectLanguage`) in [`server/nodes/translate/`](./server/nodes/translate/), backed by **three registries — one per capability**, because the asymmetry is sharper than speech's: DeepL translates and does nothing else, while Sarvam and LLM-backed providers do all three. Membership makes an unsupported selection unrepresentable rather than a runtime failure inside a paid call. Providers: **deepl** (`Authorization: DeepL-Auth-Key`, `text` is an array, free keys ending `:fx` route to a different host, and it returns `billed_characters` so cost is the provider's own figure), **sarvam** (all three over dedicated REST), **openai** (all three via `ChatUnifier` — proves the Protocol does not care what a provider does internally; billing deliberately NOT recorded because that path bills tokens the LLM layer already costs). Capabilities in `server/config/translate_defaults.json`. |
| **[Desktop App](./docs-internal/desktop_app.md)** | Electron shell in [`desktop/`](./desktop/) — a standalone bun package (NOT a root workspace member; `cli/tests/test_release_pipeline_config.py` locks `workspaces`). Bundles pinned `uv`, python-build-standalone CPython 3.12 and bun 1.4.0 — the only JavaScript runtime shipped; no Node, no npm (`desktop/runtimes.json`, verified against upstream checksum manifests by `scripts/fetch-runtimes.ts`, which also fetches bun's `LICENSE.md` from the bun repo — the release zip carries none — and prunes stale runtime dirs such as a leftover `node/`), stages the backend sibling layout from `bun pm pack --dry-run` (`scripts/stage.ts`, which runs the sidecar package's own `bun run build` — `bun build --target=bun`, express inlined — and copies `dist/index.js`), sets `OPENCOMPANY_BUN_BIN` / `BUN_INSTALL=<userData>/bun` / `BUN_INSTALL_CACHE_DIR` and prepends the bundled bun dir (then uv) to PATH for the backend (`env.ts`; `OPENCOMPANY_NODE_BIN` no longer exists), provisions the venv into Electron `userData` on first launch with `uv sync --frozen --python <bundled>` (stamp = sha256 of uv.lock + pyproject + Python + uv versions; falls back to `uv python install 3.12`), pre-warms `PYTHONPYCACHEPREFIX` (with a prefix set Python ignores in-tree .pyc — 32 s vs 3 s first boot), spawns uvicorn on `127.0.0.1:5678` (persisted port; attaches to a running `company serve` instead of double-spawning), waits on `/health/ready`, loads the backend-served SPA (zero frontend changes), routes `window.open` / `_blank` to the OS browser. Quit = `POST /api/desktop/shutdown` + stdin EOF, 30 s, then tree-kill. electron-builder: NSIS x64, DMG+zip arm64/x64, AppImage+deb x64, unsigned first (`mac.identity: null`; macOS auto-update notify-only until signed — Squirrel.Mac refuses unsigned). CI: `desktop-release.yml` (tag-triggered: `prepare` drafts the GitHub Release from the annotated tag's message, three legs build and upload version-less OS-named installers such as `OpenCompany-windows-x64.exe` / `OpenCompany-macos-arm64.dmg` / `OpenCompany-linux-x86_64.AppImage` so the README can link `releases/latest/download/<file>`, each leg refuses a desktop version that differs from the tag, `finalize` undrafts) and `desktop-ci.yml` (typecheck/unit/invariants/Playwright smoke). **Gotcha: editor-hosted terminals export `ELECTRON_RUN_AS_NODE=1`, which makes Electron boot as plain Node (`electron.app` undefined) — strip it before launching; the e2e test does.** |
| **[Desktop Host Contract](./docs-internal/desktop_host_contract.md)** | The backend's side of being owned by a GUI shell. `core/approot.py` is the ONE module that locates the app tree (`OPENCOMPANY_APP_ROOT` / `OPENCOMPANY_CLIENT_DIST` / `OPENCOMPANY_ENV_FILE` / `OPENCOMPANY_ENV_TEMPLATE`); `tests/test_no_root_climb.py` forbids any other `Path(__file__).parents[N]` climb above `server/` and any `"client"/"dist"` / `.env.template` / `package.json` path join. `core/env_defaults.apply_file_defaults_to_environ()` does the CLI's `.env.template < .env < process env` layering at the top of `main.py`, and `Settings.env_file` is the absolute `(template, .env)` tuple — so a shell spawns `python -m uvicorn` from any cwd without the CLI package. Desktop mode (`OPENCOMPANY_DESKTOP=1`, `core/desktop.py`): parent-PID watchdog with create-time PID-reuse guard, stdin-EOF watchdog (polling via `PeekNamedPipe`/`select`, never a blocking read — a daemon thread parked in `ReadFile` at finalisation crashed exit with 0xC0000005), ctypes Job Object so children die with the backend (no pywin32 in the server venv), token-gated `POST /api/desktop/shutdown` (`routers/desktop.py`, public path; raises the signal uvicorn handles so the lifespan reaps Temporal/the bun sidecar/edgymeow), 45 s hard deadline. `GET /health/ready` (public) = DB up AND (Temporal disabled OR worker manager started), with `phase` written by `services/temporal/lifecycle.py` (`installing_temporal` / `starting_temporal` / `connecting` / `starting_workers` / `ready`). Plugin shutdown hooks now run under a 10 s per-hook timeout (`HOOK_TIMEOUT_SECONDS`), and the current browser hook uses `peek_browser_runtime()` so shutdown never constructs or installs the runtime. Historical incident: the retired `get_browser_service()` lazily installed agent-browser at shutdown and wedged graceful exit (masked under the CLI by its 5 s grace + tree-kill). `server/uv.lock` is committed; `predeploy.yml` runs `uv lock --check`. |
-| **[LLM Gateway](./docs-internal/llm_gateway.md)** | Hosted LiteLLM Proxy in `gateway/` (official AWS module, mock-model local stack, contract and pin tests, `gateway-ci.yml`): pinned LiteLLM 1.103.4 and price list, served providers, and the contract OpenCompany relies on (wire model names, `budget_exceeded` refusals, `/user/info` allowance with Default keys, attribution headers, cache replay). Operator runbook: `gateway/README.md`. OpenCompany-side routing not implemented yet. |
+| **[LLM Gateway](./docs-internal/llm_gateway.md)** | Hosted LiteLLM Proxy in `gateway/` (official AWS module, mock-model local stack, contract and pin tests, `gateway-ci.yml`): pinned LiteLLM 1.103.4 and price list, served providers, and the contract OpenCompany relies on (wire model names, `budget_exceeded` refusals, `/user/info` allowance with Default keys, attribution headers, cache replay). Operator runbook: `gateway/README.md`. OpenCompany's side, `server/services/gateway/`: the Connectors card (`OPENCOMPANY_GATEWAY_ENABLED`), own-keys-first routing in ChatUnifier from `"gateway"` blocks in `llm_defaults.json`, attribution from the log context, gateway error wording. |
| **[CI/CD Pipeline](./docs-internal/ci_cd.md)** | GitHub Actions workflows, predeploy validation, release publishing, and the composite setup action (bun via SHA-pinned `oven-sh/setup-bun` reading the root `packageManager` pin + Node 22 as a CI-only convenience so bun runs vite / vitest / eslint on it via their node shebangs + Python 3.12 + uv; `bun install --frozen-lockfile` everywhere; publish is `bun publish --access public` after a `bun pm whoami` credential preflight with `~/.npmrc` written from `NPM_TOKEN` — no npm CLI, no `setup-node` step of their own in the publish jobs (they get Node through the composite action), and npm provenance attestation went away with npm). Also **Cutting a release**: the annotated tag is the version and its message is the release notes; `company version sync vX.Y.Z` bumps every version file for the `chore(release)` commit the tag sits on, and `desktop-release.yml` drafts the GitHub Release from the tag message and refuses a version that differs from the tag |
| **[Performance](./docs-internal/performance.md)** | Cold-start measurements (Application startup complete 2.90 s warm, 21.5 s cold post-`company clean` — was 71 s before the 2026-07-14 boot-delay fixes; first WS connect 8.29 s) + warm and cold per-phase timelines + optimisation history (lazy LangChain imports, lazy LLM-SDK exception refs, pre-bundled sidecar (esbuild at the time, `bun build --target=bun` today), bytecode-at-build, PartySocket reconnect, TanStack Query auth bootstrap, `manualChunks`, Vite dep-cache preservation, off-loop Temporal build-id hash) + bottleneck inventory + reproduction commands + anti-patterns to never reintroduce (incl. eager SDK import at LLM provider registration, unconditional `.vite` wipe, synchronous `Worker()` on the event loop). Pairs with the build-time pipeline doc below. |
| **[Release Build Pipeline](./docs-internal/release_build_pipeline.md)** | Build pipeline for the registry tarball (installed with `bun add -g`; bun is the only JavaScript runtime and package manager anything shipped needs): `typescript@7` (native Go compiler, exact-pinned in **root** devDependencies — client keeps `typescript@^5.x` because typescript-eslint's peer range excludes 6/7, and one manifest cannot declare `typescript` twice) for type-check, measured 5.9× faster than tsc 5.9.3 (820 ms vs 4830 ms); `bun run --filter react-flow-client typecheck` delegates up to the root gate, Vite `manualChunks` + `target: 'es2022'`, `bun build --target=bun` sidecar bundle (self-contained `dist/index.js` with express inlined, run as `bun dist/index.js`; drops the `tsx` interpreter cost and needs no `node_modules` at runtime), bytecode pre-compile in two halves — `[tool.uv] compile-bytecode = true` (uv sync compiles `.venv/` site-packages) + `python -m compileall` for project source (plain `.pyc`, **no `-O`** — per PEP 488 the non-`-O` runtimes never load `.opt-1.pyc`; excludes `.venv/`, `tests/`). Single source of truth for the bytecode-compile path list: `cli.commands.build.COMPILEALL_SOURCE_DIRS`. Dev PM is **[email protected]** (tracked text `bun.lock`, `bunfig.toml` pins the isolated linker, ranged security pins in top-level `overrides`); `company build` step [1/6] is `bun install`. |
diff --git a/client/src/components/credentials/panels/ApiKeyPanel.tsx b/client/src/components/credentials/panels/ApiKeyPanel.tsx
index 85b5efc8..cd31339f 100644
--- a/client/src/components/credentials/panels/ApiKeyPanel.tsx
+++ b/client/src/components/credentials/panels/ApiKeyPanel.tsx
@@ -169,6 +169,9 @@ const ApiKeyPanel: React.FC<CredentialPanelProps> = ({
{field?.help && !endpoints && (
<p className="mt-2 text-xs text-muted-foreground">{field.help}</p>
)}
+ {panel.statusSummary && validated && (
+ <p className="mt-2 text-sm">{panel.statusSummary}</p>
+ )}
</CardContent>
</Card>
diff --git a/client/src/components/credentials/useCredentialPanel.ts b/client/src/components/credentials/useCredentialPanel.ts
index 600cb29e..eb74cccd 100644
--- a/client/src/components/credentials/useCredentialPanel.ts
+++ b/client/src/components/credentials/useCredentialPanel.ts
@@ -230,8 +230,21 @@ export function useCredentialPanel(config: ProviderConfig, visible: boolean) {
execute(type, () => sendRequest(type, data ?? {}, timeoutMs)),
};
+ // A key-based card that declares a status hook (``ws.status``) shows the
+ // line it returns as ``summary`` once a key is stored: the OpenCompany
+ // Gateway's remaining allowance, for one. OAuth cards keep their own flow.
+ const statusHook = config.kind !== 'oauth' ? config.ws?.status : undefined;
+ const statusQuery = useQuery<{ summary?: unknown }, Error>({
+ queryKey: ['credential-status', config.id],
+ queryFn: () => sendRequest(statusHook as string, {}),
+ enabled: Boolean(visible && isConnected && statusHook && stored),
+ staleTime: 60_000,
+ });
+ const statusSummary = typeof statusQuery.data?.summary === 'string' ? statusQuery.data.summary : null;
+
return {
form, values, loading, error, stored, setStored, setError,
+ statusSummary,
sources: credentialValuesQuery.data?.sources ?? {},
verificationCode,
execute, actions, isConnected,
diff --git a/docs-internal/llm_gateway.md b/docs-internal/llm_gateway.md
index 6015494f..89fe572e 100644
--- a/docs-internal/llm_gateway.md
+++ b/docs-internal/llm_gateway.md
@@ -2,7 +2,19 @@
OpenCompany's hosted LLM gateway is the stock open-source LiteLLM Proxy, deployed from `gateway/` with BerriAI's official Terraform module. Invited people call LLMs on platform-owned vendor keys, each with a monthly allowance, and spend is recorded per person and per session. The repository holds only configuration and tests; there is no custom gateway code. The operator runbook (deployment, adding people, upgrades) is [gateway/README.md](../gateway/README.md).
-**Status:** the gateway (Phase 0) is in `gateway/`. The OpenCompany side (routing, the Connectors entry, attribution headers) is not implemented yet; the contract below is what it will rely on, and `gateway/tests/test_gateway_contract.py` pins each part of it against the local stack.
+**Status:** the gateway is in `gateway/`; OpenCompany's side is `server/services/gateway/` and covers chat through ChatUnifier (agents, chat runs, employee setup, translate). Embeddings, speech, vision, RLM and Claude Code still call vendors directly. `gateway/tests/test_gateway_contract.py` pins the contract below against the local stack; `server/tests/gateway/` covers OpenCompany's side.
+
+## OpenCompany's side
+
+**Turning it on.** `OPENCOMPANY_GATEWAY_ENABLED=true` (`gateway_enabled`, default off) lists the **OpenCompany Gateway** card in Connectors (`requires_setting` in `server/config/credential_providers.json`, honoured by `CredentialRegistry.get_all_providers`). `OPENCOMPANY_GATEWAY_URL` (`gateway_url`) points at the gateway. Off, behaviour is exactly as before. Distributed mode keeps it off: the gateway key has no 1Password support yet.
+
+**The key.** One gateway key per install, saved like any API key (`OpenCompanyGatewayCredential`, `services/gateway/credential.py`, default `validate`). Its probe calls the gateway's `/user/info` with the key: 401 means a wrong key; 403 means an LLM-API key, and the person is told to create a Default one. The card's `ws.status` hook (`gateway_status`, `services/gateway/handlers.py`) returns the allowance as one line, which the API-key panel shows under the key (`statusSummary` in `useCredentialPanel`).
+
+**Routing: own keys first** (`services/gateway/routing.py`). A call whose provider has a key of the install's own goes to that vendor as always. Otherwise, if the gateway is enabled, its key is stored and the provider has a `"gateway"` block in `server/config/llm_defaults.json`, `ChatUnifier.chat` builds the client with the gateway key and base URL (instead of the `{ref}_proxy` row) and passes the block's `model_prefix` to `OpenAIProvider`, which applies it to the wire model only. The gateway key stays inside the router and ChatUnifier: callers pass no key (`_resolve_activity_api_key` returns `""`), and it never enters node parameters or Temporal payloads. `ChatUnifier.fetch_models` never routes (it validates a key against its own vendor); `AIService.fetch_models` lists a gateway-covered provider's curated models with no network call. The checks that used to refuse "no key configured" accept gateway coverage (`GatewayRouter.covers`, or `gateway_covers` outside the DI graph): the three in `services/ai.py`, `_resolve_activity_api_key` and `prepare_agent_payload`, `Connections.ai_providers` (employee setup and the chat model picker), the global model list, and translate.
+
+**Attribution** (`services/gateway/attribution.py`). Ids come from the structlog log context: `BaseNode.execute` binds the workflow, execution, root execution, chat run, employee job and user ids; the Temporal activity interceptor binds `temporal_workflow_id`, `temporal_run_id`, `temporal_activity_id` and `temporal_attempt`; the LLM-step and compaction activities bind the `attribution` dict their payload carries (`_attribution_ids` in `agent_workflow.py`). `log_context` restores outer values on exit, so a tool node run inside an agent keeps the agent's ids. Headers: session = chat run, else root execution, else execution, else Temporal workflow; tags `source:`, `workflow:`, `user:`; every allowlisted id as metadata. Inside a Temporal activity, calls send a replay-cache namespace from the key's fingerprint and the Temporal workflow, run and activity ids (not the attempt), so a retried activity replays at $0 and a re-run never replays another run's answer. Gemini's native route gets the headers but no cache body (unverified there).
+
+**Errors** (`services/gateway/errors.py`, `services/llm/protocol.py`). `_classify_error` maps the `budget_exceeded` type to BILLING before any status check, so the run fails once with no retry loop, and a deployment-spawned run pauses its deployment. Failures of gateway calls get gateway wording through `public_message` and the new `public_hint` (allowance used up; model not on the gateway, from `model_cost_map_missing`; reconnect the gateway key). Compaction inside the agent workflow raises such failures instead of retrying them (`stop_on_user_action`).
## Layout
diff --git a/docs-internal/native_llm_sdk.md b/docs-internal/native_llm_sdk.md
index 2c2ef17f..7aefe762 100644
--- a/docs-internal/native_llm_sdk.md
+++ b/docs-internal/native_llm_sdk.md
@@ -147,6 +147,14 @@ the stored key as stored (no provider rewrites a key any more). The OpenAI SDK
then sends to the configured local endpoint — traffic never reaches
api.openai.com.
+**OpenCompany Gateway** — a `chat` call that arrives with no key (the install
+has none for that provider) goes to the hosted gateway when it is connected and
+the provider's `llm_defaults.json` block has a `"gateway"` entry: the gateway's
+URL replaces the `{ref}_proxy` row, the gateway key is sent, `OpenAIProvider`
+prefixes the wire model (`groq/...`), and every provider passes the per-request
+`extra_headers` / `extra_body` (attribution, replay cache). A call with a key is
+untouched. See [LLM Gateway](./llm_gateway.md).
+
**Provider detection** ([`server/constants.py:detect_ai_provider`](../server/constants.py)) MUST list `ollama` / `lmstudio` substrings, or a chat-model node falls through to `'openai'`. Agents need no edit: their `provider` field is loader-driven.
**Open-world providers** — local model names like `qwen/qwen3.6-27b` don't contain provider substrings, so the cloud-style pattern check would always reject them. A provider block declares `"open_world_models": true` (openrouter, groq, ollama, lmstudio, openai_compatible) and `is_model_valid_for_provider` accepts any id for it. The upstream server still rejects genuinely missing models with a clear 404.
diff --git a/gateway/tests/test_pins.py b/gateway/tests/test_pins.py
index f5ed90bf..b51ec031 100644
--- a/gateway/tests/test_pins.py
+++ b/gateway/tests/test_pins.py
@@ -3,7 +3,8 @@
- LiteLLM's version and the price-list commit agree across Terraform, the
local stack and CI.
- The pinned price list prices the default model of every provider the
- gateway serves, so OpenCompany's defaults are never refused as unpriced.
+ gateway serves (a "gateway" block in server/config/llm_defaults.json), so
+ OpenCompany's defaults are never refused as unpriced.
"""
from __future__ import annotations
@@ -23,18 +24,10 @@ COMPOSE = yaml.safe_load((REPO / "gateway/local/docker-compose.yml").read_text(e
CI = yaml.safe_load((REPO / ".github/workflows/gateway-ci.yml").read_text(encoding="utf-8"))
LITELLM = COMPOSE["services"]["litellm"]
-# Providers the gateway serves: OpenCompany provider id (server/config/
-# llm_defaults.json) -> the prefix of its keys in LiteLLM's price list.
-SERVED_PROVIDERS = {
- "openai": "",
- "anthropic": "",
- "gemini": "gemini/",
- "groq": "groq/",
- "cerebras": "cerebras/",
- "xai": "xai/",
- "kimi": "moonshot/",
- "mistral": "mistral/",
-}
+# The providers the gateway serves are the ones with a "gateway" block in
+# server/config/llm_defaults.json (OpenCompany's services/gateway/routing.py).
+PROVIDERS = json.loads((REPO / "server/config/llm_defaults.json").read_text(encoding="utf-8"))["providers"]
+SERVED = {name: config["gateway"] for name, config in PROVIDERS.items() if isinstance(config.get("gateway"), dict)}
def _terraform_string(name: str) -> str:
@@ -61,11 +54,14 @@ def test_served_default_models_are_priced() -> None:
if os.environ.get("CI"):
raise
pytest.skip(f"price list unreachable: {exc}")
- providers = json.loads((REPO / "server/config/llm_defaults.json").read_text(encoding="utf-8"))["providers"]
+ assert SERVED, "no provider in llm_defaults.json has a gateway block"
unpriced = []
- for provider, prefix in SERVED_PROVIDERS.items():
- key = prefix + providers[provider]["default_model"]
- entry = prices.get(key) or {}
+ for provider, block in SERVED.items():
+ model = PROVIDERS[provider]["default_model"]
+ # LiteLLM prices a model under its wire name, its bare id, or (for
+ # the native Gemini route, whose wire name is bare) gemini/<id>.
+ candidates = (block.get("model_prefix", "") + model, model, f"{provider}/{model}")
+ entry = next((prices[key] for key in candidates if key in prices), {})
if "input_cost_per_token" not in entry and "output_cost_per_token" not in entry:
- unpriced.append(key)
+ unpriced.append(f"{provider}: {model}")
assert not unpriced, f"served default models with no price in the pinned list: {unpriced}"
diff --git a/server/config/credential_providers.json b/server/config/credential_providers.json
index 2a3e5e22..e0de1ff3 100644
--- a/server/config/credential_providers.json
+++ b/server/config/credential_providers.json
@@ -113,6 +113,34 @@
}
]
},
+ "opencompany_gateway": {
+ "_description": "The hosted OpenCompany Gateway (gateway/, docs-internal/llm_gateway.md). One key per install, stored like any API key; LLM calls for a provider the install has no key of its own for go through the gateway on the key owner's monthly allowance (services/gateway). Not an LLM provider itself: no llm_defaults.json block, so it never appears in model pickers. Listed only while OPENCOMPANY_GATEWAY_ENABLED is on (requires_setting). ws.status feeds the card's allowance line.",
+ "name": "OpenCompany Gateway",
+ "publisher": "OpenCompany",
+ "verified": true,
+ "consumer_category": "ai",
+ "description": "Use your OpenCompany allowance for AI models you have no key for.",
+ "category": "ai",
+ "category_label": "AI Providers",
+ "color": "dracula.purple",
+ "kind": "apiKey",
+ "icon_ref": "lucide:Sparkles",
+ "requires_setting": "gateway_enabled",
+ "ws": {
+ "status": "gateway_status"
+ },
+ "fields": [
+ {
+ "key": "apiKey",
+ "label": "Gateway key",
+ "type": "password",
+ "secret": true,
+ "required": true,
+ "placeholder": "sk-...",
+ "help": "Create a key in the OpenCompany Gateway console (Virtual Keys, key type Default) and paste it here. Your own provider keys still take precedence."
+ }
+ ]
+ },
"openai": {
"extends": "_ai_base",
"name": "OpenAI",
diff --git a/server/config/llm_defaults.json b/server/config/llm_defaults.json
index 1b86ec91..b0d017b6 100644
--- a/server/config/llm_defaults.json
+++ b/server/config/llm_defaults.json
@@ -26,6 +26,7 @@
"providers": {
"openai": {
"display_name": "OpenAI",
+ "gateway": {"model_prefix": "openai/", "base_path": "/v1"},
"streaming": true,
"default_model": "gpt-6-sol",
"popular_models": ["gpt-6-sol", "gpt-6-astra", "gpt-6-astra-pro", "gpt-6-sol-pro", "gpt-6-luna", "gpt-6-luna-pro", "gpt-5.6-sol", "gpt-5.6-sol-pro", "gpt-5.6-terra", "gpt-5.6-terra-pro", "gpt-5.6-luna", "gpt-5.6-luna-pro", "gpt-5.5", "gpt-5.4", "gpt-4.1"],
@@ -86,6 +87,7 @@
},
"anthropic": {
"display_name": "Anthropic",
+ "gateway": {"model_prefix": "", "base_path": ""},
"streaming": true,
"default_model": "claude-opus-5-5",
"popular_models": ["claude-opus-5-5", "claude-fable-5-1", "claude-fable-5", "claude-opus-5", "claude-sonnet-5-5", "claude-sonnet-5", "claude-opus-4-8", "claude-opus-4-7", "claude-sonnet-4-6"],
@@ -157,6 +159,7 @@
},
"gemini": {
"display_name": "Gemini",
+ "gateway": {"model_prefix": "", "base_path": ""},
"default_model": "gemini-3.8-flash",
"popular_models": ["gemini-3.8-flash", "gemini-3.7-flash", "gemini-3.6-flash", "gemini-3.5-flash", "gemini-3.5-flash-lite", "gemini-3.1-pro-preview", "gemini-3.1-flash-lite", "gemini-3-flash-preview", "gemini-2.5-pro", "gemini-2.5-flash", "gemini-2.5-flash-lite"],
"detection_patterns": ["gemini", "google", "gemma"],
@@ -237,6 +240,7 @@
},
"groq": {
"display_name": "Groq",
+ "gateway": {"model_prefix": "groq/", "base_path": "/v1"},
"default_model": "openai/gpt-oss-120b",
"popular_models": [],
"_popular_note": "Empty per the 1M-context policy — no Groq-served model reaches 1M (all top out at 131072). Models remain reachable via live fetch_models.",
@@ -272,6 +276,7 @@
},
"cerebras": {
"display_name": "Cerebras",
+ "gateway": {"model_prefix": "cerebras/", "base_path": "/v1"},
"default_model": "gpt-oss-120b",
"popular_models": [],
"_popular_note": "Empty per the 1M-context policy — no Cerebras model reaches 1M (all 131072). Models remain reachable via live fetch_models.",
@@ -298,6 +303,7 @@
},
"xai": {
"display_name": "xAI",
+ "gateway": {"model_prefix": "xai/", "base_path": "/v1"},
"default_model": "grok-4.20",
"popular_models": ["grok-4.20", "grok-4.20-multi-agent", "grok-4.3"],
"_popular_note": "grok-4.7, grok-4.6 and grok-4.5 are excluded per the 1M-context policy (500K each); reachable via live fetch_models. Default stays grok-4.20 (2M) so the default is selectable from the global list.",
@@ -356,6 +362,7 @@
},
"kimi": {
"display_name": "Kimi",
+ "gateway": {"model_prefix": "moonshot/", "base_path": "/v1"},
"default_model": "kimi-k3",
"popular_models": ["kimi-k3"],
"_popular_note": "Only kimi-k3 meets the 1M-context policy (k2.x are 262144); k2.x remain reachable via live fetch_models.",
@@ -384,6 +391,7 @@
},
"mistral": {
"display_name": "Mistral",
+ "gateway": {"model_prefix": "mistral/", "base_path": "/v1"},
"default_model": "mistral-large-latest",
"popular_models": [],
"_popular_note": "Empty per the 1M-context policy — Mistral tops out at 262144 (large/medium/small). Models remain reachable via live fetch_models.",
diff --git a/server/core/config.py b/server/core/config.py
index f4f54c3b..c9ea3b65 100644
--- a/server/core/config.py
+++ b/server/core/config.py
@@ -356,6 +356,12 @@ class Settings(BaseSettings):
# AI Proxy Configuration (Ollama-style proxy)
ai_proxy_default_port: int = Field(default=11434, env="AI_PROXY_DEFAULT_PORT")
+ # OpenCompany Gateway (hosted LiteLLM, gateway/): with a gateway key in
+ # Connectors, LLM calls for a provider this install has no key of its
+ # own for go through the gateway on the person's allowance.
+ gateway_enabled: bool = Field(default=False, validation_alias="OPENCOMPANY_GATEWAY_ENABLED")
+ gateway_url: str = Field(default="https://api.gw.opencompany.sh", validation_alias="OPENCOMPANY_GATEWAY_URL")
+
# Residential Proxy Configuration
proxy_enabled: bool = Field(default=False, env="PROXY_ENABLED")
proxy_budget_daily_usd: float = Field(default=50.0, env="PROXY_BUDGET_DAILY_USD")
diff --git a/server/core/container.py b/server/core/container.py
index 56ac75e3..ff4f804b 100644
--- a/server/core/container.py
+++ b/server/core/container.py
@@ -54,7 +54,18 @@ def _create_temporal_client(server_address: str, namespace: str):
return TemporalClientWrapper(server_address=server_address, namespace=namespace)
-def _build_chat_unifier(auth_service):
+def _build_gateway_router(settings, auth_service):
+ """The OpenCompany Gateway router (services/gateway/routing.py).
+
+ Imported here, not at module scope, so the container's import doesn't
+ register the gateway's Connectors handlers as a side effect.
+ """
+ from services.gateway.routing import GatewayRouter
+
+ return GatewayRouter(settings=settings, auth_service=auth_service, defaults=LLM_DEFAULTS)
+
+
+def _build_chat_unifier(auth_service, gateway_router):
"""Construct the chat-model unifier with the shared JSON defaults.
Wrapped in a factory function so the DI container doesn't have to
@@ -62,7 +73,7 @@ def _build_chat_unifier(auth_service):
loaded once at import time by ``services.llm.config`` and reflects
the on-disk ``server/config/llm_defaults.json``.
"""
- return ChatUnifier(defaults=LLM_DEFAULTS, auth_service=auth_service)
+ return ChatUnifier(defaults=LLM_DEFAULTS, auth_service=auth_service, gateway=gateway_router)
# Wave 12 C4 sub-piece C: plugin-owned service factories.
@@ -151,7 +162,11 @@ class Container(containers.DeclarativeContainer):
# JSON-driven typed-exception translation + ``incompatible_models``
# filter. ``AIService`` delegates to this rather than calling
# provider classes directly.
- chat_unifier = providers.Singleton(_build_chat_unifier, auth_service=auth_service)
+ # Own keys first; a provider with no key of the install's own runs on
+ # the OpenCompany Gateway when it is connected (services/gateway).
+ gateway_router = providers.Singleton(_build_gateway_router, settings=settings, auth_service=auth_service)
+
+ chat_unifier = providers.Singleton(_build_chat_unifier, auth_service=auth_service, gateway_router=gateway_router)
ai_service = providers.Singleton(
AIService,
@@ -160,6 +175,7 @@ class Container(containers.DeclarativeContainer):
cache=cache,
settings=settings,
chat_unifier=chat_unifier,
+ gateway_router=gateway_router,
)
maps_service = providers.Factory(_build_maps_service, auth_service=auth_service, settings=settings)
diff --git a/server/core/logging.py b/server/core/logging.py
index a8b132aa..f24c49c6 100644
--- a/server/core/logging.py
+++ b/server/core/logging.py
@@ -407,23 +407,20 @@ async def log_context(**fields: Any) -> AsyncIterator[None]:
async with log_context(workflow_id=wf_id, node_id=node_id):
await do_work() # all logs inside carry workflow_id + node_id
- Bindings clear on ``__aexit__`` even if the inner block raises.
+ On ``__aexit__`` each field goes back to the value it had before, even if
+ the inner block raises: a node executed inside another node (a tool call)
+ must not wipe its parent's fields. The LLM gateway's attribution headers
+ read the same fields (services.gateway.attribution).
"""
- structlog.contextvars.bind_contextvars(**fields)
- try:
+ with structlog.contextvars.bound_contextvars(**fields):
yield
- finally:
- structlog.contextvars.unbind_contextvars(*fields.keys())
@contextmanager
def log_context_sync(**fields: Any) -> Iterator[None]:
"""Synchronous variant of :func:`log_context` for sync callers."""
- structlog.contextvars.bind_contextvars(**fields)
- try:
+ with structlog.contextvars.bound_contextvars(**fields):
yield
- finally:
- structlog.contextvars.unbind_contextvars(*fields.keys())
def log_execution_time(logger: structlog.BoundLogger, operation: str, start_time: float, end_time: float, **kwargs) -> None:
diff --git a/server/main.py b/server/main.py
index 0e49485c..e2f86738 100644
--- a/server/main.py
+++ b/server/main.py
@@ -154,6 +154,10 @@ async def lifespan(app: FastAPI):
# save_api_key / delete_api_key).
import services.credentials # noqa: F401
+ # services/gateway/__init__.py registers the OpenCompany Gateway's
+ # Connectors credential and its gateway_status handler.
+ import services.gateway # noqa: F401
+
# Wave 13.7: services/workflow_storage/__init__.py self-registers the
# 5 workflow-record CRUD handlers (save_workflow / import_workflow /
# get_workflow / get_all_workflows / delete_workflow).
diff --git a/server/nodes/translate/_base.py b/server/nodes/translate/_base.py
index a5cdf049..f3d4d03a 100644
--- a/server/nodes/translate/_base.py
+++ b/server/nodes/translate/_base.py
@@ -20,10 +20,21 @@ async def provider_api_key(ctx: NodeContext, provider: str) -> str:
``BaseNode.execute`` turns into a credential envelope plus a CloudEvents
broadcast — lighting up the right provider in the Credentials modal.
"""
+ from services.gateway.routing import gateway_covers
+
credential_id = translate_config.credential_id(provider)
- async with ctx.connection(credential_id) as conn:
- secrets = await conn.credentials()
+ try:
+ async with ctx.connection(credential_id) as conn:
+ secrets = await conn.credentials()
+ except PermissionError:
+ # No key of the install's own: an LLM provider the OpenCompany
+ # Gateway serves still runs, through ChatUnifier (services/gateway).
+ if await gateway_covers(provider):
+ return ""
+ raise
api_key = str(secrets.get("api_key") or "")
+ if not api_key and await gateway_covers(provider):
+ return ""
if not api_key:
raise NodeUserError(
f"No API key stored for '{credential_id}'. Add one in the "
diff --git a/server/services/ai.py b/server/services/ai.py
index 68b93f61..9aae4695 100644
--- a/server/services/ai.py
+++ b/server/services/ai.py
@@ -349,6 +349,7 @@ class AIService:
cache,
settings: Settings,
chat_unifier=None,
+ gateway_router=None,
):
self.auth = auth_service
self.database = database
@@ -361,6 +362,10 @@ class AIService:
# the DI container can still instantiate ``AIService`` (the chat
# paths will fall back to direct factory calls).
self.chat_unifier = chat_unifier
+ # services/gateway/routing.py: a provider with no key of the
+ # install's own may still run on the OpenCompany Gateway.
+ self.gateway_router = gateway_router
+ # RLM service (lazy import to avoid circular deps)
# RLM service (lazy import to avoid circular deps)
from services.rlm import RLMService
@@ -370,6 +375,11 @@ class AIService:
"""Detect AI provider from model name."""
return detect_provider_from_model(model)
+ async def _gateway_covers(self, provider: str) -> bool:
+ """No key of the install's own, but the OpenCompany Gateway serves
+ ``provider``: ChatUnifier sends the call there (services/gateway)."""
+ return self.gateway_router is not None and await self.gateway_router.covers(provider)
+
async def _prepare_context(
self,
*,
@@ -647,7 +657,8 @@ class AIService:
)
# Trigger compaction if threshold exceeded
- if tracking.get("needs_compaction") and memory_content and api_key:
+ # No own key is fine when the OpenCompany Gateway serves the provider.
+ if tracking.get("needs_compaction") and memory_content and (api_key or await self._gateway_covers(provider)):
logger.info(f"[Compaction] Threshold exceeded for session {session_id}, triggering compaction")
if broadcaster:
@@ -708,6 +719,11 @@ class AIService:
"ChatUnifier is not injected. AIService must be constructed via "
"the DI container (core.container.Container)."
)
+ if not api_key and await self._gateway_covers(provider):
+ # No key of the install's own: the gateway serves this provider.
+ # Its curated models, with no network call: a gateway key must
+ # never reach a vendor's model-list URL.
+ return self._get_curated_models(provider)
try:
return await self.chat_unifier.fetch_models(provider=provider, api_key=api_key)
except NodeUserError:
@@ -750,7 +766,7 @@ class AIService:
provider = detect_ai_provider(node_type, flattened)
- if not api_key:
+ if not api_key and not await self._gateway_covers(provider):
_require_saved_endpoint(provider)
raise ValueError("API key is required")
@@ -961,7 +977,7 @@ class AIService:
else:
logger.info(f"No model specified, using default: {model}")
- if not api_key:
+ if not api_key and not await self._gateway_covers(provider):
_require_saved_endpoint(provider)
raise ValueError("API key is required for AI Agent")
@@ -1804,7 +1820,7 @@ class AIService:
else:
logger.info(f"No model specified, using default: {model}")
- if not api_key:
+ if not api_key and not await self._gateway_covers(provider):
_require_saved_endpoint(provider)
raise ValueError("API key is required for Zeenie")
diff --git a/server/services/compaction.py b/server/services/compaction.py
index d27882d4..7a92b07b 100644
--- a/server/services/compaction.py
+++ b/server/services/compaction.py
@@ -20,6 +20,7 @@ from datetime import datetime, timezone
from typing import Dict, Any, Optional, TYPE_CHECKING
from core.logging import get_logger
+from services.llm.protocol import LLMError
from services.pricing import get_pricing_service
if TYPE_CHECKING:
@@ -304,10 +305,16 @@ class CompactionService:
model: str,
*,
explicit_max_retries: int = 2,
+ stop_on_user_action: bool = False,
) -> Dict[str, Any]:
"""Perform compaction by summarizing conversation history.
Uses the AI service to generate a structured summary following Claude Code pattern.
+
+ A failed summarizer call returns ``success: False``. With
+ ``stop_on_user_action``, a failure only the user can fix (a spent
+ allowance or exhausted credits, a rejected key, missing access) raises
+ the :class:`LLMError` instead, since retrying compaction cannot help.
"""
if not self._ai_service:
logger.warning("[Compaction] AI service not wired")
@@ -367,6 +374,7 @@ Provide a concise but complete summary."""
max_tokens=summary_tokens,
sdk_max_retries=0,
explicit_max_retries=explicit_max_retries,
+ translate_errors=False,
)
compaction_usage = asdict(response.usage)
await self._record_usage_metric(
@@ -394,6 +402,16 @@ Provide a concise but complete summary."""
"usage": compaction_usage,
}
+ except LLMError as e:
+ if stop_on_user_action and e.requires_user_action:
+ raise
+ # user_message: the raw SDK text can carry payload fragments.
+ logger.error(f"[Compaction] Failed: {e.user_message}")
+ return {
+ "success": False,
+ "error": e.user_message,
+ "usage": compaction_usage,
+ }
except Exception as e:
logger.error(f"[Compaction] Failed: {e}")
return {
diff --git a/server/services/credential_registry.py b/server/services/credential_registry.py
index d16a4442..ec334ed4 100644
--- a/server/services/credential_registry.py
+++ b/server/services/credential_registry.py
@@ -125,14 +125,20 @@ class CredentialRegistry:
# ----- public API -----
def get_all_providers(self) -> List[Dict[str, Any]]:
- """Return all concrete (non-abstract) providers in JSON order."""
+ """Return all concrete (non-abstract) providers in JSON order.
+
+ An entry declaring ``requires_setting`` is listed only while that
+ ``Settings`` attribute is ``True`` (the OpenCompany Gateway's card,
+ hidden unless the gateway is enabled). ``get_provider`` still
+ resolves it by id.
+ """
resolved = self._resolve_all()
# Preserve insertion order of the raw providers dict.
raw = self._load_raw().get("providers", {})
out: List[Dict[str, Any]] = []
for provider_id in raw:
entry = resolved[provider_id]
- if entry.get("_abstract"):
+ if entry.get("_abstract") or not _setting_enabled(entry):
continue
out.append(entry)
return out
@@ -245,6 +251,17 @@ class CredentialRegistry:
# ----- deep-merge helper (n8n mergeNodeProperties pattern) -----
+def _setting_enabled(entry: Dict[str, Any]) -> bool:
+ """Whether an entry's ``requires_setting`` (if any) is on. ``is True``:
+ under test the container's Settings is a MagicMock."""
+ name = entry.get("requires_setting")
+ if not name:
+ return True
+ from core.container import container
+
+ return getattr(container.settings(), str(name), False) is True
+
+
def _deep_merge(parent: Dict[str, Any], child: Dict[str, Any]) -> Dict[str, Any]:
"""Return a new dict: parent deep-merged with child overrides.
diff --git a/server/services/employees/connections.py b/server/services/employees/connections.py
index 03ac018f..edda2659 100644
--- a/server/services/employees/connections.py
+++ b/server/services/employees/connections.py
@@ -88,10 +88,12 @@ class Connections:
async def ai_providers(self) -> List[str]:
"""LLM providers an agent could run on right now: one with a key
- stored or a local server saved, and each saved OpenAI-compatible
- endpoint by its reference (``openai_compatible:<slug>``). The bare
- ``openai_compatible`` id is never one: an agent set to it has no
- server to call."""
+ stored or a local server saved, one the OpenCompany Gateway serves
+ when it is connected (services/gateway), and each saved
+ OpenAI-compatible endpoint by its reference
+ (``openai_compatible:<slug>``). The bare ``openai_compatible`` id is
+ never one: an agent set to it has no server to call."""
+ from services.gateway.routing import gateway_covers
from services.llm.config import ENDPOINT_PROVIDER, PROVIDER_CONFIGS
usable = []
@@ -100,6 +102,8 @@ class Connections:
continue
if self.provider(provider_id) is not None and await self.is_connected(provider_id):
usable.append(provider_id)
+ elif await gateway_covers(provider_id):
+ usable.append(provider_id)
try:
from services.llm.endpoints import list_endpoints
diff --git a/server/services/gateway/__init__.py b/server/services/gateway/__init__.py
new file mode 100644
index 00000000..64f0a9cb
--- /dev/null
+++ b/server/services/gateway/__init__.py
@@ -0,0 +1,20 @@
+"""OpenCompany Gateway: LLM calls on a hosted, capped allowance.
+
+The gateway itself is the stock LiteLLM Proxy in ``gateway/``
+(docs-internal/llm_gateway.md). This package is OpenCompany's side:
+
+- ``routing``: which calls go to the gateway (own keys first) and where.
+- ``attribution``: the headers that put each call on its spend row.
+- ``errors``: what the user is told when a gateway call fails.
+- ``credential`` / ``handlers``: the Connectors card for the gateway key.
+
+Side-effect import (server/main.py) registers the credential and the
+``gateway_status`` handler.
+"""
+
+from services.ws_handler_registry import register_ws_handlers as _register_ws_handlers
+
+from . import credential as _credential # noqa: F401 (registers OpenCompanyGatewayCredential)
+from .handlers import WS_HANDLERS as _WS_HANDLERS
+
+_register_ws_handlers(_WS_HANDLERS)
diff --git a/server/services/gateway/allowance.py b/server/services/gateway/allowance.py
new file mode 100644
index 00000000..69def972
--- /dev/null
+++ b/server/services/gateway/allowance.py
@@ -0,0 +1,62 @@
+"""The monthly allowance a gateway key draws on, read from the gateway.
+
+LiteLLM's ``GET /user/info`` with the key itself (no ``user_id``) returns
+its owner's spend, limit and reset time. Only Default-type keys may call it;
+LLM-API keys get 401/403 (gateway/tests pins both).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime
+from typing import Any, Mapping, Optional
+
+import httpx
+
+TIMEOUT_SECONDS = 10.0
+
+
+@dataclass(frozen=True)
+class Allowance:
+ spend: float
+ max_budget: Optional[float]
+ reset_at: Optional[datetime]
+ account: Optional[str]
+
+ @property
+ def summary(self) -> str:
+ """``$12.40 of $50.00 used · resets Nov 1``."""
+ used = f"${self.spend:,.2f}"
+ text = f"{used} of ${self.max_budget:,.2f} used" if self.max_budget is not None else f"{used} used"
+ if self.reset_at is not None:
+ text += f" · resets {self.reset_at.strftime('%b')} {self.reset_at.day}"
+ return text
+
+
+def _reset_time(value: Any) -> Optional[datetime]:
+ if not isinstance(value, str) or not value:
+ return None
+ try:
+ return datetime.fromisoformat(value.replace("Z", "+00:00"))
+ except ValueError:
+ return None
+
+
+def parse_user_info(body: Mapping[str, Any]) -> Allowance:
+ info = body.get("user_info") if isinstance(body.get("user_info"), Mapping) else {}
+ max_budget = info.get("max_budget")
+ return Allowance(
+ spend=float(info.get("spend") or 0.0),
+ max_budget=float(max_budget) if max_budget is not None else None,
+ reset_at=_reset_time(info.get("budget_reset_at")),
+ account=info.get("user_email") or body.get("user_id") or info.get("user_id"),
+ )
+
+
+async def user_info(gateway_url: str, api_key: str) -> httpx.Response:
+ """``GET /user/info`` with ``api_key``; the caller reads the status."""
+ async with httpx.AsyncClient(timeout=TIMEOUT_SECONDS) as client:
+ return await client.get(
+ f"{gateway_url.rstrip('/')}/user/info",
+ headers={"Authorization": f"Bearer {api_key}"},
+ )
diff --git a/server/services/gateway/attribution.py b/server/services/gateway/attribution.py
new file mode 100644
index 00000000..c76d1a79
--- /dev/null
+++ b/server/services/gateway/attribution.py
@@ -0,0 +1,86 @@
+"""What a call the OpenCompany Gateway serves says about itself.
+
+Three LiteLLM headers put each call on its spend row: the session (the chat
+run, else the root execution, else the execution, else the Temporal
+workflow), a few low-cardinality tags, and every id as metadata. The ids come
+from the log context (``core.logging.log_context``), bound by
+``BaseNode.execute``, the Temporal activity interceptor and the agent
+activities, so whatever a log line carries, the spend row carries. Only the
+allowlisted ids below are sent; nothing else from the log context leaves the
+install.
+
+Inside a Temporal activity a call also opts in to the gateway's response
+cache under a namespace that stays the same across that activity's retries,
+so a retried call replays at $0 instead of being billed twice.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from typing import Any, Dict, Optional
+
+import structlog
+
+from core.encryption import fingerprint_credential
+
+#: First present id names the session.
+SESSION_KEYS = ("chat_run_id", "root_execution_id", "execution_id", "temporal_workflow_id")
+#: Ids sent as spend-log metadata. Never extend this with free text.
+METADATA_KEYS = (
+ "chat_run_id",
+ "root_execution_id",
+ "execution_id",
+ "workflow_id",
+ "node_id",
+ "node_type",
+ "employee_job_id",
+ "user_id",
+ "temporal_workflow_id",
+ "temporal_run_id",
+ "temporal_activity_id",
+ "temporal_attempt",
+)
+#: How long a replayable response stays cached (LiteLLM cache_params ttl).
+REPLAY_TTL_SECONDS = 172_800
+
+
+def _ids() -> Dict[str, str]:
+ bound = structlog.contextvars.get_contextvars()
+ return {key: str(bound[key]) for key in METADATA_KEYS if bound.get(key) not in (None, "")}
+
+
+def request_headers() -> Dict[str, str]:
+ """The attribution headers for the current log context."""
+ ids = _ids()
+ headers: Dict[str, str] = {}
+ session = next((ids[key] for key in SESSION_KEYS if key in ids), None)
+ if session:
+ headers["x-litellm-session-id"] = session
+ source = "chat" if "chat_run_id" in ids else ("workflow" if "workflow_id" in ids else "direct")
+ tags = [f"source:{source}"]
+ if "workflow_id" in ids:
+ tags.append(f"workflow:{ids['workflow_id']}")
+ if "user_id" in ids:
+ tags.append(f"user:{ids['user_id']}")
+ headers["x-litellm-tags"] = ",".join(tags)
+ if ids:
+ headers["x-litellm-spend-logs-metadata"] = json.dumps(ids, sort_keys=True, separators=(",", ":"))
+ return headers
+
+
+def replay_cache(api_key: str) -> Optional[Dict[str, Any]]:
+ """The replay-cache request body inside a Temporal activity, else ``None``.
+
+ The namespace is the gateway key's fingerprint plus the Temporal workflow,
+ run and activity ids: an activity's retries share it, and a re-run of the
+ same workflow id (a trigger re-firing a failed event) gets a new run id,
+ so it never replays another run's answer.
+ """
+ bound = structlog.contextvars.get_contextvars()
+ scope = [bound.get(key) for key in ("temporal_workflow_id", "temporal_run_id", "temporal_activity_id")]
+ if not all(scope):
+ return None
+ digest = hashlib.sha256("\x1f".join(str(part) for part in scope).encode("utf-8")).hexdigest()[:32]
+ namespace = f"oc1-{fingerprint_credential(api_key)[:16]}-{digest}"
+ return {"cache": {"use-cache": True, "namespace": namespace, "ttl": REPLAY_TTL_SECONDS}}
diff --git a/server/services/gateway/credential.py b/server/services/gateway/credential.py
new file mode 100644
index 00000000..a252d490
--- /dev/null
+++ b/server/services/gateway/credential.py
@@ -0,0 +1,39 @@
+"""The gateway key as a Connectors credential.
+
+A standard API-key connector: the key is stored install-wide like every
+other key (``Credential.validate`` in services/plugin/credential.py). The
+probe reads the key's allowance from the gateway, which also tells a
+Default key (needed to show the allowance) from an LLM-API one.
+"""
+
+from __future__ import annotations
+
+from services.plugin.credential import ApiKeyCredential, ProbeResult
+
+from .allowance import parse_user_info, user_info
+from .routing import GATEWAY_CREDENTIAL_ID
+
+
+class OpenCompanyGatewayCredential(ApiKeyCredential):
+ id = GATEWAY_CREDENTIAL_ID
+ display_name = "OpenCompany Gateway"
+ category = "AI"
+ key_location = "bearer"
+ docs_url = "https://docs.litellm.ai/docs/proxy/virtual_keys"
+
+ @classmethod
+ async def _probe(cls, api_key: str) -> ProbeResult:
+ from core.container import container
+
+ response = await user_info(container.settings().gateway_url, api_key)
+ if response.status_code == 403:
+ return ProbeResult(
+ valid=False,
+ message=(
+ "This key can't read its allowance. In the OpenCompany Gateway console, "
+ "create a key of type Default and paste that one."
+ ),
+ )
+ # 401 and 5xx: Credential.validate words them (classify_credential_error).
+ response.raise_for_status()
+ return ProbeResult(valid=True, message=f"Connected: {parse_user_info(response.json()).summary}.")
diff --git a/server/services/gateway/errors.py b/server/services/gateway/errors.py
new file mode 100644
index 00000000..3c73fdcf
--- /dev/null
+++ b/server/services/gateway/errors.py
@@ -0,0 +1,42 @@
+"""What the user is told when a call the OpenCompany Gateway served fails.
+
+The category texts name the vendor and point at its billing or key ("OpenAI
+blocked this request because a spending limit ... was exhausted"), which is
+the wrong account for a call billed to an OpenCompany allowance and made
+with the gateway key. ChatUnifier passes every failure of a gateway call
+through :func:`explain`; the category, and so retry and pause behaviour,
+stays as classified.
+"""
+
+from __future__ import annotations
+
+from typing import Optional
+
+from services.llm.protocol import LLMError, LLMErrorCategory, error_type
+
+_RECONNECT = "Reconnect the OpenCompany Gateway in Settings > Connectors, then resume."
+_OWN_KEY = "add your own key for this provider in Settings > Connectors"
+
+
+def explain(error: LLMError, exc: Optional[BaseException]) -> LLMError:
+ """Set gateway wording on ``error`` (in place) and return it."""
+ kind = error_type(exc) if exc is not None else None
+ if error.category is LLMErrorCategory.BILLING:
+ error.public_message = "Your OpenCompany allowance for this month is used up."
+ error.public_hint = f"Ask your OpenCompany admin to raise it, or {_OWN_KEY}, then resume."
+ elif kind == "model_cost_map_missing":
+ error.public_message = "This model isn't available on the OpenCompany Gateway yet."
+ error.public_hint = f"Choose another model, or {_OWN_KEY}."
+ elif error.category is LLMErrorCategory.AUTHENTICATION:
+ error.public_message = "The OpenCompany Gateway rejected the gateway key."
+ error.public_hint = _RECONNECT
+ elif error.category is LLMErrorCategory.PERMISSION:
+ error.public_message = "The OpenCompany Gateway denied this request."
+ error.public_hint = _RECONNECT
+ elif error.category is LLMErrorCategory.RATE_LIMIT:
+ error.public_message = "The OpenCompany Gateway is rate-limiting requests. Retry after a short delay."
+ elif error.category in (LLMErrorCategory.CONNECTION, LLMErrorCategory.TIMEOUT):
+ error.public_message = "Could not reach the OpenCompany Gateway."
+ elif error.category is LLMErrorCategory.SERVER:
+ error.public_message = "The OpenCompany Gateway is temporarily unavailable."
+ return error
diff --git a/server/services/gateway/handlers.py b/server/services/gateway/handlers.py
new file mode 100644
index 00000000..a721f0c2
--- /dev/null
+++ b/server/services/gateway/handlers.py
@@ -0,0 +1,49 @@
+"""WebSocket handlers for the OpenCompany Gateway's Connectors card.
+
+``gateway_status`` is the card's ``ws.status`` hook
+(server/config/credential_providers.json): the allowance the stored gateway
+key draws on, as one ``summary`` line the card shows.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+import httpx
+
+from core.logging import get_logger
+from services.ws_handler_registry import ws_handler
+
+from .allowance import parse_user_info, user_info
+from .routing import GATEWAY_CREDENTIAL_ID
+
+logger = get_logger(__name__)
+
+
+@ws_handler()
+async def handle_gateway_status(data: Dict[str, Any], websocket: Any) -> Dict[str, Any]:
+ from core.container import container
+
+ api_key = await container.auth_service().get_api_key(GATEWAY_CREDENTIAL_ID)
+ if not api_key:
+ return {"connected": False}
+ try:
+ response = await user_info(container.settings().gateway_url, api_key)
+ response.raise_for_status()
+ except httpx.HTTPError as exc:
+ logger.warning("Gateway allowance read failed", error=type(exc).__name__)
+ return {"connected": True, "summary": "Couldn't read the allowance from the OpenCompany Gateway."}
+ allowance = parse_user_info(response.json())
+ return {
+ "connected": True,
+ "summary": allowance.summary,
+ "account_label": allowance.account,
+ "spend": allowance.spend,
+ "max_budget": allowance.max_budget,
+ "budget_reset_at": allowance.reset_at.isoformat() if allowance.reset_at else None,
+ }
+
+
+WS_HANDLERS = {
+ "gateway_status": handle_gateway_status,
+}
diff --git a/server/services/gateway/routing.py b/server/services/gateway/routing.py
new file mode 100644
index 00000000..d95b5f5d
--- /dev/null
+++ b/server/services/gateway/routing.py
@@ -0,0 +1,93 @@
+"""Which LLM calls go through the OpenCompany Gateway, and where to.
+
+Own keys first: a provider this install has a key for is called directly, as
+always. Otherwise, when the gateway is enabled and connected and serves the
+provider, ChatUnifier sends the call to the gateway with the install's
+gateway key. The key never leaves this module and ChatUnifier: it is not
+injected into node parameters, Temporal payloads or requests to a vendor.
+
+Which providers the gateway serves, and how it names their models, is data:
+the ``"gateway"`` block of each served provider in
+``server/config/llm_defaults.json``. OpenAI-compatible providers use the
+gateway's ``/v1`` routes with a LiteLLM provider prefix on the model name
+(``groq/...``); Anthropic and Gemini use the gateway's native routes with
+bare model ids.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any, List, Mapping, Optional
+
+from services.llm.config import split_provider_ref
+
+#: Catalogue entry and credential id of the gateway key (Connectors).
+GATEWAY_CREDENTIAL_ID = "opencompany_gateway"
+
+
+@dataclass(frozen=True)
+class GatewayRoute:
+ """Where one LLM call goes when the gateway serves it."""
+
+ api_key: str
+ base_url: str
+ model_prefix: str
+
+
+class GatewayRouter:
+ """Decides, per provider, whether the gateway serves a call."""
+
+ def __init__(self, *, settings: Any, auth_service: Any, defaults: Mapping[str, Any]) -> None:
+ self._settings = settings
+ self._auth = auth_service
+ self._defaults = defaults
+
+ def enabled(self) -> bool:
+ # ``is True``: under test the container's Settings is a MagicMock,
+ # whose attributes are all truthy. Distributed mode keeps credentials
+ # in 1Password only, which the gateway key doesn't support yet.
+ return (
+ getattr(self._settings, "gateway_enabled", False) is True
+ and getattr(self._settings, "distributed_mode", False) is not True
+ )
+
+ def _block(self, provider: str) -> Optional[Mapping[str, Any]]:
+ providers = self._defaults.get("providers") or {}
+ config = providers.get(split_provider_ref(provider)[0]) or {}
+ block = config.get("gateway") if isinstance(config, Mapping) else None
+ return block if isinstance(block, Mapping) else None
+
+ def served_providers(self) -> List[str]:
+ """Provider ids the gateway serves (a ``"gateway"`` block in llm_defaults.json)."""
+ providers = self._defaults.get("providers") or {}
+ return [name for name in providers if self._block(name) is not None]
+
+ async def route(self, provider: str) -> Optional[GatewayRoute]:
+ """The route for a call to ``provider`` with no key of the install's own."""
+ if not self.enabled():
+ return None
+ block = self._block(provider)
+ if block is None:
+ return None
+ api_key = await self._auth.get_api_key(GATEWAY_CREDENTIAL_ID)
+ if not api_key:
+ return None
+ base_url = str(self._settings.gateway_url).rstrip("/") + str(block.get("base_path") or "")
+ return GatewayRoute(api_key=api_key, base_url=base_url, model_prefix=str(block.get("model_prefix") or ""))
+
+ async def covers(self, provider: str) -> bool:
+ """Whether a call to ``provider`` can run on the gateway: what a "no key
+ configured" check asks before refusing."""
+ return await self.route(provider) is not None
+
+
+async def gateway_covers(provider: str) -> bool:
+ """:meth:`GatewayRouter.covers` on the container's router, for call sites
+ outside the DI graph. Under test the container is a MagicMock (whose
+ router would claim to cover everything) or a stub without a router, so
+ only a real :class:`GatewayRouter` is asked."""
+ from core.container import container
+
+ factory = getattr(container, "gateway_router", None)
+ router = factory() if callable(factory) else None
+ return isinstance(router, GatewayRouter) and await router.covers(provider)
diff --git a/server/services/llm/protocol.py b/server/services/llm/protocol.py
index 8f2880ec..4be87292 100644
--- a/server/services/llm/protocol.py
+++ b/server/services/llm/protocol.py
@@ -364,6 +364,11 @@ class LLMError(Exception):
# Must already be safe for public surfaces: no credential, no query
# string, no userinfo.
public_message: Optional[str] = None
+ # Replaces the category's fixed recovery advice, under the same safety
+ # rule. The LLM gateway sets both for calls it served
+ # (services/gateway/errors.py): "check the provider's billing" would
+ # point at the wrong account.
+ public_hint: Optional[str] = None
def __post_init__(self) -> None:
Exception.__init__(self, self.message)
@@ -464,6 +469,8 @@ class LLMError(Exception):
@property
def hint(self) -> Optional[str]:
"""Fixed recovery advice; never copy the provider's raw response."""
+ if self.public_hint:
+ return self.public_hint
return {
LLMErrorCategory.BILLING: "Review the provider project's billing, spending cap, and credits. Resume after access is restored; repeating the request will not fix it.",
LLMErrorCategory.QUOTA: "Check the model's quota for your API project. Resume after the quota resets or is increased, or choose a model with available quota.",
@@ -832,6 +839,16 @@ def _error_body(exc: BaseException) -> Mapping[str, Any]:
return {}
+def error_type(exc: BaseException) -> Optional[str]:
+ """The ``type`` of an SDK error's JSON body (``budget_exceeded``), if any.
+
+ ``LLMError.provider_code`` prefers the body's ``code``, which LiteLLM sets
+ to the HTTP status, so the type has to be read from the body itself.
+ """
+ value = _error_body(exc).get("type")
+ return str(value) if value else None
+
+
def _google_error_details(exc: BaseException) -> List[Mapping[str, Any]]:
"""Read Google's typed details without exposing provider bodies publicly."""
body = getattr(exc, "details", None)
@@ -876,6 +893,11 @@ def _classify_error(
body = _error_body(exc)
message += " " + str(body.get("message") or "").lower()
codes = {str(value).lower() for value in (getattr(exc, "code", None), body.get("code"), body.get("type"), body.get("status")) if value is not None}
+ # The LLM gateway (LiteLLM) refuses a spent allowance with this type, as
+ # 429 before LiteLLM 1.104 and 402 or 422 after. Whatever the status, only
+ # the user can fix it.
+ if "budget_exceeded" in codes:
+ return LLMErrorCategory.BILLING
# A quota/credit exhaustion can arrive as 429, but backoff cannot fix it.
# Match narrow billing signatures before the generic HTTP classifiers.
if status in {None, 400, 402, 403, 429} and (
@@ -953,12 +975,18 @@ class LLMProvider(Protocol):
context_management: Optional[Dict[str, Any]] = None,
on_event: Optional[StreamSink] = None,
effort: Optional[str] = None,
+ extra_headers: Optional[Mapping[str, str]] = None,
+ extra_body: Optional[Mapping[str, Any]] = None,
) -> LLMResponse:
"""One turn. A provider that streams (``streaming`` in
llm_defaults.json) hands each delta to ``on_event`` as it arrives and
still returns the same response; the others never receive one.
``effort`` (``low`` / ``high``) is the owner's chat choice, sent only
- for a model its provider lists in ``effort_models``."""
+ for a model its provider lists in ``effort_models``.
+
+ ``extra_headers`` and ``extra_body`` ride on this request only: the
+ LLM gateway's attribution headers and replay-cache body. ChatUnifier
+ passes them only for calls that go through the gateway."""
...
async def fetch_models(self, api_key: str) -> List[str]: ...
diff --git a/server/services/llm/providers/anthropic.py b/server/services/llm/providers/anthropic.py
index a9c222d0..ce63630c 100644
--- a/server/services/llm/providers/anthropic.py
+++ b/server/services/llm/providers/anthropic.py
@@ -2,7 +2,7 @@
from __future__ import annotations
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Mapping, Optional
from core.logging import get_logger
from services.llm.protocol import (
@@ -58,6 +58,8 @@ class AnthropicProvider:
context_management: Optional[Dict[str, Any]] = None,
on_event: Optional[StreamSink] = None,
effort: Optional[str] = None,
+ extra_headers: Optional[Mapping[str, str]] = None,
+ extra_body: Optional[Mapping[str, Any]] = None,
) -> LLMResponse:
system, api_msgs = self._split_system(messages)
policy = self._model_policy(model)
@@ -104,6 +106,13 @@ class AnthropicProvider:
if tools:
params["tools"] = [self._to_api_tool(t) for t in tools]
+ # Per-request extras: the LLM gateway's attribution headers and
+ # replay-cache body (ChatUnifier sends them for gateway calls only).
+ if extra_headers:
+ params["extra_headers"] = dict(extra_headers)
+ if extra_body:
+ params["extra_body"] = dict(extra_body)
+
# Always stream. The SDK refuses a non-streaming request whose
# ``max_tokens`` implies more than ten minutes of generation
# (``_calculate_nonstreaming_timeout``), and ``resolve_max_tokens``
diff --git a/server/services/llm/providers/gemini.py b/server/services/llm/providers/gemini.py
index 5f09ca00..94cbd9ee 100644
--- a/server/services/llm/providers/gemini.py
+++ b/server/services/llm/providers/gemini.py
@@ -5,7 +5,7 @@ from __future__ import annotations
import base64
import hashlib
import json
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Mapping, Optional
from core.logging import get_logger
from services.llm.protocol import (
@@ -80,8 +80,13 @@ class GeminiProvider:
tools: Optional[List[ToolDef]] = None,
context_management: Optional[Dict[str, Any]] = None,
effort: Optional[str] = None,
+ extra_headers: Optional[Mapping[str, str]] = None,
+ extra_body: Optional[Mapping[str, Any]] = None,
) -> LLMResponse:
del context_management # Gemini stays on portable Context checkpoints.
+ # The LLM gateway's replay cache is verified only on its OpenAI and
+ # Anthropic routes, not the native Gemini one; never sent here.
+ del extra_body
from google.genai import types
system_instruction, contents = self._split_system_and_contents(messages)
@@ -118,6 +123,11 @@ class GeminiProvider:
if tools:
config["tools"] = [self._to_api_tool(t) for t in tools]
+ if extra_headers:
+ # Request-level options patch the client's: headers merge, the
+ # base URL stays.
+ config["http_options"] = types.HttpOptions(headers=dict(extra_headers))
+
resp = await self._client.aio.models.generate_content(
model=model,
contents=contents,
diff --git a/server/services/llm/providers/openai.py b/server/services/llm/providers/openai.py
index f2516136..0b4bdd6a 100644
--- a/server/services/llm/providers/openai.py
+++ b/server/services/llm/providers/openai.py
@@ -7,7 +7,7 @@ Also used as base for OpenRouter (same API shape, different base_url).
from __future__ import annotations
import json
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Mapping, Optional
from core.logging import get_logger
from services.llm.protocol import (
@@ -79,6 +79,10 @@ def _detail(source: Dict[str, Any]) -> str:
class OpenAIProvider:
provider_name = "openai"
+ #: Prepended to the model name on the wire only. The LLM gateway
+ #: (LiteLLM) routes OpenAI-format calls by provider prefix
+ #: (``groq/llama-...``); everything else here keeps the bare id.
+ _model_prefix: str = ""
def __init__(
self,
@@ -88,10 +92,12 @@ class OpenAIProvider:
base_url: Optional[str] = None,
provider_name: Optional[str] = None,
max_retries: int = 2,
+ model_prefix: str = "",
):
import openai
self.provider_name = provider_name or type(self).provider_name
+ self._model_prefix = model_prefix
kwargs: Dict[str, Any] = {
"api_key": api_key,
"max_retries": max(0, int(max_retries)),
@@ -110,6 +116,29 @@ class OpenAIProvider:
"""The base URL requests actually go to."""
return str(self._client.base_url)
+ def _wire_model(self, model: str) -> str:
+ """The model name a request carries (see ``_model_prefix``)."""
+ return f"{self._model_prefix}{model}" if self._model_prefix else model
+
+ def _bare_model(self, model: str) -> str:
+ """A model name the server echoed, without the wire prefix."""
+ prefix = self._model_prefix
+ return model[len(prefix):] if prefix and model.startswith(prefix) else model
+
+ @staticmethod
+ def _add_request_extras(
+ params: Dict[str, Any],
+ extra_headers: Optional[Mapping[str, str]],
+ extra_body: Optional[Mapping[str, Any]],
+ ) -> None:
+ """Per-request headers and body fields (the LLM gateway's
+ attribution and replay cache), merged with any ``extra_body``
+ the request already carries."""
+ if extra_headers:
+ params["extra_headers"] = dict(extra_headers)
+ if extra_body:
+ params.setdefault("extra_body", {}).update(extra_body)
+
def _raise_if_error_body(self, resp: Any) -> None:
"""Turn a 2xx that carried an error body into a ``PROTOCOL`` failure.
@@ -158,6 +187,8 @@ class OpenAIProvider:
context_management: Optional[Dict[str, Any]] = None,
on_event: Optional[StreamSink] = None,
effort: Optional[str] = None,
+ extra_headers: Optional[Mapping[str, str]] = None,
+ extra_body: Optional[Mapping[str, Any]] = None,
) -> LLMResponse:
model = self._clean_model(model)
policy = self._model_policy(model, thinking)
@@ -175,10 +206,12 @@ class OpenAIProvider:
context_management=context_management,
on_event=on_event,
effort=effort,
+ extra_headers=extra_headers,
+ extra_body=extra_body,
)
params: Dict[str, Any] = {
- "model": model,
+ "model": self._wire_model(model),
"messages": [self._to_api_message(m) for m in messages],
}
@@ -221,6 +254,7 @@ class OpenAIProvider:
# Tools
if tools:
params["tools"] = [self._to_api_tool(t) for t in tools]
+ self._add_request_extras(params, extra_headers, extra_body)
if on_event is not None:
return self._normalize(await self._stream_completion(params, on_event), model)
@@ -264,11 +298,13 @@ class OpenAIProvider:
context_management: Optional[Dict[str, Any]],
on_event: Optional[StreamSink] = None,
effort: Optional[str] = None,
+ extra_headers: Optional[Mapping[str, str]] = None,
+ extra_body: Optional[Mapping[str, Any]] = None,
) -> LLMResponse:
"""Run a self-contained Responses API turn for reasoning tool use."""
params: Dict[str, Any] = {
- "model": model,
+ "model": self._wire_model(model),
"input": self._to_responses_input(messages),
"max_output_tokens": max_tokens,
# Durable encrypted reasoning state is recorded in our message;
@@ -301,6 +337,7 @@ class OpenAIProvider:
"compact_threshold": threshold,
}
]
+ self._add_request_extras(params, extra_headers, extra_body)
if on_event is not None:
return self._normalize_responses(await self._stream_response(params, on_event), model)
response = await self._client.responses.create(**params)
@@ -754,7 +791,7 @@ class OpenAIProvider:
thinking=thinking_text,
tool_calls=tool_calls,
usage=usage,
- model=str(getattr(resp, "model", "") or model),
+ model=self._bare_model(str(getattr(resp, "model", "") or model)),
finish_reason=str(
getattr(resp, "status", "") or "stop"
).lower(),
diff --git a/server/services/llm/providers/openrouter.py b/server/services/llm/providers/openrouter.py
index 17d57017..ffeb3c58 100644
--- a/server/services/llm/providers/openrouter.py
+++ b/server/services/llm/providers/openrouter.py
@@ -2,7 +2,7 @@
from __future__ import annotations
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Mapping, Optional
from core.logging import get_logger
from services.llm.protocol import (
@@ -51,6 +51,8 @@ class OpenRouterProvider(OpenAIProvider):
tools: Optional[List[ToolDef]] = None,
context_management: Optional[Dict[str, Any]] = None,
effort: Optional[str] = None,
+ extra_headers: Optional[Mapping[str, str]] = None,
+ extra_body: Optional[Mapping[str, Any]] = None,
) -> LLMResponse:
# OpenAI-compatible endpoints must prove support independently; the
# OpenRouter adapter currently uses portable Context checkpoints.
@@ -66,6 +68,8 @@ class OpenRouterProvider(OpenAIProvider):
tools=tools,
context_management=None,
effort=effort,
+ extra_headers=extra_headers,
+ extra_body=extra_body,
)
async def fetch_models(self, api_key: str) -> List[str]:
diff --git a/server/services/llm/unifier.py b/server/services/llm/unifier.py
index 1096e708..57ca4d25 100644
--- a/server/services/llm/unifier.py
+++ b/server/services/llm/unifier.py
@@ -42,6 +42,7 @@ from services.plugin import NodeUserError
if TYPE_CHECKING:
from services.auth import AuthService
+ from services.gateway.routing import GatewayRoute, GatewayRouter
logger = get_logger(__name__)
@@ -55,6 +56,24 @@ def _provider_failure(provider: str, exc: BaseException) -> LLMError:
return error
+def _gateway_request_extras(route: "GatewayRoute") -> Dict[str, Any]:
+ """Per-request headers and body for a call the gateway serves."""
+ from services.gateway.attribution import replay_cache, request_headers
+
+ extras: Dict[str, Any] = {"extra_headers": request_headers()}
+ cache = replay_cache(route.api_key)
+ if cache is not None:
+ extras["extra_body"] = cache
+ return extras
+
+
+def _explain_gateway_failure(error: LLMError, exc: BaseException) -> None:
+ """Gateway wording for a failed gateway call (services/gateway/errors.py)."""
+ from services.gateway.errors import explain
+
+ explain(error, exc)
+
+
@dataclass(eq=False)
class _ClientEntry:
"""One cached SDK client plus its in-flight lease count."""
@@ -72,6 +91,11 @@ class ChatUnifier:
Public surface: ``chat()``, ``fetch_models()``, ``is_registered()``.
Everything else is private.
+
+ ``gateway`` (services/gateway/routing.py) sends a chat call that comes
+ with no key of the install's own to the OpenCompany Gateway when it serves
+ the provider. ``fetch_models`` never goes there: it validates a key
+ against its own vendor.
"""
def __init__(
@@ -80,9 +104,11 @@ class ChatUnifier:
auth_service: "AuthService",
*,
client_cache_size: int = 32,
+ gateway: Optional["GatewayRouter"] = None,
):
self._defaults = defaults
self._auth = auth_service
+ self._gateway = gateway
self._client_cache_size = max(0, int(client_cache_size))
self._client_cache: OrderedDict[str, _ClientEntry] = OrderedDict()
self._retired_entries: List[_ClientEntry] = []
@@ -129,15 +155,21 @@ class ChatUnifier:
``effort`` is the owner's chat choice (``low`` / ``high``). It reaches
the provider only for a model listed in its ``effort_models``; any
other model answers as it would without it.
+
+ An empty ``api_key`` means the install has no key of its own for the
+ provider. If the OpenCompany Gateway serves it and is connected, the
+ call goes there with the gateway key, attribution headers and, inside
+ a Temporal activity, a replay-cache namespace (services/gateway).
"""
provider_id = split_provider_ref(provider)[0]
spec = get_provider(provider_id)
streams = on_event is not None and self._streams(provider_id)
- api_key = self._resolve_credential(provider, api_key)
+ route = await self._gateway.route(provider) if not api_key and self._gateway is not None else None
+ api_key = route.api_key if route is not None else self._resolve_credential(provider, api_key)
entry: Optional[_ClientEntry] = None
try:
entry = await self._acquire_client(
- spec, api_key, provider_ref=provider, sdk_max_retries=sdk_max_retries
+ spec, api_key, provider_ref=provider, sdk_max_retries=sdk_max_retries, route=route
)
kwargs: Dict[str, Any] = {
"model": model,
@@ -151,6 +183,8 @@ class ChatUnifier:
kwargs["on_event"] = on_event
if effort and supports_effort(provider, model):
kwargs["effort"] = effort
+ if route is not None:
+ kwargs.update(_gateway_request_extras(route))
response = await entry.client.chat(messages, **kwargs)
if on_event is not None and not streams:
await _replay_response(response, on_event)
@@ -160,12 +194,16 @@ class ChatUnifier:
# carried an error body instead of a completion. Logged whether
# or not it is translated: agent steps take it untranslated, and
# only this layer knows where the call went (RFC-0003 D10).
+ if route is not None:
+ _explain_gateway_failure(error, error)
self._log_failure("LLM provider request failed", error, entry, provider)
if not translate_errors:
raise
raise error.as_node_error() from error
except spec.sdk_exception_types + (httpx.TransportError, httpx.HTTPStatusError) as e:
error = _provider_failure(provider, e)
+ if route is not None:
+ _explain_gateway_failure(error, e)
self._log_failure("LLM provider request failed", error, entry, provider)
if not translate_errors:
raise error from e
@@ -188,6 +226,8 @@ class ChatUnifier:
category=category,
retryable=category == LLMErrorCategory.CONNECTION,
)
+ if route is not None:
+ _explain_gateway_failure(error, e)
logger.warning(
"LLM client construction failed",
provider=error.provider,
@@ -322,6 +362,7 @@ class ChatUnifier:
*,
sdk_max_retries: int,
provider_ref: Optional[str] = None,
+ route: Optional["GatewayRoute"] = None,
) -> _ClientEntry:
"""Return an entry leased until ``_release_client`` is called."""
@@ -331,6 +372,7 @@ class ChatUnifier:
provider_ref=provider_ref,
sdk_max_retries=sdk_max_retries,
acquire=True,
+ route=route,
)
async def _get_or_create_entry(
@@ -341,22 +383,30 @@ class ChatUnifier:
sdk_max_retries: int,
acquire: bool,
provider_ref: Optional[str] = None,
+ route: Optional["GatewayRoute"] = None,
) -> _ClientEntry:
# The base-URL row is keyed by the full reference, so each named
# endpoint reads its own URL while sharing one registration.
ref = provider_ref or spec.name
- proxy_url = await self._auth.resolve_api_key(base_url_key(ref))
- unconfigured = unconfigured_endpoint_message(ref)
- if unconfigured and not proxy_url:
- # A named endpoint exists only as its credential rows. Without
- # the URL row the SDK would default to api.openai.com.
- raise NodeUserError(unconfigured)
+ if route is not None:
+ # A gateway call goes to the gateway, never to a saved base URL.
+ proxy_url: Optional[str] = route.base_url
+ else:
+ proxy_url = await self._auth.resolve_api_key(base_url_key(ref))
+ unconfigured = unconfigured_endpoint_message(ref)
+ if unconfigured and not proxy_url:
+ # A named endpoint exists only as its credential rows. Without
+ # the URL row the SDK would default to api.openai.com.
+ raise NodeUserError(unconfigured)
+ model_prefix = route.model_prefix if route is not None else ""
factory_kwargs = {
"api_key": api_key,
"proxy_url": proxy_url,
"max_retries": max(0, int(sdk_max_retries)),
**spec.client_kwargs,
}
+ if model_prefix:
+ factory_kwargs["model_prefix"] = model_prefix
if not self._client_cache_size:
return _ClientEntry(
client=spec.factory(**factory_kwargs),
@@ -369,6 +419,7 @@ class ChatUnifier:
api_key=api_key,
proxy_url=proxy_url,
sdk_max_retries=sdk_max_retries,
+ model_prefix=model_prefix,
)
evicted: Optional[_ClientEntry] = None
async with self._cache_lock:
@@ -412,6 +463,7 @@ class ChatUnifier:
api_key: str,
proxy_url: Optional[str],
sdk_max_retries: int,
+ model_prefix: str = "",
) -> str:
material = json.dumps(
{
@@ -419,6 +471,7 @@ class ChatUnifier:
"proxy_url": proxy_url or "",
"client_kwargs": spec.client_kwargs,
"max_retries": max(0, int(sdk_max_retries)),
+ "model_prefix": model_prefix,
},
sort_keys=True,
separators=(",", ":"),
diff --git a/server/services/plugin/base.py b/server/services/plugin/base.py
index 2fa01b32..763ece3f 100644
--- a/server/services/plugin/base.py
+++ b/server/services/plugin/base.py
@@ -582,16 +582,26 @@ class BaseNode:
the same identifiers as the log context.
"""
start_time = time.time()
- workflow_id_attr: Optional[str] = None
- if isinstance(context.raw, dict):
- workflow_id_attr = context.raw.get("workflow_id")
+ raw = context.raw if isinstance(context.raw, dict) else {}
+ workflow_id_attr: Optional[str] = raw.get("workflow_id")
+
+ from services.chat.stream import chat_run_id_of
log_fields: Dict[str, Any] = {
"node_id": node_id,
"node_type": self.type,
}
- if workflow_id_attr is not None:
- log_fields["workflow_id"] = workflow_id_attr
+ # Which run this work belongs to. Logs carry them, and so do the LLM
+ # gateway's attribution headers (services.gateway.attribution).
+ run_fields = {
+ "workflow_id": workflow_id_attr,
+ "execution_id": getattr(context, "execution_id", None),
+ "root_execution_id": raw.get("root_execution_id"),
+ "chat_run_id": chat_run_id_of(raw),
+ "employee_job_id": raw.get("employee_job_id"),
+ "user_id": getattr(context, "user_id", None),
+ }
+ log_fields.update({key: value for key, value in run_fields.items() if value})
async with log_context(**log_fields):
with tracer.start_as_current_span(
diff --git a/server/services/settings/handlers.py b/server/services/settings/handlers.py
index 34276fb7..8ae1ce62 100644
--- a/server/services/settings/handlers.py
+++ b/server/services/settings/handlers.py
@@ -158,13 +158,19 @@ async def handle_get_validated_ai_providers(
except Exception:
llm_defaults = {"providers": {}}
+ from services.gateway.routing import gateway_covers
+ from services.llm.config import curated_models
+
providers = []
for provider in AI_PROVIDERS:
- if not await auth_service.has_valid_key(provider, data.get("session_id", "default"), principal=principal):
+ if await auth_service.has_valid_key(provider, data.get("session_id", "default"), principal=principal):
+ stored_models = await auth_service.get_stored_models(provider, data.get("session_id", "default"), principal=principal)
+ elif await gateway_covers(provider):
+ # No key of the install's own; the OpenCompany Gateway serves it.
+ stored_models = curated_models(provider)
+ else:
continue
- stored_models = await auth_service.get_stored_models(provider, data.get("session_id", "default"), principal=principal)
-
provider_config = llm_defaults.get("providers", {}).get(provider, {})
default_model = provider_config.get("default_model", "")
popular_models = provider_config.get("popular_models") or [
diff --git a/server/services/temporal/_interceptors.py b/server/services/temporal/_interceptors.py
index f525b887..1ef2f0cb 100644
--- a/server/services/temporal/_interceptors.py
+++ b/server/services/temporal/_interceptors.py
@@ -34,13 +34,18 @@ from temporalio.worker import (
WorkflowInterceptorClassInput,
)
-from core.logging import get_logger
+from core.logging import get_logger, log_context
logger = get_logger(__name__)
class ActivityObservabilityInterceptor(ActivityInboundInterceptor):
- """Log start / retry / end (with outcome) for every activity."""
+ """Log start / retry / end (with outcome) for every activity.
+
+ Also binds the Temporal identifiers to the log context for the whole
+ activity, under their own keys (``workflow_id`` elsewhere is the graph
+ id). The LLM gateway's attribution reads them (services.gateway).
+ """
async def execute_activity(self, input: ExecuteActivityInput) -> Any:
info = activity.info()
@@ -57,8 +62,15 @@ class ActivityObservabilityInterceptor(ActivityInboundInterceptor):
logger.warning("activity_retry", **extra)
else:
logger.debug("activity_start", **extra)
+ bound = {
+ "temporal_workflow_id": info.workflow_id,
+ "temporal_run_id": info.workflow_run_id,
+ "temporal_activity_id": info.activity_id,
+ "temporal_attempt": info.attempt,
+ }
try:
- result = await self.next.execute_activity(input)
+ async with log_context(**bound):
+ result = await self.next.execute_activity(input)
except Exception as exc:
logger.warning(
"activity_end",
diff --git a/server/services/temporal/agent_activities.py b/server/services/temporal/agent_activities.py
index bd3550ac..327e6c16 100644
--- a/server/services/temporal/agent_activities.py
+++ b/server/services/temporal/agent_activities.py
@@ -221,6 +221,12 @@ async def _resolve_activity_api_key(payload: Dict[str, Any]) -> str:
if isinstance(candidate, str) and candidate and parameters.get("provider", "openai") == provider:
api_key = candidate
+ from services.gateway.routing import gateway_covers
+
+ if not api_key and await gateway_covers(provider):
+ # No own key: ChatUnifier sends the call through the OpenCompany
+ # Gateway with the install's gateway key (services/gateway).
+ return ""
if not api_key:
raise _unsaved_endpoint_error(provider) or ApplicationError(
f"API key for provider {provider!r} is not configured",
@@ -720,13 +726,18 @@ async def execute_llm_step(payload: Dict[str, Any]) -> Dict[str, Any]:
``heartbeat_timeout``.
"""
+ from core.logging import log_context
+
activity.logger.info(
f"Agent LLM step: provider={payload.get('provider')} " f"model={payload.get('model')} messages={len(payload.get('messages', []))}"
)
activity.heartbeat(f"LLM step starting: {payload.get('model')}")
await _publish_llm_attempt_status(payload)
- result = await _execute_native_llm_step(payload)
+ # The run's ids reach the log context before the provider call starts, so
+ # its logs and the LLM gateway's spend row carry them.
+ async with log_context(**(payload.get("attribution") or {})):
+ result = await _execute_native_llm_step(payload)
activity.heartbeat("LLM step: model returned")
return result
@@ -1230,6 +1241,11 @@ async def prepare_agent_payload(context: Dict[str, Any]) -> Dict[str, Any]:
api_key = bool(source) or await auth.has_valid_key(provider) or await auth.has_valid_key(f"{provider}_api_key")
else:
api_key = await auth.has_valid_key(provider) or await auth.has_valid_key(f"{provider}_api_key")
+ if not api_key:
+ from services.gateway.routing import gateway_covers
+
+ # No own key, but the OpenCompany Gateway serves the provider.
+ api_key = await gateway_covers(provider)
if not api_key:
endpoint_error = _unsaved_endpoint_error(provider)
if endpoint_error:
@@ -2570,17 +2586,27 @@ async def compact_context(payload: Dict[str, Any]) -> Dict[str, Any]:
f"({len(rendered)} rendered chars) via "
f"{payload['provider']}/{payload['model']}"
)
- result = await svc.compact_context(
- session_id=str(payload.get("session_id") or "default"),
- node_id=payload["node_id"],
- memory_content=rendered,
- provider=payload["provider"],
- api_key=await _resolve_activity_api_key(payload),
- model=payload["model"],
- # Temporal owns activity retries; never repeat an ambiguous provider
- # request inside the activity or SDK.
- explicit_max_retries=0,
- )
+ from core.logging import log_context
+ from services.llm.protocol import LLMError
+
+ try:
+ async with log_context(**(payload.get("attribution") or {})):
+ result = await svc.compact_context(
+ session_id=str(payload.get("session_id") or "default"),
+ node_id=payload["node_id"],
+ memory_content=rendered,
+ provider=payload["provider"],
+ api_key=await _resolve_activity_api_key(payload),
+ model=payload["model"],
+ # Temporal owns activity retries; never repeat an ambiguous provider
+ # request inside the activity or SDK.
+ explicit_max_retries=0,
+ # A spent allowance or a rejected key stops the run with its
+ # reason, like an LLM step, instead of retrying compaction.
+ stop_on_user_action=True,
+ )
+ except LLMError as error:
+ raise _as_temporal_llm_error(error) from None
if result.get("success") and result.get("summary"):
usage = result.get("usage") or {}
activity.logger.info(
diff --git a/server/services/temporal/agent_workflow.py b/server/services/temporal/agent_workflow.py
index fa22cfd1..f187948a 100644
--- a/server/services/temporal/agent_workflow.py
+++ b/server/services/temporal/agent_workflow.py
@@ -344,6 +344,30 @@ def _inherited_scope(context: Dict[str, Any], *, delegation: bool = False) -> Di
return inherited
+def _attribution_ids(
+ context: Dict[str, Any],
+ payload: Dict[str, Any],
+ *,
+ workflow_id: Optional[str],
+ execution_id: str,
+ root_execution_id: str,
+) -> Dict[str, str]:
+ """The ids an LLM step or compaction binds to its log context.
+
+ Logs carry them, and so does the LLM gateway's spend row
+ (services/gateway/attribution.py). Deterministic workflow state only.
+ """
+ ids = {
+ "workflow_id": workflow_id,
+ "execution_id": execution_id,
+ "root_execution_id": root_execution_id,
+ "chat_run_id": payload.get("chat_run_id"),
+ "employee_job_id": context.get("employee_job_id"),
+ "user_id": context.get("user_id"),
+ }
+ return {key: str(value) for key, value in ids.items() if value}
+
+
def _child_control_scope(control: ExecutionControl) -> Dict[str, Any]:
"""Fresh intent travels to children; each root owns its chain identity."""
return {
@@ -1135,6 +1159,16 @@ class AgentWorkflow:
if payload.get("effort")
else {}
),
+ # Which run the step belongs to, for logs and the LLM
+ # gateway's spend rows. Read only by the activity's log
+ # context; activity input only, like the keys above.
+ "attribution": _attribution_ids(
+ context,
+ payload,
+ workflow_id=agent_workflow_id,
+ execution_id=execution_id,
+ root_execution_id=root_execution_id,
+ ),
}
if context.get("native_workspace_version") == 1 or context.get("browser_runtime_version") == 1 or context.get("browser_routing_version") == 1:
llm_payload["user_id"] = str(context.get("user_id") or "owner")
@@ -2658,6 +2692,13 @@ class AgentWorkflow:
"messages": compact_source,
"provider": payload["provider"],
"model": payload["model"],
+ "attribution": _attribution_ids(
+ context,
+ payload,
+ workflow_id=agent_workflow_id,
+ execution_id=execution_id,
+ root_execution_id=root_execution_id,
+ ),
}
if context.get("native_workspace_version") == 1 or context.get("browser_runtime_version") == 1 or context.get("browser_routing_version") == 1:
compact_payload["user_id"] = str(context.get("user_id") or "owner")
diff --git a/server/tests/gateway/__init__.py b/server/tests/gateway/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/server/tests/gateway/test_attribution.py b/server/tests/gateway/test_attribution.py
new file mode 100644
index 00000000..df42cb5b
--- /dev/null
+++ b/server/tests/gateway/test_attribution.py
@@ -0,0 +1,102 @@
+"""services/gateway/attribution.py: what a gateway call says about itself,
+from the log context, allowlisted ids only."""
+
+import importlib.util
+import json
+from pathlib import Path
+
+import pytest
+import structlog
+
+from services.gateway.attribution import REPLAY_TTL_SECONDS, replay_cache, request_headers
+
+
+def test_headers_name_the_chat_run_and_carry_only_allowlisted_ids():
+ with structlog.contextvars.bound_contextvars(
+ chat_run_id="r_1",
+ root_execution_id="root",
+ workflow_id="wf",
+ user_id="owner",
+ node_id="agent-1",
+ prompt="a secret the user typed",
+ ):
+ headers = request_headers()
+ assert headers["x-litellm-session-id"] == "r_1"
+ assert headers["x-litellm-tags"] == "source:chat,workflow:wf,user:owner"
+ assert json.loads(headers["x-litellm-spend-logs-metadata"]) == {
+ "chat_run_id": "r_1",
+ "root_execution_id": "root",
+ "workflow_id": "wf",
+ "user_id": "owner",
+ "node_id": "agent-1",
+ }
+ assert "a secret the user typed" not in json.dumps(headers)
+
+
+ ("bound", "session"),
+ [
+ ({"root_execution_id": "root", "execution_id": "exec", "temporal_workflow_id": "t"}, "root"),
+ ({"execution_id": "exec", "temporal_workflow_id": "t"}, "exec"),
+ ({"temporal_workflow_id": "t"}, "t"),
+ ],
+)
+def test_session_falls_back_from_the_chat_run(bound, session):
+ with structlog.contextvars.bound_contextvars(**bound):
+ assert request_headers()["x-litellm-session-id"] == session
+
+
+def test_outside_any_run_only_the_source_is_sent():
+ assert request_headers() == {"x-litellm-tags": "source:direct"}
+
+
+def test_replay_cache_needs_a_temporal_activity():
+ with structlog.contextvars.bound_contextvars(chat_run_id="r_1"):
+ assert replay_cache("sk-gw") is None
+
+
+def _namespace(api_key="sk-gw", **ids):
+ with structlog.contextvars.bound_contextvars(**ids):
+ body = replay_cache(api_key)
+ assert body is not None
+ assert body["cache"]["use-cache"] is True
+ assert body["cache"]["ttl"] == REPLAY_TTL_SECONDS
+ return body["cache"]["namespace"]
+
+
+def test_replay_namespace_is_stable_across_attempts_and_new_per_run_and_key():
+ ids = {"temporal_workflow_id": "wf", "temporal_run_id": "run-1", "temporal_activity_id": "llm-step-1"}
+ first = _namespace(**ids, temporal_attempt=1)
+ assert first.startswith("oc1-")
+ assert _namespace(**ids, temporal_attempt=4) == first
+ assert _namespace(**{**ids, "temporal_run_id": "run-2"}) != first
+ assert _namespace(**{**ids, "temporal_activity_id": "llm-step-2"}) != first
+ assert _namespace("sk-other", **ids) != first
+
+
+def test_log_context_restores_the_outer_ids():
+ """A node executed inside another (a tool call) must not wipe its
+ parent's ids. tests/conftest.py stubs core.logging, so load the file."""
+ path = Path(__file__).resolve().parents[2] / "core" / "logging.py"
+ spec = importlib.util.spec_from_file_location("_core_logging_under_test", path)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+
+ with module.log_context_sync(chat_run_id="r_outer", node_id="agent"):
+ with module.log_context_sync(node_id="tool"):
+ assert structlog.contextvars.get_contextvars()["node_id"] == "tool"
+ assert structlog.contextvars.get_contextvars()["node_id"] == "agent"
+ assert structlog.contextvars.get_contextvars()["chat_run_id"] == "r_outer"
+ assert "node_id" not in structlog.contextvars.get_contextvars()
+
+
+async def test_async_log_context_restores_the_outer_ids():
+ path = Path(__file__).resolve().parents[2] / "core" / "logging.py"
+ spec = importlib.util.spec_from_file_location("_core_logging_under_test_async", path)
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+
+ async with module.log_context(execution_id="outer"):
+ async with module.log_context(execution_id="inner"):
+ assert structlog.contextvars.get_contextvars()["execution_id"] == "inner"
+ assert structlog.contextvars.get_contextvars()["execution_id"] == "outer"
diff --git a/server/tests/gateway/test_connector.py b/server/tests/gateway/test_connector.py
new file mode 100644
index 00000000..354810a7
--- /dev/null
+++ b/server/tests/gateway/test_connector.py
@@ -0,0 +1,111 @@
+"""The OpenCompany Gateway's Connectors card: the key probe, the allowance
+line, and the card listed only while the gateway is enabled."""
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock
+
+import httpx
+import pytest
+
+from services.gateway import credential as credential_module
+from services.gateway import handlers as handlers_module
+from services.gateway.allowance import parse_user_info
+
+USER_INFO = {
+ "user_id": "alice",
+ "user_info": {
+ "spend": 12.4,
+ "max_budget": 50,
+ "budget_reset_at": "2026-11-01T00:00:00Z",
+ "user_email": "[email protected]",
+ },
+}
+
+
+def _response(status, body=None):
+ return httpx.Response(status, json=body or {}, request=httpx.Request("GET", "https://gw.example/user/info"))
+
+
+def test_the_allowance_reads_as_one_line():
+ allowance = parse_user_info(USER_INFO)
+ assert allowance.summary == "$12.40 of $50.00 used · resets Nov 1"
+ assert allowance.account == "[email protected]"
+ assert parse_user_info({"user_info": {"spend": 3}}).summary == "$3.00 used"
+
+
+async def test_the_probe_reads_the_allowance(monkeypatch):
+ monkeypatch.setattr(credential_module, "user_info", AsyncMock(return_value=_response(200, USER_INFO)))
+ result = await credential_module.OpenCompanyGatewayCredential._probe("sk-gateway")
+ assert result.valid is True
+ assert result.message == "Connected: $12.40 of $50.00 used · resets Nov 1."
+
+
+async def test_the_probe_asks_for_a_default_key(monkeypatch):
+ monkeypatch.setattr(credential_module, "user_info", AsyncMock(return_value=_response(403)))
+ result = await credential_module.OpenCompanyGatewayCredential._probe("sk-llm-api")
+ assert result.valid is False
+ assert "type Default" in result.message
+
+
+async def test_a_rejected_key_is_left_to_validate_to_word(monkeypatch):
+ monkeypatch.setattr(credential_module, "user_info", AsyncMock(return_value=_response(401)))
+ with pytest.raises(httpx.HTTPStatusError):
+ await credential_module.OpenCompanyGatewayCredential._probe("sk-wrong")
+
+
+async def test_status_without_a_stored_key(monkeypatch):
+ from core.container import container
+
+ auth = MagicMock()
+ auth.get_api_key = AsyncMock(return_value=None)
+ monkeypatch.setattr(container, "auth_service", MagicMock(return_value=auth))
+ result = await handlers_module.handle_gateway_status({}, MagicMock())
+ assert result == {"success": True, "connected": False}
+
+
+async def test_status_shows_the_allowance(monkeypatch):
+ from core.container import container
+
+ auth = MagicMock()
+ auth.get_api_key = AsyncMock(return_value="sk-gateway")
+ monkeypatch.setattr(container, "auth_service", MagicMock(return_value=auth))
+ monkeypatch.setattr(handlers_module, "user_info", AsyncMock(return_value=_response(200, USER_INFO)))
+ result = await handlers_module.handle_gateway_status({}, MagicMock())
+ assert result["connected"] is True
+ assert result["summary"] == "$12.40 of $50.00 used · resets Nov 1"
+ assert result["account_label"] == "[email protected]"
+
+
+async def test_status_survives_an_unreachable_gateway(monkeypatch):
+ from core.container import container
+
+ auth = MagicMock()
+ auth.get_api_key = AsyncMock(return_value="sk-gateway")
+ monkeypatch.setattr(container, "auth_service", MagicMock(return_value=auth))
+ monkeypatch.setattr(handlers_module, "user_info", AsyncMock(side_effect=httpx.ConnectError("down")))
+ result = await handlers_module.handle_gateway_status({}, MagicMock())
+ assert result["connected"] is True
+ assert "Couldn't read the allowance" in result["summary"]
+
+
+def test_the_card_is_listed_only_while_the_gateway_is_enabled(monkeypatch):
+ from core.container import container
+ from services.credential_registry import CredentialRegistry
+
+ monkeypatch.setattr(container, "settings", MagicMock(return_value=SimpleNamespace(gateway_enabled=True)))
+ listed = {provider["id"]: provider for provider in CredentialRegistry().get_all_providers()}
+ assert listed["opencompany_gateway"]["kind"] == "apiKey"
+ assert listed["opencompany_gateway"]["consumer_category"] == "ai"
+ assert listed["opencompany_gateway"]["ws"] == {"status": "gateway_status"}
+
+ monkeypatch.setattr(container, "settings", MagicMock(return_value=SimpleNamespace(gateway_enabled=False)))
+ assert "opencompany_gateway" not in {provider["id"] for provider in CredentialRegistry().get_all_providers()}
+ # Still resolvable by id, for the connection state of a stored key.
+ assert CredentialRegistry().get_provider("opencompany_gateway") is not None
+
+
+def test_the_card_is_no_llm_provider():
+ """No llm_defaults.json block, so it never shows up in model pickers."""
+ from services.llm.config import PROVIDER_CONFIGS
+
+ assert "opencompany_gateway" not in PROVIDER_CONFIGS
diff --git a/server/tests/gateway/test_errors.py b/server/tests/gateway/test_errors.py
new file mode 100644
index 00000000..9ae13cd1
--- /dev/null
+++ b/server/tests/gateway/test_errors.py
@@ -0,0 +1,75 @@
+"""A spent allowance is BILLING whatever the status, and gateway calls get
+gateway wording (services/llm/protocol.py, services/gateway/errors.py)."""
+
+import httpx
+import openai
+import pytest
+
+from services.gateway.errors import explain
+from services.llm.protocol import LLMError, LLMErrorCategory, error_type
+
+
+def _gateway_error(status: int, error_type_: str, message: str = "refused") -> openai.APIStatusError:
+ """What the OpenAI SDK raises for a LiteLLM error body."""
+ body = {"message": message, "type": error_type_, "param": None, "code": str(status)}
+ response = httpx.Response(
+ status,
+ request=httpx.Request("POST", "https://gw.example/v1/chat/completions"),
+ json={"error": body},
+ )
+ return openai.APIStatusError(message, response=response, body=body)
+
+
[email protected]("status", [402, 422, 429])
+def test_a_spent_allowance_is_billing_whatever_the_status(status):
+ """LiteLLM answers 429 before 1.104, 402 with our setting, 422 by default after."""
+ error = LLMError.from_exception("openai", _gateway_error(status, "budget_exceeded", "Budget has been exceeded!"))
+ assert error.category is LLMErrorCategory.BILLING
+ assert error.retryable is False
+ assert error.requires_user_action is True
+
+
+def test_the_body_type_is_readable_although_provider_code_is_the_status():
+ exc = _gateway_error(403, "model_cost_map_missing")
+ assert LLMError.from_exception("openai", exc).provider_code == "403"
+ assert error_type(exc) == "model_cost_map_missing"
+
+
+def test_a_spent_allowance_is_worded_as_the_opencompany_allowance():
+ exc = _gateway_error(429, "budget_exceeded")
+ error = explain(LLMError.from_exception("openai", exc), exc)
+ assert error.user_message == "Your OpenCompany allowance for this month is used up."
+ assert "raise it" in error.hint
+ node_error = error.as_node_error()
+ assert node_error.requires_user_action is True
+
+
+def test_an_unpriced_model_says_so():
+ exc = _gateway_error(403, "model_cost_map_missing")
+ error = explain(LLMError.from_exception("openai", exc), exc)
+ assert error.category is LLMErrorCategory.PERMISSION
+ assert error.user_message == "This model isn't available on the OpenCompany Gateway yet."
+ assert error.hint.startswith("Choose another model")
+
+
+def test_a_rejected_gateway_key_points_at_the_gateway_card():
+ exc = _gateway_error(401, "auth_error")
+ error = explain(LLMError.from_exception("openai", exc), exc)
+ assert error.category is LLMErrorCategory.AUTHENTICATION
+ assert "OpenCompany Gateway" in error.user_message
+ assert "Reconnect the OpenCompany Gateway" in error.hint
+
+
+def test_wording_never_changes_retry_behaviour():
+ exc = _gateway_error(503, "service_unavailable")
+ error = LLMError.from_exception("openai", exc)
+ before = (error.category, error.retryable)
+ explain(error, exc)
+ assert (error.category, error.retryable) == before
+ assert error.user_message == "The OpenCompany Gateway is temporarily unavailable."
+
+
+def test_public_hint_replaces_the_category_hint():
+ error = LLMError(message="x", provider="openai", category=LLMErrorCategory.BILLING, public_hint="custom")
+ assert error.hint == "custom"
+ assert LLMError(message="x", provider="openai", category=LLMErrorCategory.BILLING).hint.startswith("Review the provider")
diff --git a/server/tests/gateway/test_routing.py b/server/tests/gateway/test_routing.py
new file mode 100644
index 00000000..c27115e7
--- /dev/null
+++ b/server/tests/gateway/test_routing.py
@@ -0,0 +1,115 @@
+"""services/gateway/routing.py: own keys first; the gateway serves only the
+providers llm_defaults.json marks, and only when enabled and connected."""
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+
+from services.gateway.routing import GATEWAY_CREDENTIAL_ID, GatewayRoute, GatewayRouter, gateway_covers
+
+DEFAULTS = {
+ "providers": {
+ "openai": {"gateway": {"model_prefix": "openai/", "base_path": "/v1"}},
+ "anthropic": {"gateway": {"model_prefix": "", "base_path": ""}},
+ "deepseek": {"default_model": "deepseek-v4.1-flash"},
+ }
+}
+
+
+def _router(*, enabled=True, distributed=False, key="sk-gateway", defaults=DEFAULTS):
+ settings = SimpleNamespace(gateway_enabled=enabled, distributed_mode=distributed, gateway_url="https://gw.example/")
+ auth = MagicMock()
+ auth.get_api_key = AsyncMock(return_value=key)
+ return GatewayRouter(settings=settings, auth_service=auth, defaults=defaults), auth
+
+
+async def test_a_served_provider_gets_the_gateway_route():
+ router, auth = _router()
+ assert await router.route("openai") == GatewayRoute(
+ api_key="sk-gateway", base_url="https://gw.example/v1", model_prefix="openai/"
+ )
+ auth.get_api_key.assert_awaited_with(GATEWAY_CREDENTIAL_ID)
+
+
+async def test_native_routes_use_the_gateway_root_and_bare_ids():
+ router, _ = _router()
+ route = await router.route("anthropic")
+ assert route is not None
+ assert (route.base_url, route.model_prefix) == ("https://gw.example", "")
+
+
[email protected]("provider", ["deepseek", "openai_compatible:lab", "unknown"])
+async def test_a_provider_without_a_gateway_block_is_never_routed(provider):
+ router, _ = _router()
+ assert await router.route(provider) is None
+
+
+async def test_no_route_without_a_stored_gateway_key():
+ router, _ = _router(key=None)
+ assert await router.route("openai") is None
+ assert await router.covers("openai") is False
+
+
[email protected]("flag", [False, MagicMock(), "true", 1])
+async def test_only_a_true_flag_turns_the_gateway_on(flag):
+ router, auth = _router(enabled=flag)
+ assert await router.route("openai") is None
+ auth.get_api_key.assert_not_awaited()
+
+
+async def test_distributed_mode_turns_the_gateway_off():
+ router, _ = _router(distributed=True)
+ assert await router.route("openai") is None
+
+
+def test_served_providers_come_from_the_gateway_blocks():
+ router, _ = _router()
+ assert router.served_providers() == ["openai", "anthropic"]
+
+
+def test_llm_defaults_serves_exactly_the_priced_providers():
+ """OpenRouter, DeepSeek and Sarvam stay off until their default models
+ are priced upstream (gateway/tests/test_pins.py checks the prices)."""
+ from services.llm.config import LLM_DEFAULTS
+
+ router, _ = _router(defaults=LLM_DEFAULTS)
+ assert set(router.served_providers()) == {
+ "openai", "anthropic", "gemini", "groq", "cerebras", "xai", "kimi", "mistral",
+ }
+
+
+async def test_gateway_covers_trusts_only_a_real_router():
+ # tests/conftest.py stubs core.container with a MagicMock, whose router
+ # would claim to cover every provider.
+ assert await gateway_covers("openai") is False
+
+
+async def test_activity_key_lookup_returns_no_key_when_the_gateway_covers(monkeypatch):
+ """The gateway key never reaches activity code or a Temporal payload:
+ the activity passes no key and ChatUnifier routes the call."""
+ from core.container import container
+ from services.temporal import agent_activities
+
+ auth = MagicMock(distributed_credentials=False)
+ auth.resolve_api_key = AsyncMock(return_value=None)
+ monkeypatch.setattr(container, "auth_service", MagicMock(return_value=auth))
+ monkeypatch.setattr("services.gateway.routing.gateway_covers", AsyncMock(return_value=True))
+
+ assert await agent_activities._resolve_activity_api_key({"provider": "openai", "user_id": "owner"}) == ""
+
+
+async def test_activity_key_lookup_still_refuses_when_nothing_covers(monkeypatch):
+ from temporalio.exceptions import ApplicationError
+
+ from core.container import container
+ from services.temporal import agent_activities
+
+ auth = MagicMock(distributed_credentials=False)
+ auth.resolve_api_key = AsyncMock(return_value=None)
+ monkeypatch.setattr(container, "auth_service", MagicMock(return_value=auth))
+ monkeypatch.setattr("services.gateway.routing.gateway_covers", AsyncMock(return_value=False))
+
+ with pytest.raises(ApplicationError) as raised:
+ await agent_activities._resolve_activity_api_key({"provider": "openai", "user_id": "owner"})
+ assert raised.value.type == "MissingAgentProviderCredential"
diff --git a/server/tests/gateway/test_unifier_route.py b/server/tests/gateway/test_unifier_route.py
new file mode 100644
index 00000000..e291eaac
--- /dev/null
+++ b/server/tests/gateway/test_unifier_route.py
@@ -0,0 +1,145 @@
+"""ChatUnifier sends a call with no key of the install's own to the gateway
+route, and never asks the gateway when there is one."""
+
+from unittest.mock import AsyncMock, MagicMock
+
+import httpx
+import openai
+import pytest
+import structlog
+
+from services.gateway.routing import GatewayRoute
+from services.llm import registry
+from services.llm.protocol import LLMError, LLMErrorCategory, LLMResponse, Message
+from services.llm.registry import ProviderSpec
+from services.llm.unifier import ChatUnifier
+from services.plugin import NodeUserError
+
+ROUTE = GatewayRoute(api_key="sk-gateway", base_url="https://gw.example/v1", model_prefix="groq/")
+HI = [Message(role="user", content="hi")]
+
+
+class _Router:
+ def __init__(self, route):
+ self._route = route
+ self.asked = []
+
+ async def route(self, provider):
+ self.asked.append(provider)
+ return self._route
+
+
+def _provider(monkeypatch, name, *, chat=None):
+ """A fake provider in the registry for this test only (monkeypatch removes it)."""
+ created = []
+
+ class Client:
+ def __init__(self, **kwargs):
+ self.kwargs = kwargs
+ self.chat = chat or AsyncMock(return_value=LLMResponse(content="ok"))
+ created.append(self)
+
+ monkeypatch.setitem(
+ registry._REGISTRY, name, ProviderSpec(name=name, factory=Client, sdk_exception_refs=("openai:OpenAIError",))
+ )
+ return created
+
+
+def _unifier(router):
+ auth = MagicMock()
+ auth.resolve_api_key = AsyncMock(return_value=None)
+ return ChatUnifier(defaults={"providers": {}}, auth_service=auth, gateway=router), auth
+
+
+async def test_a_call_with_no_own_key_goes_to_the_gateway(monkeypatch):
+ created = _provider(monkeypatch, "gateway-route-test")
+ router = _Router(ROUTE)
+ unifier, auth = _unifier(router)
+
+ with structlog.contextvars.bound_contextvars(chat_run_id="r_1", workflow_id="wf"):
+ await unifier.chat(provider="gateway-route-test", api_key="", messages=HI, model="llama-3.3-70b")
+
+ client = created[0]
+ assert client.kwargs["api_key"] == "sk-gateway"
+ assert client.kwargs["proxy_url"] == "https://gw.example/v1"
+ assert client.kwargs["model_prefix"] == "groq/"
+ sent = client.chat.call_args.kwargs
+ # The provider adds the prefix on the wire; capability lookups keep bare ids.
+ assert sent["model"] == "llama-3.3-70b"
+ assert sent["extra_headers"]["x-litellm-session-id"] == "r_1"
+ # Outside a Temporal activity there is nothing to replay.
+ assert "extra_body" not in sent
+ # A gateway call never reads a saved {provider}_proxy base URL.
+ auth.resolve_api_key.assert_not_awaited()
+
+
+async def test_an_own_key_is_used_as_before_and_the_gateway_is_not_asked(monkeypatch):
+ created = _provider(monkeypatch, "gateway-own-key-test")
+ router = _Router(ROUTE)
+ unifier, _ = _unifier(router)
+
+ await unifier.chat(provider="gateway-own-key-test", api_key="sk-own", messages=HI, model="m")
+
+ assert router.asked == []
+ assert created[0].kwargs["api_key"] == "sk-own"
+ assert "model_prefix" not in created[0].kwargs
+ assert "extra_headers" not in created[0].chat.call_args.kwargs
+
+
+async def test_inside_an_activity_the_call_opts_into_replay(monkeypatch):
+ created = _provider(monkeypatch, "gateway-replay-test")
+ unifier, _ = _unifier(_Router(ROUTE))
+
+ with structlog.contextvars.bound_contextvars(
+ temporal_workflow_id="wf", temporal_run_id="run", temporal_activity_id="llm-step-1"
+ ):
+ await unifier.chat(provider="gateway-replay-test", api_key="", messages=HI, model="m")
+
+ cache = created[0].chat.call_args.kwargs["extra_body"]["cache"]
+ assert cache["use-cache"] is True and cache["namespace"].startswith("oc1-")
+
+
+async def test_no_own_key_and_no_gateway_still_refuses(monkeypatch):
+ _provider(monkeypatch, "gateway-none-test")
+ unifier, _ = _unifier(_Router(None))
+ with pytest.raises(NodeUserError, match="No API key is configured"):
+ await unifier.chat(provider="gateway-none-test", api_key="", messages=HI, model="m")
+
+
+async def test_a_gateway_failure_gets_gateway_wording(monkeypatch):
+ body = {"message": "Budget has been exceeded!", "type": "budget_exceeded", "param": None, "code": "429"}
+ response = httpx.Response(429, request=httpx.Request("POST", "https://gw.example/v1/chat/completions"), json={"error": body})
+ refused = openai.RateLimitError("Budget has been exceeded!", response=response, body=body)
+ _provider(monkeypatch, "gateway-failure-test", chat=AsyncMock(side_effect=refused))
+ unifier, _ = _unifier(_Router(ROUTE))
+
+ with pytest.raises(LLMError) as raised:
+ await unifier.chat(
+ provider="gateway-failure-test", api_key="", messages=HI, model="m", translate_errors=False
+ )
+ assert raised.value.category is LLMErrorCategory.BILLING
+ assert raised.value.retryable is False
+ assert raised.value.user_message == "Your OpenCompany allowance for this month is used up."
+
+
+async def test_openai_provider_prefixes_the_wire_model_only():
+ from services.llm.providers.openai import OpenAIProvider
+
+ provider = OpenAIProvider("sk-gateway", proxy_url="https://gw.example/v1", model_prefix="openai/")
+ resp = MagicMock()
+ resp.choices = [MagicMock(message=MagicMock(content="ok", reasoning_content=None, tool_calls=None), finish_reason="stop")]
+ resp.usage = MagicMock(prompt_tokens=1, completion_tokens=1, total_tokens=2, completion_tokens_details=None)
+ provider._client.chat.completions.create = AsyncMock(return_value=resp)
+
+ result = await provider.chat(
+ HI,
+ model="gpt-4.1-mini",
+ extra_headers={"x-litellm-session-id": "s"},
+ extra_body={"cache": {"use-cache": True}},
+ )
+
+ sent = provider._client.chat.completions.create.call_args.kwargs
+ assert sent["model"] == "openai/gpt-4.1-mini"
+ assert sent["extra_headers"] == {"x-litellm-session-id": "s"}
+ assert sent["extra_body"] == {"cache": {"use-cache": True}}
+ assert result.model == "gpt-4.1-mini"