Skip to content

Self-improving AI Employees for everyone

feat(gateway): run embeddings, speech and vision on the OpenCompany Gateway

trohitgc516c67f6259
diff --git a/.github/workflows/gateway-ci.yml b/.github/workflows/gateway-ci.yml
index 47a18b4b..8b787d55 100644
--- a/.github/workflows/gateway-ci.yml
+++ b/.github/workflows/gateway-ci.yml
@@ -8,12 +8,14 @@ on:
paths:
- "gateway/**"
- "server/config/llm_defaults.json"
+ - "server/config/speech_defaults.json"
- ".github/workflows/gateway-ci.yml"
push:
branches: [main]
paths:
- "gateway/**"
- "server/config/llm_defaults.json"
+ - "server/config/speech_defaults.json"
- ".github/workflows/gateway-ci.yml"
permissions:
diff --git a/CLAUDE.md b/CLAUDE.md
index 70e2ed9e..8582b354 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -51,7 +51,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[Translate / Transliterate / Detect](./docs-internal/speech_provider_rfc.md#8-the-pattern-generalised-nodestranslate)** | The second application of the multi-vendor pattern, and the one that retired `nodes/sarvam/`. Three nodes (`translateText`, `transliterateText`, `detectLanguage`) in [`server/nodes/translate/`](./server/nodes/translate/), backed by **three registries — one per capability**, because the asymmetry is sharper than speech's: DeepL translates and does nothing else, while Sarvam and LLM-backed providers do all three. Membership makes an unsupported selection unrepresentable rather than a runtime failure inside a paid call. Providers: **deepl** (`Authorization: DeepL-Auth-Key`, `text` is an array, free keys ending `:fx` route to a different host, and it returns `billed_characters` so cost is the provider's own figure), **sarvam** (all three over dedicated REST), **openai** (all three via `ChatUnifier` — proves the Protocol does not care what a provider does internally; billing deliberately NOT recorded because that path bills tokens the LLM layer already costs). Capabilities in `server/config/translate_defaults.json`. |
| **[Desktop App](./docs-internal/desktop_app.md)** | Electron shell in [`desktop/`](./desktop/) — a standalone bun package (NOT a root workspace member; `cli/tests/test_release_pipeline_config.py` locks `workspaces`). Bundles pinned `uv`, python-build-standalone CPython 3.12 and bun 1.4.0 — the only JavaScript runtime shipped; no Node, no npm (`desktop/runtimes.json`, verified against upstream checksum manifests by `scripts/fetch-runtimes.ts`, which also fetches bun's `LICENSE.md` from the bun repo — the release zip carries none — and prunes stale runtime dirs such as a leftover `node/`), stages the backend sibling layout from `bun pm pack --dry-run` (`scripts/stage.ts`, which runs the sidecar package's own `bun run build` — `bun build --target=bun`, express inlined — and copies `dist/index.js`), sets `OPENCOMPANY_BUN_BIN` / `BUN_INSTALL=<userData>/bun` / `BUN_INSTALL_CACHE_DIR` and prepends the bundled bun dir (then uv) to PATH for the backend (`env.ts`; `OPENCOMPANY_NODE_BIN` no longer exists), provisions the venv into Electron `userData` on first launch with `uv sync --frozen --python <bundled>` (stamp = sha256 of uv.lock + pyproject + Python + uv versions; falls back to `uv python install 3.12`), pre-warms `PYTHONPYCACHEPREFIX` (with a prefix set Python ignores in-tree .pyc — 32 s vs 3 s first boot), spawns uvicorn on `127.0.0.1:5678` (persisted port; attaches to a running `company serve` instead of double-spawning), waits on `/health/ready`, loads the backend-served SPA (zero frontend changes), routes `window.open` / `_blank` to the OS browser. Quit = `POST /api/desktop/shutdown` + stdin EOF, 30 s, then tree-kill. electron-builder: NSIS x64, DMG+zip arm64/x64, AppImage+deb x64, unsigned first (`mac.identity: null`; macOS auto-update notify-only until signed — Squirrel.Mac refuses unsigned). CI: `desktop-release.yml` (tag-triggered: `prepare` drafts the GitHub Release from the annotated tag's message, three legs build and upload version-less OS-named installers such as `OpenCompany-windows-x64.exe` / `OpenCompany-macos-arm64.dmg` / `OpenCompany-linux-x86_64.AppImage` so the README can link `releases/latest/download/<file>`, each leg refuses a desktop version that differs from the tag, `finalize` undrafts) and `desktop-ci.yml` (typecheck/unit/invariants/Playwright smoke). **Gotcha: editor-hosted terminals export `ELECTRON_RUN_AS_NODE=1`, which makes Electron boot as plain Node (`electron.app` undefined) — strip it before launching; the e2e test does.** |
| **[Desktop Host Contract](./docs-internal/desktop_host_contract.md)** | The backend's side of being owned by a GUI shell. `core/approot.py` is the ONE module that locates the app tree (`OPENCOMPANY_APP_ROOT` / `OPENCOMPANY_CLIENT_DIST` / `OPENCOMPANY_ENV_FILE` / `OPENCOMPANY_ENV_TEMPLATE`); `tests/test_no_root_climb.py` forbids any other `Path(__file__).parents[N]` climb above `server/` and any `"client"/"dist"` / `.env.template` / `package.json` path join. `core/env_defaults.apply_file_defaults_to_environ()` does the CLI's `.env.template < .env < process env` layering at the top of `main.py`, and `Settings.env_file` is the absolute `(template, .env)` tuple — so a shell spawns `python -m uvicorn` from any cwd without the CLI package. Desktop mode (`OPENCOMPANY_DESKTOP=1`, `core/desktop.py`): parent-PID watchdog with create-time PID-reuse guard, stdin-EOF watchdog (polling via `PeekNamedPipe`/`select`, never a blocking read — a daemon thread parked in `ReadFile` at finalisation crashed exit with 0xC0000005), ctypes Job Object so children die with the backend (no pywin32 in the server venv), token-gated `POST /api/desktop/shutdown` (`routers/desktop.py`, public path; raises the signal uvicorn handles so the lifespan reaps Temporal/the bun sidecar/edgymeow), 45 s hard deadline. `GET /health/ready` (public) = DB up AND (Temporal disabled OR worker manager started), with `phase` written by `services/temporal/lifecycle.py` (`installing_temporal` / `starting_temporal` / `connecting` / `starting_workers` / `ready`). Plugin shutdown hooks now run under a 10 s per-hook timeout (`HOOK_TIMEOUT_SECONDS`), and the current browser hook uses `peek_browser_runtime()` so shutdown never constructs or installs the runtime. Historical incident: the retired `get_browser_service()` lazily installed agent-browser at shutdown and wedged graceful exit (masked under the CLI by its 5 s grace + tree-kill). `server/uv.lock` is committed; `predeploy.yml` runs `uv lock --check`. |
-| **[LLM Gateway](./docs-internal/llm_gateway.md)** | Hosted LiteLLM Proxy in `gateway/` (official AWS module, mock-model local stack, contract and pin tests, `gateway-ci.yml`): pinned LiteLLM 1.103.4 and price list, served providers, and the contract OpenCompany relies on (wire model names, `budget_exceeded` refusals, `/user/info` allowance with Default keys, attribution headers, cache replay). Operator runbook: `gateway/README.md`. OpenCompany's side, `server/services/gateway/`: the Connectors card (`OPENCOMPANY_GATEWAY_ENABLED`), own-keys-first routing in ChatUnifier from `"gateway"` blocks in `llm_defaults.json`, attribution from the log context, gateway error wording. |
+| **[LLM Gateway](./docs-internal/llm_gateway.md)** | Hosted LiteLLM Proxy in `gateway/` (official AWS module, mock-model local stack, contract and pin tests, `gateway-ci.yml`): pinned LiteLLM 1.103.4 and price list, served providers, and the contract OpenCompany relies on (wire model names, `budget_exceeded` refusals, `/user/info` allowance with Default keys, attribution headers, cache replay). Operator runbook: `gateway/README.md`. OpenCompany's side, `server/services/gateway/`: the Connectors card (`OPENCOMPANY_GATEWAY_ENABLED`), own-keys-first routing in ChatUnifier from `"gateway"` blocks in `llm_defaults.json`, and for embeddings, OpenAI speech (`"gateway": true` in `speech_defaults.json`) and vision through `key_or_route`; attribution from the log context, gateway error wording. |
| **[CI/CD Pipeline](./docs-internal/ci_cd.md)** | GitHub Actions workflows, predeploy validation, release publishing, and the composite setup action (bun via SHA-pinned `oven-sh/setup-bun` reading the root `packageManager` pin + Node 22 as a CI-only convenience so bun runs vite / vitest / eslint on it via their node shebangs + Python 3.12 + uv; `bun install --frozen-lockfile` everywhere; publish is `bun publish --access public` after a `bun pm whoami` credential preflight with `~/.npmrc` written from `NPM_TOKEN` — no npm CLI, no `setup-node` step of their own in the publish jobs (they get Node through the composite action), and npm provenance attestation went away with npm). Also **Cutting a release**: the annotated tag is the version and its message is the release notes; `company version sync vX.Y.Z` bumps every version file for the `chore(release)` commit the tag sits on, and `desktop-release.yml` drafts the GitHub Release from the tag message and refuses a version that differs from the tag |
| **[Performance](./docs-internal/performance.md)** | Cold-start measurements (Application startup complete 2.90 s warm, 21.5 s cold post-`company clean` — was 71 s before the 2026-07-14 boot-delay fixes; first WS connect 8.29 s) + warm and cold per-phase timelines + optimisation history (lazy LangChain imports, lazy LLM-SDK exception refs, pre-bundled sidecar (esbuild at the time, `bun build --target=bun` today), bytecode-at-build, PartySocket reconnect, TanStack Query auth bootstrap, `manualChunks`, Vite dep-cache preservation, off-loop Temporal build-id hash) + bottleneck inventory + reproduction commands + anti-patterns to never reintroduce (incl. eager SDK import at LLM provider registration, unconditional `.vite` wipe, synchronous `Worker()` on the event loop). Pairs with the build-time pipeline doc below. |
| **[Release Build Pipeline](./docs-internal/release_build_pipeline.md)** | Build pipeline for the registry tarball (installed with `bun add -g`; bun is the only JavaScript runtime and package manager anything shipped needs): `typescript@7` (native Go compiler, exact-pinned in **root** devDependencies — client keeps `typescript@^5.x` because typescript-eslint's peer range excludes 6/7, and one manifest cannot declare `typescript` twice) for type-check, measured 5.9× faster than tsc 5.9.3 (820 ms vs 4830 ms); `bun run --filter react-flow-client typecheck` delegates up to the root gate, Vite `manualChunks` + `target: 'es2022'`, `bun build --target=bun` sidecar bundle (self-contained `dist/index.js` with express inlined, run as `bun dist/index.js`; drops the `tsx` interpreter cost and needs no `node_modules` at runtime), bytecode pre-compile in two halves — `[tool.uv] compile-bytecode = true` (uv sync compiles `.venv/` site-packages) + `python -m compileall` for project source (plain `.pyc`, **no `-O`** — per PEP 488 the non-`-O` runtimes never load `.opt-1.pyc`; excludes `.venv/`, `tests/`). Single source of truth for the bytecode-compile path list: `cli.commands.build.COMPILEALL_SOURCE_DIRS`. Dev PM is **[email protected]** (tracked text `bun.lock`, `bunfig.toml` pins the isolated linker, ranged security pins in top-level `overrides`); `company build` step [1/6] is `bun install`. |
diff --git a/docs-internal/llm_gateway.md b/docs-internal/llm_gateway.md
index 89fe572e..b9bf93c2 100644
--- a/docs-internal/llm_gateway.md
+++ b/docs-internal/llm_gateway.md
@@ -2,7 +2,7 @@
OpenCompany's hosted LLM gateway is the stock open-source LiteLLM Proxy, deployed from `gateway/` with BerriAI's official Terraform module. Invited people call LLMs on platform-owned vendor keys, each with a monthly allowance, and spend is recorded per person and per session. The repository holds only configuration and tests; there is no custom gateway code. The operator runbook (deployment, adding people, upgrades) is [gateway/README.md](../gateway/README.md).
-**Status:** the gateway is in `gateway/`; OpenCompany's side is `server/services/gateway/` and covers chat through ChatUnifier (agents, chat runs, employee setup, translate). Embeddings, speech, vision, RLM and Claude Code still call vendors directly. `gateway/tests/test_gateway_contract.py` pins the contract below against the local stack; `server/tests/gateway/` covers OpenCompany's side.
+**Status:** the gateway is in `gateway/`; OpenCompany's side is `server/services/gateway/` and covers chat through ChatUnifier (agents, chat runs, employee setup, translate), OpenAI embeddings (long-term memory, the Embedding Generator), OpenAI speech (Text to Speech, Speech to Text, chat dictation) and the vision tool. RLM and Claude Code still call vendors directly. `gateway/tests/test_gateway_contract.py` pins the contract below against the local stack; `server/tests/gateway/` covers OpenCompany's side.
## OpenCompany's side
@@ -12,7 +12,14 @@ OpenCompany's hosted LLM gateway is the stock open-source LiteLLM Proxy, deploye
**Routing: own keys first** (`services/gateway/routing.py`). A call whose provider has a key of the install's own goes to that vendor as always. Otherwise, if the gateway is enabled, its key is stored and the provider has a `"gateway"` block in `server/config/llm_defaults.json`, `ChatUnifier.chat` builds the client with the gateway key and base URL (instead of the `{ref}_proxy` row) and passes the block's `model_prefix` to `OpenAIProvider`, which applies it to the wire model only. The gateway key stays inside the router and ChatUnifier: callers pass no key (`_resolve_activity_api_key` returns `""`), and it never enters node parameters or Temporal payloads. `ChatUnifier.fetch_models` never routes (it validates a key against its own vendor); `AIService.fetch_models` lists a gateway-covered provider's curated models with no network call. The checks that used to refuse "no key configured" accept gateway coverage (`GatewayRouter.covers`, or `gateway_covers` outside the DI graph): the three in `services/ai.py`, `_resolve_activity_api_key` and `prepare_agent_payload`, `Connections.ai_providers` (employee setup and the chat model picker), the global model list, and translate.
-**Attribution** (`services/gateway/attribution.py`). Ids come from the structlog log context: `BaseNode.execute` binds the workflow, execution, root execution, chat run, employee job and user ids; the Temporal activity interceptor binds `temporal_workflow_id`, `temporal_run_id`, `temporal_activity_id` and `temporal_attempt`; the LLM-step and compaction activities bind the `attribution` dict their payload carries (`_attribution_ids` in `agent_workflow.py`). `log_context` restores outer values on exit, so a tool node run inside an agent keeps the agent's ids. Headers: session = chat run, else root execution, else execution, else Temporal workflow; tags `source:`, `workflow:`, `user:`; every allowlisted id as metadata. Inside a Temporal activity, calls send a replay-cache namespace from the key's fingerprint and the Temporal workflow, run and activity ids (not the attempt), so a retried activity replays at $0 and a re-run never replays another run's answer. Gemini's native route gets the headers but no cache body (unverified there).
+**Embeddings, speech and vision** call the gateway from their own clients, with the same rule and the provider's same `"gateway"` block: the gateway's key, its URL plus the block's `base_path`, the `model_prefix` on the wire model only, and the attribution headers built per call.
+- **The key.** `key_or_route(ctx, credential_id, provider)` gives a plugin node its key: its own through `ctx.connection`, else the gateway's with the route. With neither, the annotated missing-key error still reaches the Credentials modal.
+- **Embeddings** (`services/memory/vector_store.py`): `openai_embedding_access` tries the given key, then the stored OpenAI key, then the gateway. It routes only on OpenAI's own endpoint: an `endpoint` override names a server of the user's, which never gets the gateway key. `OpenAIEmbedder` never falls back to `OPENAI_API_KEY` from the environment.
+- **Speech** (`nodes/speech/`): only providers marked `"gateway": true` in `speech_defaults.json` are routed (`provider_key`, `gateway_serves`). That is OpenAI, both directions; Groq is not, because 1.103.4 bills its transcription at $0. The gateway has no `/audio/translations` route, so Translate falls back to in-language transcription there. Dictation takes the first provider with an own key, else the first the gateway serves.
+- **Vision** (`nodes/vision/vision_analyze`): routes all three SDK clients. The node now declares its OpenAI, Anthropic and Gemini credentials. It never did, so every real run failed at `ctx.connection`.
+- **Error wording.** Failures of these calls get gateway wording from `describe` (`services/gateway/errors.py`), which reads LiteLLM's error type from an SDK or `httpx` error.
+
+**Attribution** (`services/gateway/attribution.py`). Ids come from the structlog log context: `BaseNode.execute` binds the workflow, execution, root execution, chat run, employee job and user ids; the Temporal activity interceptor binds `temporal_workflow_id`, `temporal_run_id`, `temporal_activity_id` and `temporal_attempt`; the LLM-step and compaction activities bind the `attribution` dict their payload carries (`_attribution_ids` in `agent_workflow.py`). `log_context` restores outer values on exit, so a tool node run inside an agent keeps the agent's ids. Headers: session = chat run, else root execution, else execution, else Temporal workflow; tags `source:`, `workflow:`, `user:`; every allowlisted id as metadata. Inside a Temporal activity, calls send a replay-cache namespace from the key's fingerprint and the Temporal workflow, run and activity ids (not the attempt), so a retried activity replays at $0 and a re-run never replays another run's answer. Gemini's native route is the exception: LiteLLM 1.103.4 records its session id and tags but drops the metadata header, and ignores the cache body, so OpenCompany sends no cache body there and a retried Gemini step is billed again.
**Errors** (`services/gateway/errors.py`, `services/llm/protocol.py`). `_classify_error` maps the `budget_exceeded` type to BILLING before any status check, so the run fails once with no retry loop, and a deployment-spawned run pauses its deployment. Failures of gateway calls get gateway wording through `public_message` and the new `public_hint` (allowance used up; model not on the gateway, from `model_cost_map_missing`; reconnect the gateway key). Compaction inside the agent workflow raises such failures instead of retrying them (`stop_on_user_action`).
@@ -21,9 +28,9 @@ OpenCompany's hosted LLM gateway is the stock open-source LiteLLM Proxy, deploye
| Path | Contents |
|---|---|
| `gateway/litellm/config.yaml` | Production proxy config: provider wildcards, budget defaults, unpriced-model refusal, opt-in Redis cache, spend-log retention |
-| `gateway/local/` | Local stack: Postgres 16, Redis 7, LiteLLM with mock models, and `fake_anthropic.py`, a real upstream event stream for streamed-pricing tests |
+| `gateway/local/` | Local stack: Postgres 16, Redis 7, LiteLLM with mock models, and `fake_upstream.py`, fake Anthropic, Gemini and OpenAI-audio upstreams. Mocks reuse one response id on `/v1/messages` (LiteLLM keys spend rows by it and drops repeats), carry no HTTP response on the native Gemini route (LiteLLM's logging drops those) and don't apply on the audio routes, so those tests use the fake upstreams |
| `gateway/tests/test_gateway_contract.py` | Contract tests against the local stack (skipped when it isn't running) |
-| `gateway/tests/test_pins.py` | Offline checks: version and price-list pins agree; every served default model is priced |
+| `gateway/tests/test_pins.py` | Offline checks: version and price-list pins agree; every served default chat, speech and embedding model is priced |
| `gateway/infra/terraform/` | Wrapper around `BerriAI/litellm/aws`: ECS Fargate, Aurora Postgres (IAM auth), ElastiCache, ALB, ACM, Route 53 records in the delegated `gw.opencompany.sh` zone |
| `.github/workflows/gateway-ci.yml` | Terraform checks, cosign verification of the pinned images, locked and audited test dependencies, contract and pin tests |
@@ -34,9 +41,16 @@ OpenCompany's hosted LLM gateway is the stock open-source LiteLLM Proxy, deploye
## Served providers
-OpenAI, Anthropic, Gemini, Groq, Cerebras, xAI, Moonshot (OpenCompany's `kimi`) and Mistral for chat; OpenAI and Groq speech; OpenAI embeddings. `test_pins.py` lists them as `SERVED_PROVIDERS` and checks each default model in `server/config/llm_defaults.json` against the pinned list.
+OpenAI, Anthropic, Gemini, Groq, Cerebras, xAI, Moonshot (OpenCompany's `kimi`) and Mistral for chat (a `"gateway"` block in `server/config/llm_defaults.json`); OpenAI speech (`"gateway": true` in `server/config/speech_defaults.json`); OpenAI embeddings. `test_pins.py` checks each one's default models against the pinned price list, counting any `cost_per` field as a price, as LiteLLM does when it refuses unpriced models.
+
+Not served: OpenRouter, DeepSeek and Sarvam (no price for their default models), Groq speech (1.103.4 bills its transcription at $0), ElevenLabs and Deepgram (OpenCompany speaks their native formats), and every tool API.
+
+**Price override:** `gpt-4o-mini-tts`, OpenCompany's Text to Speech default.
+- **Why:** the speech route returns no usage for the audio, so LiteLLM would bill the input text only, roughly 1% of the cost.
+- **How:** `gateway/litellm/config.yaml` gives the model its own deployment, priced per input token (`input_cost_per_token: 0.000075`). That is about $0.015 per minute of speech, at roughly 200 text tokens a minute.
+- **Checks:** `test_pins.py` keeps the local config at the same price, and the contract test checks the row.
-Not served: OpenRouter, DeepSeek and Sarvam (no price for their default models), ElevenLabs and Deepgram (OpenCompany speaks their native formats), and every tool API.
+`tts-1` and `tts-1-hd` are billed per character, and transcription per second of audio, measured from the file.
## Contract OpenCompany relies on
@@ -46,9 +60,11 @@ Not served: OpenRouter, DeepSeek and Sarvam (no price for their default models),
| Spent allowance | Refused with error type `budget_exceeded` on all three routes; HTTP 429 on 1.103.x, 402 from 1.104. Recognise it by type | `test_budget_refusal_is_typed_on_every_route` |
| Unpriced model | HTTP 403, error type `model_cost_map_missing` | `test_unpriced_model_is_refused` |
| Allowance | `GET /user/info` with the person's key returns `user_info.spend`, `max_budget` and `budget_reset_at`. Needs a Default-type key; LLM-API keys are refused; another user's id is refused | `test_user_info_needs_a_default_key_and_stays_scoped` |
-| Attribution | `x-litellm-session-id`, `x-litellm-tags` (comma-separated, low cardinality: `workflow:`, `source:`, `user:`) and `x-litellm-spend-logs-metadata` (JSON) land on the spend row, queryable with `/spend/logs/v2?session_id=` | `test_attribution_headers_reach_spend_logs` |
-| Retry replay | Request body `cache: {"use-cache": true, "namespace": <stable per activity>, "ttl": 172800}` (OpenAI format through `extra_body`). A repeated call with the same namespace is a cache hit billed at $0, streamed or not, on `/v1/chat/completions` and `/v1/messages` | `test_cache_namespace_replays_retry_for_free`, `test_streamed_retry_replays_for_free`, `test_anthropic_messages_cache_namespace` |
+| Attribution | `x-litellm-session-id`, `x-litellm-tags` (comma-separated, low cardinality: `workflow:`, `source:`, `user:`) and `x-litellm-spend-logs-metadata` (JSON) land on the spend row, queryable with `/spend/logs/v2?session_id=`. On the native Gemini route only the session id and tags land: there the router's metadata goes to `litellm_metadata`, which the spend row prefers, and only key fields are copied across | `test_attribution_headers_reach_spend_logs`, `test_messages_route_writes_a_priced_row_for_the_session`, `test_gemini_native_route_writes_a_priced_row_for_the_session` |
+| Retry replay | Request body `cache: {"use-cache": true, "namespace": <stable per activity>, "ttl": 172800}` (OpenAI format through `extra_body`). A repeated call with the same namespace is a cache hit billed at $0, streamed or not, on `/v1/chat/completions` and `/v1/messages`. The native Gemini route ignores it | `test_cache_namespace_replays_retry_for_free`, `test_streamed_retry_replays_for_free`, `test_anthropic_messages_cache_namespace` |
| Streamed pricing | A real upstream stream through a dotted alias is billed at the deployment's price | `test_streamed_alias_is_priced_at_the_deployment` |
+| Embeddings and speech | `/v1/embeddings`, `/v1/audio/speech` and `/v1/audio/transcriptions` with the `openai/` prefix bill the person and keep all three attribution headers; `gpt-4o-mini-tts` at the override price. There is no `/v1/audio/translations` | `test_embeddings_route_writes_an_attributed_row`, `test_speech_routes_write_attributed_rows`, `test_the_gateway_has_no_audio_translation_route`, `test_price_overrides_agree` |
+| Known gaps | Groq transcription billed at $0; on the native Gemini route, metadata dropped and the cache body ignored. Strict xfail: each test fails the suite once a LiteLLM upgrade closes its gap, so the workaround can be lifted | `test_groq_transcription_is_billed`, `test_gemini_native_route_keeps_the_spend_logs_metadata`, `test_gemini_native_route_replays_retry_for_free` |
## Keys
diff --git a/gateway/README.md b/gateway/README.md
index c7aa435c..ece4e5f9 100644
--- a/gateway/README.md
+++ b/gateway/README.md
@@ -25,16 +25,20 @@ The hosted LLM gateway for OpenCompany: invited people use LLMs on platform-owne
| Kind | Providers |
|---|---|
| Chat | OpenAI, Anthropic, Gemini, Groq, Cerebras, xAI, Moonshot (Kimi), Mistral |
-| Speech | OpenAI (speech and transcription), Groq (transcription) |
+| Speech | OpenAI (speech and transcription) |
| Embeddings | OpenAI |
-Not served: OpenRouter, DeepSeek and Sarvam (their default models have no price in the list), ElevenLabs and Deepgram (OpenCompany uses their native formats), and every tool API (Brave, Serper, TikHub, Google Maps, Perplexity, DeepL, Apify and the rest). Those stay on each install's own keys.
+Not served: OpenRouter, DeepSeek and Sarvam (their default models have no price in the list), Groq speech (1.103.4 bills Groq transcription at $0), ElevenLabs and Deepgram (OpenCompany uses their native formats), and every tool API (Brave, Serper, TikHub, Google Maps, Perplexity, DeepL, Apify and the rest). Those stay on each install's own keys.
-To serve another provider: add its secret name to `secret_env_names`, its wildcard to `litellm/config.yaml`, and its OpenCompany provider id to `SERVED_PROVIDERS` in `tests/test_pins.py` (which checks its default model is priced), then apply.
+To serve another provider:
+1. Add its secret name to `secret_env_names` and its wildcard to `litellm/config.yaml`.
+2. Give it a `"gateway"` block in `server/config/llm_defaults.json` (chat), and `"gateway": true` in `server/config/speech_defaults.json` if its speech is served too.
+3. Run the tests. `tests/test_pins.py` checks that its default models are priced; check its routes bill the person on the local stack.
+4. Apply.
## Local stack and tests
-Needs Docker. Mock models plus a fake Anthropic upstream ([local/fake_anthropic.py](local/fake_anthropic.py)) for streamed calls: nothing is billed and no vendor key is needed.
+Needs Docker. Mock models plus fake Anthropic, Gemini and OpenAI-audio upstreams ([local/fake_upstream.py](local/fake_upstream.py)) for what mocks can't show (spend rows on `/v1/messages`, the native Gemini route and the audio routes, streamed pricing): nothing is billed and no vendor key is needed. Three contract tests are strict expected failures that pin LiteLLM gaps OpenCompany works around. When an upgrade closes a gap, its test passes and fails the suite: lift the workaround and drop the marker.
```bash
docker compose -f gateway/local/docker-compose.yml up -d --wait
@@ -108,5 +112,8 @@ Why 1.103.x: it is the only stable line with the streamed-alias billing fix (Lit
## Known limits
- Replay covers completed LLM calls only; an abandoned stream is billed for what it produced.
+- Gemini SDK calls (the native `generateContent` route) carry the session id and tags, but 1.103.4 drops their per-call metadata and doesn't replay them from the cache, so a retried Gemini step is billed again.
+- `gpt-4o-mini-tts` (OpenCompany's Text to Speech default) carries a price override in `litellm/config.yaml`. The speech route returns no usage for the audio, so LiteLLM would otherwise bill the input text only, about 1% of the cost. The override's per-token rate assumes about 200 text tokens per minute of speech; revisit it if OpenAI's price changes.
+- Groq speech is not served: 1.103.4 bills Groq transcription at $0. The `groq/*` wildcard still lets a person call Groq Whisper directly with their key, billed $0 to them; Groq charges about $0.11 per hour of audio.
- A Default key can create and delete that person's own keys, but not raise their allowance. Revoke a leaked key in the console.
- The open-source tier has no SSO beyond 5 users, no audit log, no automatic key rotation, and no route permissions for tool pass-through. Those need a LiteLLM Enterprise license.
diff --git a/gateway/litellm/config.yaml b/gateway/litellm/config.yaml
index 079d862c..c89e23a9 100644
--- a/gateway/litellm/config.yaml
+++ b/gateway/litellm/config.yaml
@@ -57,8 +57,21 @@ model_list:
litellm_params:
model: mistral/*
api_key: os.environ/MISTRAL_API_KEY
- # Speech (OpenAI and Groq) and embeddings (OpenAI) use the wildcards above:
- # /v1/audio/speech, /v1/audio/transcriptions and /v1/embeddings.
+ # OpenCompany's speech (OpenAI) and embeddings (OpenAI) use the openai/*
+ # wildcard: /v1/audio/speech, /v1/audio/transcriptions and /v1/embeddings.
+ #
+ # Except gpt-4o-mini-tts, OpenCompany's Text to Speech default. The price
+ # list prices its audio output, but the speech route returns no usage, so
+ # LiteLLM would bill the input text only, about 1% of the cost. Priced per
+ # input token instead: about $0.015 per minute of speech at ~200 text tokens
+ # a minute. An exact name wins over the wildcard. gateway/local/config.local.yaml
+ # carries the same price (gateway/tests/test_pins.py checks).
+ - model_name: openai/gpt-4o-mini-tts
+ litellm_params:
+ model: openai/gpt-4o-mini-tts
+ api_key: os.environ/OPENAI_API_KEY
+ input_cost_per_token: 0.000075
+ output_cost_per_token: 0
router_settings:
# Shared state across gateway tasks: rate-limit and budget counters,
diff --git a/gateway/local/config.local.yaml b/gateway/local/config.local.yaml
index 7411fdaf..493f4a5e 100644
--- a/gateway/local/config.local.yaml
+++ b/gateway/local/config.local.yaml
@@ -1,9 +1,10 @@
-# Local OpenCompany Gateway config: mock models instead of real vendors, so
-# nothing is billed and no vendor keys are needed.
+# Local OpenCompany Gateway config: mock models and fake upstreams instead of
+# real vendors, so nothing is billed and no vendor keys are needed.
#
# The policy blocks mirror gateway/litellm/config.yaml. Differences, all
-# deliberate: mock_response on every model, spend logs flush every second so
-# the tests can read them, and there is no Slack alerting or proxy-range trust.
+# deliberate: mock responses or fake_upstream.py behind every model, spend logs
+# flush every second so the tests can read them, and there is no Slack alerting
+# or proxy-range trust.
model_list:
- model_name: openai/*
@@ -11,6 +12,9 @@ model_list:
model: openai/*
api_key: sk-local-unused
mock_response: pong
+ # The audio routes ignore mock_response and call the upstream: the fake
+ # one (fake_upstream.py), never api.openai.com.
+ api_base: http://fake-upstream:8080/v1
- model_name: claude-*
litellm_params:
model: anthropic/claude-*
@@ -21,15 +25,43 @@ model_list:
model: gemini/gemini-*
api_key: local-unused
mock_response: pong
+ # Real upstreams (fake_upstream.py) for what mocks can't show: spend rows on
+ # /v1/messages, the native Gemini route and the audio routes, and streamed
+ # pricing. Exact names win over the wildcards above.
+ #
# A dotted alias that is not a price-list key, the shape LiteLLM's own test
# for PR #44341 uses: a stream through it must still be priced at the
- # deployment's model. It streams from fake_anthropic.py, because mock
- # responses skip LiteLLM's upstream streaming path.
+ # deployment's model.
- model_name: claude-opus-4.8-20261010
litellm_params:
model: anthropic/claude-opus-4-8
api_key: sk-local-unused
- api_base: http://fake-anthropic:8080
+ api_base: http://fake-upstream:8080
+ - model_name: gemini-2.5-flash-live
+ litellm_params:
+ model: gemini/gemini-2.5-flash
+ api_key: local-unused
+ api_base: http://fake-upstream:8080
+ # The price override of gateway/litellm/config.yaml, same values.
+ - model_name: openai/gpt-4o-mini-tts
+ litellm_params:
+ model: openai/gpt-4o-mini-tts
+ api_key: sk-local-unused
+ api_base: http://fake-upstream:8080/v1
+ input_cost_per_token: 0.000075
+ output_cost_per_token: 0
+ # OpenCompany's default embedding model under its wire name: the OpenAI SDK
+ # asks for base64 vectors, which the mock's "pong" can't decode to.
+ - model_name: openai/text-embedding-3-small
+ litellm_params:
+ model: openai/text-embedding-3-small
+ api_key: sk-local-unused
+ api_base: http://fake-upstream:8080/v1
+ - model_name: groq/whisper-large-v3
+ litellm_params:
+ model: groq/whisper-large-v3
+ api_key: local-unused
+ api_base: http://fake-upstream:8080/v1
router_settings:
redis_host: os.environ/REDIS_HOST
diff --git a/gateway/local/docker-compose.yml b/gateway/local/docker-compose.yml
index aa7cec3f..252ebc07 100644
--- a/gateway/local/docker-compose.yml
+++ b/gateway/local/docker-compose.yml
@@ -30,12 +30,13 @@ services:
timeout: 3s
retries: 30
- fake-anthropic:
- # A real upstream stream for the streamed-pricing test (see the file).
+ fake-upstream:
+ # Anthropic and Gemini answering the way the vendors do, for the tests
+ # that need a real upstream (see the file).
image: python:3.12-alpine
- command: ["python", "/app/fake_anthropic.py"]
+ command: ["python", "/app/fake_upstream.py"]
volumes:
- - ./fake_anthropic.py:/app/fake_anthropic.py:ro
+ - ./fake_upstream.py:/app/fake_upstream.py:ro
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"]
interval: 2s
@@ -65,7 +66,7 @@ services:
condition: service_healthy
redis:
condition: service_healthy
- fake-anthropic:
+ fake-upstream:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
diff --git a/gateway/local/fake_anthropic.py b/gateway/local/fake_anthropic.py
deleted file mode 100644
index 564235ee..00000000
--- a/gateway/local/fake_anthropic.py
+++ /dev/null
@@ -1,75 +0,0 @@
-"""Fake Anthropic upstream for the gateway contract tests.
-
-Answers POST /v1/messages the way Anthropic does: a JSON message, or a
-server-sent event stream when the request sets "stream": true, always with
-the same usage. Mock responses inside LiteLLM skip its streaming path, so the
-streamed-pricing test needs a real upstream stream; this is the smallest one.
-Standard library only.
-"""
-
-from __future__ import annotations
-
-import json
-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
-
-INPUT_TOKENS = 30
-OUTPUT_TOKENS = 40
-
-
-def _message(model: str) -> dict:
- return {
- "id": "msg_contract",
- "type": "message",
- "role": "assistant",
- "model": model,
- "content": [{"type": "text", "text": "pong"}],
- "stop_reason": "end_turn",
- "stop_sequence": None,
- "usage": {"input_tokens": INPUT_TOKENS, "output_tokens": OUTPUT_TOKENS},
- }
-
-
-def _events(model: str) -> list[tuple[str, dict]]:
- start = {**_message(model), "content": [], "stop_reason": None, "usage": {"input_tokens": INPUT_TOKENS, "output_tokens": 1}}
- return [
- ("message_start", {"type": "message_start", "message": start}),
- ("content_block_start", {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}),
- ("content_block_delta", {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "pong"}}),
- ("content_block_stop", {"type": "content_block_stop", "index": 0}),
- ("message_delta", {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": None}, "usage": {"output_tokens": OUTPUT_TOKENS}}),
- ("message_stop", {"type": "message_stop"}),
- ]
-
-
-class Handler(BaseHTTPRequestHandler):
- def do_GET(self) -> None: # health check
- self.send_response(200 if self.path == "/health" else 404)
- self.end_headers()
-
- def do_POST(self) -> None:
- if not self.path.endswith("/v1/messages"):
- self.send_error(404)
- return
- body = json.loads(self.rfile.read(int(self.headers.get("content-length") or 0)) or b"{}")
- model = body.get("model", "claude-opus-4-8")
- if body.get("stream"):
- self.send_response(200)
- self.send_header("content-type", "text/event-stream")
- self.end_headers()
- for name, payload in _events(model):
- self.wfile.write(f"event: {name}\ndata: {json.dumps(payload)}\n\n".encode())
- self.wfile.flush()
- return
- data = json.dumps(_message(model)).encode()
- self.send_response(200)
- self.send_header("content-type", "application/json")
- self.send_header("content-length", str(len(data)))
- self.end_headers()
- self.wfile.write(data)
-
- def log_message(self, format: str, *args: object) -> None:
- pass
-
-
-if __name__ == "__main__":
- ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
diff --git a/gateway/local/fake_upstream.py b/gateway/local/fake_upstream.py
new file mode 100644
index 00000000..10659f1d
--- /dev/null
+++ b/gateway/local/fake_upstream.py
@@ -0,0 +1,141 @@
+"""Fake Anthropic, Gemini and OpenAI-audio upstreams for the gateway contract tests.
+
+Mock responses inside LiteLLM skip the paths real traffic takes: on
+/v1/messages a mock ignores ``stream`` and reuses one message id, on the
+native Gemini route it carries no HTTP response, so LiteLLM's logging drops
+it, and the audio routes ignore mocks altogether. This server answers like
+the vendors do, with fixed usage:
+
+- ``POST .../v1/messages``: an Anthropic message, or its server-sent events
+ when the request sets ``"stream": true``.
+- ``POST .../models/<id>:generateContent``: a Gemini response.
+- ``POST .../audio/speech``: a few bytes of "audio" (OpenAI).
+- ``POST .../audio/transcriptions``: a transcript of AUDIO_SECONDS (OpenAI
+ and Groq).
+- ``POST .../embeddings``: one small vector per input, as floats or, as the
+ OpenAI SDK asks by default, base64 (OpenAI).
+
+Every response gets a fresh id: LiteLLM keys a spend row by it and drops a
+repeat. Standard library only.
+"""
+
+from __future__ import annotations
+
+import base64
+import json
+import struct
+import uuid
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+INPUT_TOKENS = 30
+OUTPUT_TOKENS = 40
+AUDIO_SECONDS = 1.0
+
+
+def _message(model: str) -> dict:
+ return {
+ "id": f"msg_{uuid.uuid4().hex}",
+ "type": "message",
+ "role": "assistant",
+ "model": model,
+ "content": [{"type": "text", "text": "pong"}],
+ "stop_reason": "end_turn",
+ "stop_sequence": None,
+ "usage": {"input_tokens": INPUT_TOKENS, "output_tokens": OUTPUT_TOKENS},
+ }
+
+
+def _events(model: str) -> list[tuple[str, dict]]:
+ start = {**_message(model), "content": [], "stop_reason": None, "usage": {"input_tokens": INPUT_TOKENS, "output_tokens": 1}}
+ return [
+ ("message_start", {"type": "message_start", "message": start}),
+ ("content_block_start", {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}),
+ ("content_block_delta", {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "pong"}}),
+ ("content_block_stop", {"type": "content_block_stop", "index": 0}),
+ ("message_delta", {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": None}, "usage": {"output_tokens": OUTPUT_TOKENS}}),
+ ("message_stop", {"type": "message_stop"}),
+ ]
+
+
+def _embeddings(body: dict) -> dict:
+ inputs = body.get("input")
+ vector = [0.1, 0.2, 0.3]
+ encoded = base64.b64encode(struct.pack(f"<{len(vector)}f", *vector)).decode()
+ embedding = encoded if body.get("encoding_format") == "base64" else vector
+ return {
+ "object": "list",
+ "data": [
+ {"object": "embedding", "index": i, "embedding": embedding}
+ for i in range(len(inputs) if isinstance(inputs, list) else 1)
+ ],
+ "model": body.get("model"),
+ "usage": {"prompt_tokens": INPUT_TOKENS, "total_tokens": INPUT_TOKENS},
+ }
+
+
+def _gemini(model: str) -> dict:
+ return {
+ "candidates": [{"content": {"role": "model", "parts": [{"text": "pong"}]}, "finishReason": "STOP", "index": 0}],
+ "usageMetadata": {
+ "promptTokenCount": INPUT_TOKENS,
+ "candidatesTokenCount": OUTPUT_TOKENS,
+ "totalTokenCount": INPUT_TOKENS + OUTPUT_TOKENS,
+ },
+ "modelVersion": model,
+ "responseId": uuid.uuid4().hex,
+ }
+
+
+class Handler(BaseHTTPRequestHandler):
+ def do_GET(self) -> None: # health check
+ self.send_response(200 if self.path == "/health" else 404)
+ self.end_headers()
+
+ def _send(self, content_type: str, data: bytes) -> None:
+ self.send_response(200)
+ self.send_header("content-type", content_type)
+ self.send_header("content-length", str(len(data)))
+ self.end_headers()
+ self.wfile.write(data)
+
+ def _json(self, body: dict) -> None:
+ self._send("application/json", json.dumps(body).encode())
+
+ def do_POST(self) -> None:
+ raw = self.rfile.read(int(self.headers.get("content-length") or 0))
+ path = self.path.split("?", 1)[0]
+ if path.endswith("/audio/speech"):
+ self._send("audio/mpeg", b"ID3" + uuid.uuid4().bytes)
+ return
+ if path.endswith("/audio/transcriptions"): # multipart; the audio is not read
+ # Like the vendors, only verbose_json reports the duration.
+ verbose = b"verbose_json" in raw
+ self._json({"text": "pong", **({"language": "english", "duration": AUDIO_SECONDS} if verbose else {})})
+ return
+ body = json.loads(raw or b"{}")
+ if path.endswith("/embeddings"):
+ self._json(_embeddings(body))
+ return
+ if path.endswith(":generateContent"):
+ self._json(_gemini(path.rsplit("/", 1)[-1].split(":", 1)[0]))
+ return
+ if not path.endswith("/v1/messages"):
+ self.send_error(404)
+ return
+ model = body.get("model", "claude-opus-4-8")
+ if body.get("stream"):
+ self.send_response(200)
+ self.send_header("content-type", "text/event-stream")
+ self.end_headers()
+ for name, payload in _events(model):
+ self.wfile.write(f"event: {name}\ndata: {json.dumps(payload)}\n\n".encode())
+ self.wfile.flush()
+ return
+ self._json(_message(model))
+
+ def log_message(self, format: str, *args: object) -> None:
+ pass
+
+
+if __name__ == "__main__":
+ ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
diff --git a/gateway/tests/test_gateway_contract.py b/gateway/tests/test_gateway_contract.py
index a24f8616..6b766bff 100644
--- a/gateway/tests/test_gateway_contract.py
+++ b/gateway/tests/test_gateway_contract.py
@@ -2,8 +2,8 @@
They run against the local stack in gateway/local (mock models, no real keys)
and pin the LiteLLM behaviours the OpenCompany integration relies on: budget
-refusals, key scope, attribution, streamed pricing, retry replay, and the
-native Anthropic and Gemini routes.
+refusals, key scope, attribution, streamed pricing, retry replay, the native
+Anthropic and Gemini routes, and the embedding and speech routes.
docker compose -f gateway/local/docker-compose.yml up -d --wait
uv run --project gateway pytest gateway/tests -v
@@ -13,10 +13,12 @@ Set GATEWAY_URL and GATEWAY_MASTER_KEY to point the suite at another stack.
from __future__ import annotations
+import io
import json
import os
import time
import uuid
+import wave
from collections.abc import Callable, Iterator
from dataclasses import dataclass
from datetime import date, timedelta
@@ -27,9 +29,18 @@ import pytest
GATEWAY_URL = os.environ.get("GATEWAY_URL", "http://127.0.0.1:4000")
MASTER_KEY = os.environ.get("GATEWAY_MASTER_KEY", "sk-local-master-key")
CHAT_MODEL = "openai/gpt-4o-mini"
-# A dotted alias that is not a price-list key, streaming from a fake Anthropic
-# upstream (gateway/local/config.local.yaml, gateway/local/fake_anthropic.py).
+# Models answered by gateway/local/fake_upstream.py (gateway/local/config.local.yaml).
+# Mocks reuse one response id on /v1/messages and carry no HTTP response on the
+# native Gemini route, and LiteLLM writes no spend row for either, so every
+# spend-row check on those routes uses these. ALIAS_MODEL is also a dotted
+# alias that is not a price-list key.
ALIAS_MODEL = "claude-opus-4.8-20261010"
+GEMINI_LIVE_MODEL = "gemini-2.5-flash-live"
+# OpenCompany's default speech models under their wire names, answered by
+# fake_upstream.py too: mocks don't apply on the audio routes.
+TTS_MODEL = "openai/gpt-4o-mini-tts"
+STT_MODEL = "openai/whisper-1"
+GROQ_STT_MODEL = "groq/whisper-large-v3"
# 1.103.x always answers a spent allowance with 429; budget_exceeded_status_code
# makes it 402 from 1.104. OpenCompany reads the error type either way.
BUDGET_STATUSES = {402, 429}
@@ -99,6 +110,36 @@ def _error_type(resp: httpx.Response) -> str | None:
return error.get("type") if isinstance(error, dict) else None
+def _attribution(session: str) -> dict[str, str]:
+ """The attribution headers OpenCompany sends (services/gateway/attribution.py)."""
+ return {
+ "x-litellm-session-id": session,
+ "x-litellm-tags": "source:chat",
+ "x-litellm-spend-logs-metadata": json.dumps({"chat_run_id": session}),
+ }
+
+
+def _attributed_row(admin: httpx.Client, session: str) -> dict:
+ """The session's spend row, priced and carrying all three attribution headers."""
+ rows = _spend_rows(admin, session_id=session)
+ assert rows, f"no spend row for session {session}"
+ row = rows[0]
+ assert float(row.get("spend") or 0) > 0, row
+ assert "source:chat" in (row.get("request_tags") or []), row
+ assert ((row.get("metadata") or {}).get("spend_logs_metadata") or {}).get("chat_run_id") == session, row
+ return row
+
+
+def _wav(seconds: float = 1.0, rate: int = 16000) -> bytes:
+ buffer = io.BytesIO()
+ with wave.open(buffer, "wb") as handle:
+ handle.setnchannels(1)
+ handle.setsampwidth(2)
+ handle.setframerate(rate)
+ handle.writeframes(b"\x00\x00" * int(rate * seconds))
+ return buffer.getvalue()
+
+
def _spend_rows(admin: httpx.Client, **filters: str) -> list[dict]:
"""Poll /spend/logs/v2 until rows matching the filters appear (spend is batch-written)."""
today = date.today()
@@ -288,3 +329,151 @@ def test_gemini_native_route_accepts_virtual_key(person: Callable[..., Person])
"""The gemini-* wildcard serves the Gemini SDK, which sends its key as x-goog-api-key."""
resp = _generate_content(person().key)
assert resp.status_code == 200, resp.text
+
+
+def test_messages_route_writes_a_priced_row_for_the_session(person: Callable[..., Person], admin: httpx.Client) -> None:
+ """The Anthropic SDK's own route bills the person and carries attribution, streamed or not."""
+ key = person().key
+ for stream in (False, True):
+ session = f"r_{uuid.uuid4().hex}"
+ resp = _messages(
+ key,
+ headers={"x-litellm-session-id": session, "x-litellm-tags": "source:chat"},
+ body={"model": ALIAS_MODEL, "temperature": 1, **({"stream": True} if stream else {})},
+ )
+ assert resp.status_code == 200, resp.text
+ rows = _spend_rows(admin, session_id=session)
+ assert rows, f"no spend row for the {'streamed' if stream else 'plain'} /v1/messages call"
+ assert float(rows[0].get("spend") or 0) > 0, rows[0]
+ assert "source:chat" in (rows[0].get("request_tags") or []), rows[0]
+
+
+def test_gemini_native_route_writes_a_priced_row_for_the_session(person: Callable[..., Person], admin: httpx.Client) -> None:
+ """The Gemini SDK's native route bills the person and keeps the session id and tags.
+
+ It is the only route where 1.103.4 drops x-litellm-spend-logs-metadata (router metadata
+ goes to litellm_metadata there and wins) and ignores the opt-in cache body.
+ """
+ session = f"r_{uuid.uuid4().hex}"
+ resp = httpx.post(
+ f"{GATEWAY_URL}/v1beta/models/{GEMINI_LIVE_MODEL}:generateContent",
+ headers={"x-goog-api-key": person().key, "x-litellm-session-id": session, "x-litellm-tags": "source:chat"},
+ json={"contents": [{"role": "user", "parts": [{"text": "ping"}]}]},
+ timeout=30,
+ )
+ assert resp.status_code == 200, resp.text
+ rows = _spend_rows(admin, session_id=session)
+ assert rows, "no spend row for the native Gemini call"
+ assert float(rows[0].get("spend") or 0) > 0, rows[0]
+ assert "source:chat" in (rows[0].get("request_tags") or []), rows[0]
+
+
+def test_embeddings_route_writes_an_attributed_row(person: Callable[..., Person], admin: httpx.Client) -> None:
+ """OpenAI embeddings (long-term memory, the Embedding Generator) bill the person with attribution."""
+ session = f"r_{uuid.uuid4().hex}"
+ resp = httpx.post(
+ f"{GATEWAY_URL}/v1/embeddings",
+ headers={**_auth(person().key), **_attribution(session)},
+ json={"model": "openai/text-embedding-3-small", "input": ["ping"]},
+ timeout=30,
+ )
+ assert resp.status_code == 200, resp.text
+ _attributed_row(admin, session)
+
+
+def test_speech_routes_write_attributed_rows(person: Callable[..., Person], admin: httpx.Client) -> None:
+ """Text to Speech and Speech to Text bill the person with attribution.
+
+ Transcription is billed per second of audio, measured from the file.
+ gpt-4o-mini-tts is billed per input token at the config's override: the
+ speech route returns no usage for the audio, which the price list prices.
+ """
+ key = person().key
+ speech_session, transcription_session = f"r_{uuid.uuid4().hex}", f"r_{uuid.uuid4().hex}"
+ speech = httpx.post(
+ f"{GATEWAY_URL}/v1/audio/speech",
+ headers={**_auth(key), **_attribution(speech_session)},
+ json={"model": TTS_MODEL, "input": "ping", "voice": "alloy"},
+ timeout=30,
+ )
+ assert speech.status_code == 200 and speech.content, speech.text
+ transcription = httpx.post(
+ f"{GATEWAY_URL}/v1/audio/transcriptions",
+ headers={**_auth(key), **_attribution(transcription_session)},
+ data={"model": STT_MODEL},
+ files={"file": ("ping.wav", _wav(), "audio/wav")},
+ timeout=30,
+ )
+ assert transcription.status_code == 200, transcription.text
+ assert transcription.json().get("text") == "pong", transcription.text
+ speech_row = _attributed_row(admin, speech_session)
+ # At least one input token at the override's $0.000075, not the price
+ # list's $0.0000006.
+ assert float(speech_row["spend"]) >= 0.000075, speech_row
+ _attributed_row(admin, transcription_session)
+
+
+def test_the_gateway_has_no_audio_translation_route(person: Callable[..., Person]) -> None:
+ """Why OpenCompany never asks the gateway to translate audio (nodes/speech/speech_to_text.py)."""
+ resp = httpx.post(
+ f"{GATEWAY_URL}/v1/audio/translations",
+ headers=_auth(person().key),
+ data={"model": STT_MODEL},
+ files={"file": ("ping.wav", _wav(), "audio/wav")},
+ timeout=30,
+ )
+ assert resp.status_code in (404, 405), resp.text
+
+
+# Gaps in 1.103.4 that OpenCompany works around. Each test passes once LiteLLM
+# fixes its gap, and strict xfail then fails the suite: lift the workaround
+# and drop the marker.
+
+
[email protected](strict=True, reason="1.103.4 bills Groq transcription at $0, so Groq speech isn't served")
+def test_groq_transcription_is_billed(person: Callable[..., Person], admin: httpx.Client) -> None:
+ """Serve Groq speech ("gateway": true in server/config/speech_defaults.json) once this passes."""
+ session = f"r_{uuid.uuid4().hex}"
+ resp = httpx.post(
+ f"{GATEWAY_URL}/v1/audio/transcriptions",
+ headers={**_auth(person().key), **_attribution(session)},
+ data={"model": GROQ_STT_MODEL},
+ files={"file": ("ping.wav", _wav(), "audio/wav")},
+ timeout=30,
+ )
+ assert resp.status_code == 200, resp.text
+ _attributed_row(admin, session)
+
+
[email protected](strict=True, reason="1.103.4 drops x-litellm-spend-logs-metadata on the native Gemini route")
+def test_gemini_native_route_keeps_the_spend_logs_metadata(person: Callable[..., Person], admin: httpx.Client) -> None:
+ session = f"r_{uuid.uuid4().hex}"
+ resp = httpx.post(
+ f"{GATEWAY_URL}/v1beta/models/{GEMINI_LIVE_MODEL}:generateContent",
+ headers={"x-goog-api-key": person().key, **_attribution(session)},
+ json={"contents": [{"role": "user", "parts": [{"text": "ping"}]}]},
+ timeout=30,
+ )
+ assert resp.status_code == 200, resp.text
+ _attributed_row(admin, session)
+
+
[email protected](strict=True, reason="1.103.4 ignores the cache body on the native Gemini route")
+def test_gemini_native_route_replays_retry_for_free(person: Callable[..., Person], admin: httpx.Client) -> None:
+ """Send the replay-cache body from services/llm/providers/gemini.py once this passes."""
+ key = person().key
+ namespace = f"oc1-{uuid.uuid4().hex}"
+ for suffix in ("first", "retry"):
+ resp = httpx.post(
+ f"{GATEWAY_URL}/v1beta/models/{GEMINI_LIVE_MODEL}:generateContent",
+ headers={"x-goog-api-key": key, "x-litellm-session-id": f"{namespace}-{suffix}"},
+ json={
+ "contents": [{"role": "user", "parts": [{"text": "ping"}]}],
+ "cache": {"use-cache": True, "namespace": namespace, "ttl": 600},
+ },
+ timeout=30,
+ )
+ assert resp.status_code == 200, resp.text
+ rows = _spend_rows(admin, session_id=f"{namespace}-retry")
+ assert rows, "no spend row for the retry"
+ assert float(rows[0].get("spend") or 0) == 0.0, rows[0]
diff --git a/gateway/tests/test_pins.py b/gateway/tests/test_pins.py
index b51ec031..6e542a89 100644
--- a/gateway/tests/test_pins.py
+++ b/gateway/tests/test_pins.py
@@ -3,8 +3,11 @@
- LiteLLM's version and the price-list commit agree across Terraform, the
local stack and CI.
- The pinned price list prices the default model of every provider the
- gateway serves (a "gateway" block in server/config/llm_defaults.json), so
- OpenCompany's defaults are never refused as unpriced.
+ gateway serves (a "gateway" block in server/config/llm_defaults.json),
+ the default speech models of the providers whose speech it serves
+ ("gateway": true in server/config/speech_defaults.json) and OpenAI's
+ default embedding model, so OpenCompany's defaults are never refused as
+ unpriced.
"""
from __future__ import annotations
@@ -20,6 +23,8 @@ import yaml
REPO = Path(__file__).resolve().parents[2]
TERRAFORM = (REPO / "gateway/infra/terraform/main.tf").read_text(encoding="utf-8")
+GATEWAY_CONFIG = yaml.safe_load((REPO / "gateway/litellm/config.yaml").read_text(encoding="utf-8"))
+LOCAL_CONFIG = yaml.safe_load((REPO / "gateway/local/config.local.yaml").read_text(encoding="utf-8"))
COMPOSE = yaml.safe_load((REPO / "gateway/local/docker-compose.yml").read_text(encoding="utf-8"))
CI = yaml.safe_load((REPO / ".github/workflows/gateway-ci.yml").read_text(encoding="utf-8"))
LITELLM = COMPOSE["services"]["litellm"]
@@ -28,6 +33,10 @@ LITELLM = COMPOSE["services"]["litellm"]
# server/config/llm_defaults.json (OpenCompany's services/gateway/routing.py).
PROVIDERS = json.loads((REPO / "server/config/llm_defaults.json").read_text(encoding="utf-8"))["providers"]
SERVED = {name: config["gateway"] for name, config in PROVIDERS.items() if isinstance(config.get("gateway"), dict)}
+SPEECH = json.loads((REPO / "server/config/speech_defaults.json").read_text(encoding="utf-8"))["providers"]
+SPEECH_SERVED = {name: block for name, block in SPEECH.items() if block.get("gateway") is True}
+# DEFAULT_EMBEDDING_MODELS["openai"] in server/services/memory/vector_store.py.
+EMBEDDING_MODEL = "text-embedding-3-small"
def _terraform_string(name: str) -> str:
@@ -45,23 +54,63 @@ def test_pins_agree() -> None:
assert LITELLM["environment"]["LITELLM_MODEL_COST_MAP_URL"] == _terraform_string("LITELLM_MODEL_COST_MAP_URL")
-def test_served_default_models_are_priced() -> None:
+def _price_overrides(config: dict) -> dict[str, dict]:
+ return {
+ deployment["model_name"]: {key: value for key, value in deployment["litellm_params"].items() if "cost_per" in key}
+ for deployment in config["model_list"]
+ if any("cost_per" in key for key in deployment["litellm_params"])
+ }
+
+
+def test_price_overrides_agree() -> None:
+ """The contract tests run on the local config, so it must bill as the deployed one does."""
+ overrides = _price_overrides(GATEWAY_CONFIG)
+ assert overrides, "gateway/litellm/config.yaml has no price overrides"
+ assert _price_overrides(LOCAL_CONFIG) == overrides
+
+
[email protected](scope="module")
+def prices() -> dict:
url = LITELLM["environment"]["LITELLM_MODEL_COST_MAP_URL"]
assert re.search(r"/[0-9a-f]{40}/", url), "the price list must be pinned to a commit"
try:
- prices = httpx.get(url, timeout=60).raise_for_status().json()
+ return httpx.get(url, timeout=60).raise_for_status().json()
except httpx.HTTPError as exc:
if os.environ.get("CI"):
raise
pytest.skip(f"price list unreachable: {exc}")
+
+
+def _priced(prices: dict, *candidates: str) -> bool:
+ """LiteLLM prices a model under its wire name or its bare id, and counts
+ it as priced when that entry has any cost_per field: per token, per
+ character (tts-1) or per second (Whisper). The same test refuses
+ unpriced models (proxy/auth/auth_checks.py)."""
+ entry = next((prices[key] for key in candidates if key in prices), {})
+ return any("cost_per" in key for key in entry)
+
+
+def test_served_default_models_are_priced(prices: dict) -> None:
assert SERVED, "no provider in llm_defaults.json has a gateway block"
unpriced = []
for provider, block in SERVED.items():
model = PROVIDERS[provider]["default_model"]
- # LiteLLM prices a model under its wire name, its bare id, or (for
- # the native Gemini route, whose wire name is bare) gemini/<id>.
- candidates = (block.get("model_prefix", "") + model, model, f"{provider}/{model}")
- entry = next((prices[key] for key in candidates if key in prices), {})
- if "input_cost_per_token" not in entry and "output_cost_per_token" not in entry:
+ # The native Gemini route's wire name is bare; its price is under gemini/<id>.
+ if not _priced(prices, block.get("model_prefix", "") + model, model, f"{provider}/{model}"):
unpriced.append(f"{provider}: {model}")
assert not unpriced, f"served default models with no price in the pinned list: {unpriced}"
+
+
+def test_served_speech_and_embedding_defaults_are_priced(prices: dict) -> None:
+ # The route (URL and prefix) of a speech provider is its llm_defaults.json gateway block.
+ assert SPEECH_SERVED and set(SPEECH_SERVED) <= set(SERVED), SPEECH_SERVED.keys()
+ unpriced = []
+ for provider, block in SPEECH_SERVED.items():
+ prefix = SERVED[provider].get("model_prefix", "")
+ for direction in ("tts", "stt"):
+ model = (block.get(direction) or {}).get("default_model")
+ if model and not _priced(prices, prefix + model, model):
+ unpriced.append(f"{provider} {direction}: {model}")
+ if not _priced(prices, SERVED["openai"]["model_prefix"] + EMBEDDING_MODEL, EMBEDDING_MODEL):
+ unpriced.append(f"openai embeddings: {EMBEDDING_MODEL}")
+ assert not unpriced, f"served speech and embedding defaults with no price in the pinned list: {unpriced}"
diff --git a/server/config/speech_defaults.json b/server/config/speech_defaults.json
index 3cddce2b..7a503b63 100644
--- a/server/config/speech_defaults.json
+++ b/server/config/speech_defaults.json
@@ -11,6 +11,8 @@
"openai": {
"base_url": "https://api.openai.com/v1",
"credential_id": "openai",
+ "_gateway_comment": "Served by the OpenCompany Gateway when the install has no key of its own (nodes/speech/_base.py provider_key). The gateway's URL and LiteLLM model prefix come from this provider's gateway block in llm_defaults.json. It has no translation route, so translate falls back to in-language transcription there. gpt-4o-mini-tts is billed through a price override in gateway/litellm/config.yaml (the speech route returns no audio usage); tts-1 and tts-1-hd are billed per character, transcription per second.",
+ "gateway": true,
"tts": {
"endpoint": "/audio/speech",
"default_model": "gpt-4o-mini-tts",
@@ -67,6 +69,7 @@
"_comment": "Serves OpenAI's exact transcription wire format at its own base URL, so it rides the same provider class. Transcription-only: no TTS product.",
"base_url": "https://api.groq.com/openai/v1",
"credential_id": "groq",
+ "_gateway_comment": "Not served by the OpenCompany Gateway: LiteLLM 1.103.4 bills Groq transcription at $0 (gateway/tests/test_gateway_contract.py test_groq_transcription_is_billed).",
"stt": {
"endpoint": "/audio/transcriptions",
"translate_endpoint": "/audio/translations",
diff --git a/server/nodes/document/embedding_generator/__init__.py b/server/nodes/document/embedding_generator/__init__.py
index 72af71f9..0761cc23 100644
--- a/server/nodes/document/embedding_generator/__init__.py
+++ b/server/nodes/document/embedding_generator/__init__.py
@@ -9,6 +9,7 @@ from pydantic import BaseModel, ConfigDict, Field
from services.memory.vector_store import (
EmbeddingError,
create_embedder,
+ openai_embedding_access,
)
from services.plugin import ActionNode, NodeContext, NodeUserError, Operation, TaskQueue
@@ -99,6 +100,14 @@ class EmbeddingGeneratorNode(ActionNode):
)
texts = [c.get("content", "") if isinstance(c, dict) else str(c) for c in chunks]
+ route = None
+ if provider == "openai" and not api_key.strip():
+ # Own keys first (the install's stored key), then the gateway.
+ from services.plugin.deps import get_auth_service
+
+ api_key, route = await openai_embedding_access(
+ api_key, params.endpoint.strip(), get_auth_service()
+ )
embedder = None
try:
embedder = create_embedder(
@@ -106,6 +115,7 @@ class EmbeddingGeneratorNode(ActionNode):
model=model,
api_key=api_key,
endpoint=params.endpoint,
+ route=route,
)
embeddings = await embedder.embed_documents(
texts,
diff --git a/server/nodes/speech/_base.py b/server/nodes/speech/_base.py
index c8f9aa6d..7f8117ac 100644
--- a/server/nodes/speech/_base.py
+++ b/server/nodes/speech/_base.py
@@ -7,7 +7,7 @@ parts both nodes need and neither should own.
from __future__ import annotations
from pathlib import Path
-from typing import Any, Dict, Optional, Tuple
+from typing import TYPE_CHECKING, Any, Dict, Optional, Sequence, Tuple
from core.logging import get_logger
from services.media import coerce_file_param, read_media_bytes, resolve_media
@@ -16,28 +16,46 @@ from services.plugin import NodeContext, NodeUserError, coerce_blank_params # n
from . import _config as speech_config
+if TYPE_CHECKING:
+ from services.gateway.routing import GatewayRoute
+
logger = get_logger(__name__)
-async def provider_api_key(ctx: NodeContext, provider: str) -> str:
- """Resolve the stored key for ``provider`` through its credential.
+def gateway_serves(provider: str) -> bool:
+ """Whether the OpenCompany Gateway serves this provider's speech
+ (``"gateway": true`` in speech_defaults.json)."""
+ return speech_config.provider_block(provider).get("gateway") is True
+
+
+async def provider_key(ctx: NodeContext, provider: str) -> Tuple[str, Optional[GatewayRoute]]:
+ """Resolve the key for ``provider`` through its credential, and the
+ OpenCompany Gateway's route when the gateway serves the call.
Goes through ``ctx.connection(...)`` rather than reading the auth
service directly so a missing key raises the framework's annotated
``PermissionError`` -- which ``BaseNode.execute`` turns into a
credential envelope plus a CloudEvents broadcast, so the Credentials
modal lights up the right provider.
+
+ Own keys first: only with no key of the install's own does a provider
+ marked ``"gateway": true`` in speech_defaults.json run on the gateway.
"""
credential_id = speech_config.credential_id(provider)
- async with ctx.connection(credential_id) as conn:
- secrets = await conn.credentials()
- api_key = str(secrets.get("api_key") or "")
+ if gateway_serves(provider):
+ from services.gateway.routing import key_or_route
+
+ api_key, route = await key_or_route(ctx, credential_id, provider)
+ else:
+ async with ctx.connection(credential_id) as conn:
+ secrets = await conn.credentials()
+ api_key, route = str(secrets.get("api_key") or ""), None
if not api_key:
raise NodeUserError(
f"No API key stored for '{credential_id}'. Add one in the "
"Credentials modal."
)
- return api_key
+ return api_key, route
def require_provider(provider: str, available: Sequence[str], direction: str) -> str:
@@ -152,8 +170,9 @@ async def track_usage(
__all__ = [
"coerce_blank_params",
+ "gateway_serves",
"measure_seconds",
- "provider_api_key",
+ "provider_key",
"read_audio_input",
"require_provider",
"track_usage",
diff --git a/server/nodes/speech/_handlers.py b/server/nodes/speech/_handlers.py
index 6bb0d101..ede55a55 100644
--- a/server/nodes/speech/_handlers.py
+++ b/server/nodes/speech/_handlers.py
@@ -2,8 +2,9 @@
``dictation_status`` says whether a recording can be turned into text: the
first provider in ``speech_defaults.json``'s ``dictation.providers`` with a
-stored key (the microphone shows only when there is one; the browser's own
-speech recognition is not used, it does not work in the desktop app).
+stored key, else the first the OpenCompany Gateway serves (the microphone
+shows only when there is one; the browser's own speech recognition is not
+used, it does not work in the desktop app).
``transcribe_audio`` transcribes a recording the chat uploaded into the
workflow's workspace (``uploads/``) with that provider, records the usage,
and deletes the recording: a dictation leaves nothing behind.
@@ -27,7 +28,7 @@ from services.plugin.ws import ws_response
from . import _config as speech_config
from . import _unifier
-from ._base import track_usage
+from ._base import gateway_serves, track_usage
from ._protocol import SttRequest
from ._registry import stt_providers
@@ -42,16 +43,22 @@ def dictation_providers() -> list[str]:
async def dictation_provider(*, principal: str | None = None) -> Optional[str]:
- """The first dictation provider with a stored key, or None."""
+ """The first dictation provider with a stored key, else the first the
+ OpenCompany Gateway serves, or None."""
from core.container import container
+ from services.gateway.routing import gateway_covers
auth = container.auth_service()
- for provider in dictation_providers():
+ providers = dictation_providers()
+ for provider in providers:
try:
if await auth.has_valid_key(speech_config.credential_id(provider), principal=principal):
return provider
except Exception: # noqa: BLE001 - an unreadable key is no key
logger.warning("Dictation key could not be read", provider=provider, exc_info=True)
+ for provider in providers:
+ if gateway_serves(provider) and await gateway_covers(provider):
+ return provider
return None
@@ -109,9 +116,16 @@ async def handle_transcribe_audio(data: Dict[str, Any], websocket: WebSocket) ->
raise NodeUserError("The recording is too long to transcribe. Record a shorter one.")
try:
api_key = await container.auth_service().resolve_api_key(speech_config.credential_id(provider), principal=principal)
+ route = None
+ if not api_key and gateway_serves(provider):
+ from services.gateway.routing import gateway_route
+
+ route = await gateway_route(provider)
+ api_key = route.api_key if route is not None else ""
result = await _unifier.transcribe(
provider=provider,
api_key=str(api_key or ""),
+ route=route,
request=SttRequest(
audio=path.read_bytes(),
filename=path.name,
diff --git a/server/nodes/speech/_providers/_http.py b/server/nodes/speech/_providers/_http.py
index 3ff69322..f49882da 100644
--- a/server/nodes/speech/_providers/_http.py
+++ b/server/nodes/speech/_providers/_http.py
@@ -27,7 +27,7 @@ than an activity timeout.
from __future__ import annotations
-from typing import Any, Dict, Optional
+from typing import Any, Dict, Mapping, Optional
import httpx
@@ -57,6 +57,7 @@ class HttpSpeechProvider:
base_url: str = "",
provider_name: Optional[str] = None,
timeout: float = DEFAULT_TIMEOUT_SECONDS,
+ extra_headers: Optional[Mapping[str, str]] = None,
):
# provider_name may arrive via client_kwargs so one class can serve
# several registry entries (openai and groq share a transcription
@@ -67,6 +68,9 @@ class HttpSpeechProvider:
base_url or speech_config.base_url(self.provider_name)
).rstrip("/")
self._timeout = timeout
+ # Sent on every request, after the auth headers (the OpenCompany
+ # Gateway's attribution headers).
+ self._extra_headers = dict(extra_headers or {})
self._client: Optional[httpx.AsyncClient] = None
# ------------------------------------------------------------------
@@ -89,7 +93,7 @@ class HttpSpeechProvider:
self._client = httpx.AsyncClient(
base_url=self._base_url,
timeout=self._timeout,
- headers=self.auth_headers(),
+ headers={**self.auth_headers(), **self._extra_headers},
)
return self._client
diff --git a/server/nodes/speech/_providers/_openai_compat.py b/server/nodes/speech/_providers/_openai_compat.py
index 48122910..eea8fd5f 100644
--- a/server/nodes/speech/_providers/_openai_compat.py
+++ b/server/nodes/speech/_providers/_openai_compat.py
@@ -54,10 +54,18 @@ _PCM_SAMPLE_RATE = 24000
class OpenAISpeechProvider(HttpSpeechProvider):
- """Speech over OpenAI's ``/audio/*`` routes."""
+ """Speech over OpenAI's ``/audio/*`` routes.
+
+ ``model_prefix`` is the OpenCompany Gateway's LiteLLM prefix
+ (``openai/``, ``groq/``): it goes on the model name in the request only.
+ """
provider_name = "openai"
+ def __init__(self, api_key: str, *, model_prefix: str = "", **kwargs: Any):
+ super().__init__(api_key, **kwargs)
+ self._model_prefix = model_prefix
+
# ------------------------------------------------------------------
# synthesis
# ------------------------------------------------------------------
@@ -70,7 +78,7 @@ class OpenAISpeechProvider(HttpSpeechProvider):
body: Dict[str, Any] = {
"input": req.text,
- "model": model,
+ "model": f"{self._model_prefix}{model}",
"voice": req.voice
or speech_config.default_voice(self.provider_name)
or "alloy",
@@ -151,7 +159,7 @@ class OpenAISpeechProvider(HttpSpeechProvider):
)
form: Dict[str, Any] = {
- "model": model,
+ "model": f"{self._model_prefix}{model}",
"response_format": response_format,
}
# The translation route always outputs English, so a language hint
diff --git a/server/nodes/speech/_unifier.py b/server/nodes/speech/_unifier.py
index 2bc292c0..a3270951 100644
--- a/server/nodes/speech/_unifier.py
+++ b/server/nodes/speech/_unifier.py
@@ -20,12 +20,15 @@ was built.
from __future__ import annotations
-from typing import Awaitable, Callable, List, Optional, TypeVar
+from typing import TYPE_CHECKING, Any, Awaitable, Callable, Dict, List, Optional, TypeVar
from core.logging import get_logger
from services.plugin import NodeUserError
from services.provider_registry import ProviderSpec
+if TYPE_CHECKING:
+ from services.gateway.routing import GatewayRoute
+
from ._protocol import (
SpeechError,
SpeechErrorCategory,
@@ -48,8 +51,10 @@ async def synthesize(
api_key: str,
request: TtsRequest,
translate_errors: bool = True,
+ route: Optional[GatewayRoute] = None,
) -> TtsResult:
- """Synthesize speech via the named provider."""
+ """Synthesize speech via the named provider (through the OpenCompany
+ Gateway when ``route`` is set)."""
spec = get_tts_provider(provider)
return await _dispatch(
spec,
@@ -58,6 +63,7 @@ async def synthesize(
lambda client: client.synthesize(request),
operation="synthesis",
translate_errors=translate_errors,
+ route=route,
)
@@ -67,8 +73,10 @@ async def transcribe(
api_key: str,
request: SttRequest,
translate_errors: bool = True,
+ route: Optional[GatewayRoute] = None,
) -> SttResult:
- """Transcribe audio via the named provider."""
+ """Transcribe audio via the named provider (through the OpenCompany
+ Gateway when ``route`` is set)."""
spec = get_stt_provider(provider)
return await _dispatch(
spec,
@@ -77,6 +85,7 @@ async def transcribe(
lambda client: client.transcribe(request),
operation="transcription",
translate_errors=translate_errors,
+ route=route,
)
@@ -105,6 +114,7 @@ async def _dispatch(
*,
operation: str,
translate_errors: bool,
+ route: Optional[GatewayRoute] = None,
) -> T:
"""Build a client, run ``call``, normalize failures, always close.
@@ -112,10 +122,15 @@ async def _dispatch(
re-raises the structured :class:`SpeechError` instead of a flattened
user message, which is what a Temporal retry policy needs to read
``retryable``.
+
+ A ``route`` points the client at the OpenCompany Gateway: its URL, the
+ LiteLLM model prefix and the run's attribution headers. Only providers
+ on :class:`OpenAISpeechProvider` get one (``"gateway": true`` in
+ speech_defaults.json).
"""
client: Optional[object] = None
try:
- client = spec.factory(api_key=api_key, **spec.client_kwargs)
+ client = spec.factory(api_key=api_key, **_client_kwargs(spec, route))
return await call(client)
except spec.sdk_exception_types as exc:
error = SpeechError.from_exception(provider, exc)
@@ -130,8 +145,9 @@ async def _dispatch(
status_code=error.status_code,
provider_code=error.provider_code,
request_id=error.request_id,
+ gateway=route is not None,
)
- raise NodeUserError(error.user_message) from error
+ raise NodeUserError(_gateway_message(route, exc) or error.user_message) from error
except SpeechError as exc:
# A provider classified a failure itself -- typically a documented
# vendor error that arrived with HTTP 200, which both Deepgram and
@@ -176,6 +192,29 @@ async def _dispatch(
await _close(client)
+def _client_kwargs(spec: ProviderSpec, route: Optional[GatewayRoute]) -> Dict[str, Any]:
+ if route is None:
+ return dict(spec.client_kwargs)
+ from services.gateway.attribution import request_headers
+
+ return {
+ **spec.client_kwargs,
+ "base_url": route.base_url,
+ "model_prefix": route.model_prefix,
+ "extra_headers": request_headers(),
+ }
+
+
+def _gateway_message(route: Optional[GatewayRoute], exc: BaseException) -> Optional[str]:
+ """Gateway wording for a failed gateway call: the vendor's texts point
+ at the wrong key and billing account."""
+ if route is None:
+ return None
+ from services.gateway.errors import describe
+
+ return describe(exc)
+
+
async def _close(client: object) -> None:
"""Close a provider if it holds anything. Probed, not required."""
close = getattr(client, "aclose", None)
diff --git a/server/nodes/speech/speech_to_text.py b/server/nodes/speech/speech_to_text.py
index e447c623..b30c3666 100644
--- a/server/nodes/speech/speech_to_text.py
+++ b/server/nodes/speech/speech_to_text.py
@@ -23,6 +23,7 @@ from typing import Any, Dict, List, Optional
from pydantic import BaseModel, Field, model_validator
+from core.logging import get_logger
from services.plugin import ActionNode, NodeContext, Operation, TaskQueue
from ..model._credentials import GroqCredential, OpenAICredential, SarvamCredential
@@ -31,7 +32,7 @@ from . import _unifier
from ._base import (
coerce_blank_params,
measure_seconds,
- provider_api_key,
+ provider_key,
read_audio_input,
require_provider,
track_usage,
@@ -42,6 +43,8 @@ from ._registry import stt_providers
from . import _providers # noqa: F401 isort:skip
+logger = get_logger(__name__)
+
_PROVIDERS: List[str] = stt_providers()
@@ -177,17 +180,28 @@ class SpeechToTextNode(ActionNode):
params.audio_file, ctx, max_bytes=max_bytes
)
- api_key = await provider_api_key(ctx, provider)
+ api_key, route = await provider_key(ctx, provider)
+ translate = params.translate
+ if translate and route is not None:
+ # The OpenCompany Gateway has no translation route. Same fallback
+ # as a model that cannot translate (_openai_compat.py).
+ logger.warning(
+ "the OpenCompany Gateway cannot translate; transcribing in-language instead",
+ provider=provider,
+ model=model,
+ )
+ translate = False
result = await _unifier.transcribe(
provider=provider,
api_key=api_key,
+ route=route,
request=SttRequest(
audio=blob,
filename=filename,
mime_type=_guess_mime(filename),
model=model,
language=params.language,
- translate=params.translate,
+ translate=translate,
diarize=params.diarize,
timestamps=params.timestamps,
provider_options=dict(params.provider_options),
diff --git a/server/nodes/speech/text_to_speech.py b/server/nodes/speech/text_to_speech.py
index 18dcacd0..a4c1d54c 100644
--- a/server/nodes/speech/text_to_speech.py
+++ b/server/nodes/speech/text_to_speech.py
@@ -37,7 +37,7 @@ from . import _config as speech_config
from . import _unifier
from ._base import (
coerce_blank_params,
- provider_api_key,
+ provider_key,
require_provider,
track_usage,
)
@@ -202,10 +202,11 @@ class TextToSpeechNode(ActionNode):
self._check_length(provider, model, params.text)
- api_key = await provider_api_key(ctx, provider)
+ api_key, route = await provider_key(ctx, provider)
result = await _unifier.synthesize(
provider=provider,
api_key=api_key,
+ route=route,
request=TtsRequest(
text=params.text,
model=model,
diff --git a/server/nodes/vision/vision_analyze/__init__.py b/server/nodes/vision/vision_analyze/__init__.py
index db5c8a09..21e41bf9 100644
--- a/server/nodes/vision/vision_analyze/__init__.py
+++ b/server/nodes/vision/vision_analyze/__init__.py
@@ -14,12 +14,16 @@ models and the specialist path (OCR-style extraction).
Token cost/billing note: like the LLM-backed translate providers, usage is
deliberately NOT recorded via the pricing service — this path bills
provider tokens that the LLM pricing layer has no per-call meter for here.
+
+With no key of the install's own for the provider, the OpenCompany Gateway
+answers (``key_or_route``): the same SDK clients, pointed at the gateway
+with its key, the LiteLLM model prefix and the run's attribution headers.
"""
from __future__ import annotations
import base64
-from typing import Any, Literal, Optional
+from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, Tuple
from pydantic import BaseModel, ConfigDict, Field, model_validator
@@ -31,6 +35,11 @@ from services.plugin import (
ToolNode,
)
+from ...model._credentials import AnthropicCredential, GeminiCredential, OpenAICredential
+
+if TYPE_CHECKING:
+ from services.gateway.routing import GatewayRoute
+
VisionProvider = Literal["openai", "anthropic", "gemini"]
VisionOperation = Literal["describe", "extract_text"]
VisionBudget = Literal["small", "normal", "large"]
@@ -113,18 +122,31 @@ class VisionAnalyzeOutput(BaseModel):
model_config = ConfigDict(extra="allow")
-async def _api_key(ctx: NodeContext, provider: str) -> str:
+async def _api_key(
+ ctx: NodeContext, provider: str
+) -> Tuple[str, Optional[GatewayRoute]]:
"""Resolve the stored key through the provider's credential (the speech
- idiom) so a missing key surfaces as the annotated credential envelope."""
- async with ctx.connection(provider) as conn:
- secrets = await conn.credentials()
- api_key = str(secrets.get("api_key") or "")
+ idiom) so a missing key surfaces as the annotated credential envelope;
+ with no own key, the OpenCompany Gateway's key and route when it covers
+ the provider."""
+ from services.gateway.routing import key_or_route
+
+ api_key, route = await key_or_route(ctx, provider, provider)
if not api_key:
raise NodeUserError(
f"No API key stored for '{provider}'. Add one in the "
"Credentials modal."
)
- return api_key
+ return api_key, route
+
+
+def _gateway_client_kwargs(route: Optional[GatewayRoute]) -> Dict[str, Any]:
+ """The SDK client options that point a call at the gateway."""
+ if route is None:
+ return {}
+ from services.gateway.attribution import request_headers
+
+ return {"base_url": route.base_url, "default_headers": request_headers()}
def _default_model(provider: str) -> str:
@@ -177,6 +199,8 @@ class VisionAnalyzeNode(ToolNode):
"readonly": True,
"open_world": True, # sends image content to an external provider
}
+ # ``ctx.connection(provider)`` is a lookup over this tuple.
+ credentials = (OpenAICredential, AnthropicCredential, GeminiCredential)
task_queue = TaskQueue.AI_HEAVY
Params = VisionAnalyzeParams
@@ -236,19 +260,31 @@ class VisionAnalyzeNode(ToolNode):
if args.operation == "describe" and args.question
else _PROMPTS[args.operation]
)
- api_key = await _api_key(ctx, provider)
+ api_key, route = await _api_key(ctx, provider)
dispatch = {
"openai": self._ask_openai,
"anthropic": self._ask_anthropic,
"gemini": self._ask_gemini,
}
- text = await dispatch[provider](
- api_key=api_key,
- model=model,
- prompt=prompt,
- image_bytes=fitted,
- mime=mime,
- )
+ try:
+ text = await dispatch[provider](
+ api_key=api_key,
+ model=model,
+ prompt=prompt,
+ image_bytes=fitted,
+ mime=mime,
+ route=route,
+ )
+ except Exception as exc:
+ if route is None:
+ raise
+ from services.gateway.errors import describe
+
+ # The vendor's text would point at the wrong key and account.
+ message = describe(exc)
+ if message is None:
+ raise
+ raise NodeUserError(message) from exc
return VisionAnalyzeOutput(
operation=args.operation,
provider=provider,
@@ -261,17 +297,23 @@ class VisionAnalyzeNode(ToolNode):
@staticmethod
async def _ask_openai(
- *, api_key: str, model: str, prompt: str, image_bytes: bytes, mime: str
+ *,
+ api_key: str,
+ model: str,
+ prompt: str,
+ image_bytes: bytes,
+ mime: str,
+ route: Optional[GatewayRoute] = None,
) -> str:
from openai import AsyncOpenAI
- client = AsyncOpenAI(api_key=api_key)
+ client = AsyncOpenAI(api_key=api_key, **_gateway_client_kwargs(route))
data_url = (
f"data:{mime};base64,"
f"{base64.b64encode(image_bytes).decode('ascii')}"
)
response = await client.chat.completions.create(
- model=model,
+ model=f"{route.model_prefix}{model}" if route else model,
max_completion_tokens=_MAX_OUTPUT_TOKENS,
messages=[
{
@@ -292,11 +334,18 @@ class VisionAnalyzeNode(ToolNode):
@staticmethod
async def _ask_anthropic(
- *, api_key: str, model: str, prompt: str, image_bytes: bytes, mime: str
+ *,
+ api_key: str,
+ model: str,
+ prompt: str,
+ image_bytes: bytes,
+ mime: str,
+ route: Optional[GatewayRoute] = None,
) -> str:
from anthropic import AsyncAnthropic
- client = AsyncAnthropic(api_key=api_key)
+ # The gateway's Anthropic route takes bare model ids.
+ client = AsyncAnthropic(api_key=api_key, **_gateway_client_kwargs(route))
response = await client.messages.create(
model=model,
max_tokens=_MAX_OUTPUT_TOKENS,
@@ -327,12 +376,27 @@ class VisionAnalyzeNode(ToolNode):
@staticmethod
async def _ask_gemini(
- *, api_key: str, model: str, prompt: str, image_bytes: bytes, mime: str
+ *,
+ api_key: str,
+ model: str,
+ prompt: str,
+ image_bytes: bytes,
+ mime: str,
+ route: Optional[GatewayRoute] = None,
) -> str:
from google import genai
from google.genai import types as genai_types
- client = genai.Client(api_key=api_key)
+ gateway = _gateway_client_kwargs(route)
+ # The gateway's native Gemini route takes bare model ids.
+ http_options = (
+ genai_types.HttpOptions(
+ base_url=gateway["base_url"], headers=gateway["default_headers"]
+ )
+ if gateway
+ else None
+ )
+ client = genai.Client(api_key=api_key, http_options=http_options)
response = await client.aio.models.generate_content(
model=model,
contents=[
diff --git a/server/services/gateway/errors.py b/server/services/gateway/errors.py
index 3c73fdcf..e0851b07 100644
--- a/server/services/gateway/errors.py
+++ b/server/services/gateway/errors.py
@@ -5,38 +5,106 @@ blocked this request because a spending limit ... was exhausted"), which is
the wrong account for a call billed to an OpenCompany allowance and made
with the gateway key. ChatUnifier passes every failure of a gateway call
through :func:`explain`; the category, and so retry and pause behaviour,
-stays as classified.
+stays as classified. Embeddings, speech and vision, which call the gateway
+from their own clients, word their failures with :func:`describe`.
"""
from __future__ import annotations
-from typing import Optional
+from typing import Mapping, Optional
+
+import httpx
from services.llm.protocol import LLMError, LLMErrorCategory, error_type
-_RECONNECT = "Reconnect the OpenCompany Gateway in Settings > Connectors, then resume."
+_ALLOWANCE = "Your OpenCompany allowance for this month is used up."
+_UNPRICED = "This model isn't available on the OpenCompany Gateway yet."
+_KEY_REJECTED = "The OpenCompany Gateway rejected the gateway key."
+_DENIED = "The OpenCompany Gateway denied this request."
+_RATE_LIMITED = "The OpenCompany Gateway is rate-limiting requests. Retry after a short delay."
+_UNREACHABLE = "Could not reach the OpenCompany Gateway."
+_UNAVAILABLE = "The OpenCompany Gateway is temporarily unavailable."
+_RECONNECT = "Reconnect the OpenCompany Gateway in Settings > Connectors"
_OWN_KEY = "add your own key for this provider in Settings > Connectors"
+_RAISE = f"Ask your OpenCompany admin to raise it, or {_OWN_KEY}"
+_OTHER_MODEL = f"Choose another model, or {_OWN_KEY}."
def explain(error: LLMError, exc: Optional[BaseException]) -> LLMError:
"""Set gateway wording on ``error`` (in place) and return it."""
kind = error_type(exc) if exc is not None else None
if error.category is LLMErrorCategory.BILLING:
- error.public_message = "Your OpenCompany allowance for this month is used up."
- error.public_hint = f"Ask your OpenCompany admin to raise it, or {_OWN_KEY}, then resume."
+ error.public_message = _ALLOWANCE
+ error.public_hint = f"{_RAISE}, then resume."
elif kind == "model_cost_map_missing":
- error.public_message = "This model isn't available on the OpenCompany Gateway yet."
- error.public_hint = f"Choose another model, or {_OWN_KEY}."
+ error.public_message = _UNPRICED
+ error.public_hint = _OTHER_MODEL
elif error.category is LLMErrorCategory.AUTHENTICATION:
- error.public_message = "The OpenCompany Gateway rejected the gateway key."
- error.public_hint = _RECONNECT
+ error.public_message = _KEY_REJECTED
+ error.public_hint = f"{_RECONNECT}, then resume."
elif error.category is LLMErrorCategory.PERMISSION:
- error.public_message = "The OpenCompany Gateway denied this request."
- error.public_hint = _RECONNECT
+ error.public_message = _DENIED
+ error.public_hint = f"{_RECONNECT}, then resume."
elif error.category is LLMErrorCategory.RATE_LIMIT:
- error.public_message = "The OpenCompany Gateway is rate-limiting requests. Retry after a short delay."
+ error.public_message = _RATE_LIMITED
elif error.category in (LLMErrorCategory.CONNECTION, LLMErrorCategory.TIMEOUT):
- error.public_message = "Could not reach the OpenCompany Gateway."
+ error.public_message = _UNREACHABLE
elif error.category is LLMErrorCategory.SERVER:
- error.public_message = "The OpenCompany Gateway is temporarily unavailable."
+ error.public_message = _UNAVAILABLE
return error
+
+
+def describe(exc: BaseException) -> Optional[str]:
+ """The user-facing text for a failed gateway call made outside
+ ChatUnifier, or ``None`` to keep the caller's own wording.
+
+ Reads the error type LiteLLM puts in the body (``budget_exceeded``,
+ ``model_cost_map_missing``) from an SDK error or an ``httpx`` one, then
+ the HTTP status."""
+ kind = _body_type(exc)
+ status = _status(exc)
+ if kind == "budget_exceeded":
+ return f"{_ALLOWANCE} {_RAISE}."
+ if kind == "model_cost_map_missing":
+ return f"{_UNPRICED} {_OTHER_MODEL}"
+ if status == 401:
+ return f"{_KEY_REJECTED} {_RECONNECT}."
+ if status == 403:
+ return f"{_DENIED} {_RECONNECT}."
+ if status == 429:
+ return _RATE_LIMITED
+ if status is not None and status >= 500:
+ return _UNAVAILABLE
+ if status is None and _transport_failure(exc):
+ return _UNREACHABLE
+ return None
+
+
+def _body_type(exc: BaseException) -> Optional[str]:
+ kind = error_type(exc)
+ if kind:
+ return kind
+ try:
+ body = exc.response.json() # type: ignore[attr-defined]
+ except Exception: # noqa: BLE001 - no response, or not JSON
+ return None
+ error = body.get("error") if isinstance(body, Mapping) else None
+ value = error.get("type") if isinstance(error, Mapping) else None
+ return str(value) if value else None
+
+
+def _status(exc: BaseException) -> Optional[int]:
+ response = getattr(exc, "response", None)
+ # status_code: OpenAI and Anthropic SDK errors; response: httpx errors;
+ # code: google-genai errors.
+ for value in (getattr(exc, "status_code", None), getattr(response, "status_code", None), getattr(exc, "code", None)):
+ if isinstance(value, int) and not isinstance(value, bool):
+ return value
+ return None
+
+
+def _transport_failure(exc: BaseException) -> bool:
+ return isinstance(exc, (httpx.TransportError, OSError, TimeoutError)) or type(exc).__name__ in (
+ "APIConnectionError",
+ "APITimeoutError",
+ )
diff --git a/server/services/gateway/routing.py b/server/services/gateway/routing.py
index d95b5f5d..99253510 100644
--- a/server/services/gateway/routing.py
+++ b/server/services/gateway/routing.py
@@ -11,13 +11,15 @@ the ``"gateway"`` block of each served provider in
``server/config/llm_defaults.json``. OpenAI-compatible providers use the
gateway's ``/v1`` routes with a LiteLLM provider prefix on the model name
(``groq/...``); Anthropic and Gemini use the gateway's native routes with
-bare model ids.
+bare model ids. The same route serves a provider's embeddings, speech and
+vision calls, which go out from their own SDK clients
+(:func:`key_or_route`).
"""
from __future__ import annotations
from dataclasses import dataclass
-from typing import Any, List, Mapping, Optional
+from typing import Any, List, Mapping, Optional, Tuple
from services.llm.config import split_provider_ref
@@ -81,8 +83,8 @@ class GatewayRouter:
return await self.route(provider) is not None
-async def gateway_covers(provider: str) -> bool:
- """:meth:`GatewayRouter.covers` on the container's router, for call sites
+async def gateway_route(provider: str) -> Optional[GatewayRoute]:
+ """:meth:`GatewayRouter.route` on the container's router, for call sites
outside the DI graph. Under test the container is a MagicMock (whose
router would claim to cover everything) or a stub without a router, so
only a real :class:`GatewayRouter` is asked."""
@@ -90,4 +92,27 @@ async def gateway_covers(provider: str) -> bool:
factory = getattr(container, "gateway_router", None)
router = factory() if callable(factory) else None
- return isinstance(router, GatewayRouter) and await router.covers(provider)
+ return await router.route(provider) if isinstance(router, GatewayRouter) else None
+
+
+async def gateway_covers(provider: str) -> bool:
+ """:meth:`GatewayRouter.covers` outside the DI graph (see :func:`gateway_route`)."""
+ return await gateway_route(provider) is not None
+
+
+async def key_or_route(ctx: Any, credential_id: str, provider: str) -> Tuple[str, Optional[GatewayRoute]]:
+ """A plugin node's key: the install's own for ``credential_id``, else the
+ gateway's for ``provider``, with its route (``None`` for an own key).
+
+ The own key is read through ``ctx.connection``, so with neither key the
+ node still raises the annotated missing-key error that points the
+ Credentials modal at the provider."""
+ try:
+ async with ctx.connection(credential_id) as conn:
+ secrets = await conn.credentials()
+ except PermissionError as exc:
+ route = await gateway_route(provider) if getattr(exc, "reason", None) == "missing" else None
+ if route is None:
+ raise
+ return route.api_key, route
+ return str(secrets.get("api_key") or ""), None
diff --git a/server/services/llm/providers/gemini.py b/server/services/llm/providers/gemini.py
index 94cbd9ee..0001f1a9 100644
--- a/server/services/llm/providers/gemini.py
+++ b/server/services/llm/providers/gemini.py
@@ -84,8 +84,8 @@ class GeminiProvider:
extra_body: Optional[Mapping[str, Any]] = None,
) -> LLMResponse:
del context_management # Gemini stays on portable Context checkpoints.
- # The LLM gateway's replay cache is verified only on its OpenAI and
- # Anthropic routes, not the native Gemini one; never sent here.
+ # The LLM gateway doesn't serve its replay cache on the native Gemini
+ # route (checked on LiteLLM 1.103.4); never sent here.
del extra_body
from google.genai import types
diff --git a/server/services/memory/vector_store.py b/server/services/memory/vector_store.py
index 0e343706..52719f2b 100644
--- a/server/services/memory/vector_store.py
+++ b/server/services/memory/vector_store.py
@@ -14,12 +14,15 @@ import math
import sys
import threading
from dataclasses import dataclass, field
-from typing import Any, Dict, Iterable, List, Literal, Optional, Protocol
+from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Literal, Optional, Protocol
from uuid import uuid4
from core.encryption import fingerprint_credential
from core.logging import get_logger
+if TYPE_CHECKING:
+ from services.gateway.routing import GatewayRoute
+
logger = get_logger(__name__)
EmbeddingProvider = Literal["huggingface", "openai", "ollama"]
@@ -197,7 +200,12 @@ class SentenceTransformerEmbedder:
class OpenAIEmbedder:
- """Direct ``AsyncOpenAI.embeddings`` adapter."""
+ """Direct ``AsyncOpenAI.embeddings`` adapter.
+
+ With a ``route``, the OpenCompany Gateway serves the calls: its key and
+ URL, the LiteLLM prefix on the model name, and the attribution headers
+ of whichever run makes each call.
+ """
def __init__(
self,
@@ -205,6 +213,7 @@ class OpenAIEmbedder:
*,
api_key: Optional[str],
endpoint: Optional[str] = None,
+ route: Optional[GatewayRoute] = None,
) -> None:
try:
from openai import AsyncOpenAI
@@ -213,10 +222,19 @@ class OpenAIEmbedder:
"OpenAI embeddings unavailable. Install the 'openai' package."
) from exc
- kwargs: Dict[str, Any] = {"api_key": api_key or None}
+ if route is not None:
+ api_key, endpoint = route.api_key, route.base_url
+ if not api_key:
+ # AsyncOpenAI would fall back to OPENAI_API_KEY from the
+ # environment and send it wherever ``endpoint`` points.
+ raise EmbedderUnavailableError(
+ "OpenAI API key is required for embeddings."
+ )
+ kwargs: Dict[str, Any] = {"api_key": api_key}
if endpoint:
kwargs["base_url"] = endpoint
self.model_name = model_name
+ self._route = route
try:
self._client = AsyncOpenAI(**kwargs)
except Exception as exc:
@@ -225,6 +243,26 @@ class OpenAIEmbedder:
f"({type(exc).__name__})"
) from exc
+ def _request(self, batch: List[str]) -> Dict[str, Any]:
+ if self._route is None:
+ return {"model": self.model_name, "input": batch}
+ from services.gateway.attribution import request_headers
+
+ return {
+ "model": f"{self._route.model_prefix}{self.model_name}",
+ "input": batch,
+ "extra_headers": request_headers(),
+ }
+
+ def _failure(self, exc: Exception) -> str:
+ if self._route is not None:
+ from services.gateway.errors import describe
+
+ message = describe(exc)
+ if message:
+ return message
+ return f"OpenAI embedding request failed ({type(exc).__name__})"
+
async def embed_documents(
self,
texts: List[str],
@@ -236,14 +274,10 @@ class OpenAIEmbedder:
batch = texts[start : start + batch_size]
try:
response = await self._client.embeddings.create(
- model=self.model_name,
- input=batch,
+ **self._request(batch)
)
except Exception as exc:
- raise EmbeddingProviderError(
- "OpenAI embedding request failed "
- f"({type(exc).__name__})"
- ) from exc
+ raise EmbeddingProviderError(self._failure(exc)) from exc
data = sorted(response.data, key=lambda item: int(item.index))
vectors = [
[float(value) for value in item.embedding]
@@ -342,8 +376,13 @@ def create_embedder(
model: Optional[str] = None,
api_key: Optional[str] = None,
endpoint: Optional[str] = None,
+ route: Optional[GatewayRoute] = None,
) -> AsyncEmbedder:
- """Build a direct-SDK embedder without making a network request."""
+ """Build a direct-SDK embedder without making a network request.
+
+ ``route`` (OpenAI only) sends the calls through the OpenCompany Gateway;
+ see :func:`openai_embedding_access`.
+ """
normalized = _normalize_provider(provider)
model_name = str(model or "").strip() or default_embedding_model(normalized)
@@ -356,7 +395,7 @@ def create_embedder(
)
return SentenceTransformerEmbedder(model_name)
if normalized == "openai":
- if not str(api_key or "").strip():
+ if route is None and not str(api_key or "").strip():
raise EmbedderUnavailableError(
"OpenAI API key is required for embeddings."
)
@@ -364,6 +403,7 @@ def create_embedder(
model_name,
api_key=api_key,
endpoint=normalized_endpoint or None,
+ route=route,
)
return OllamaEmbedder(
model_name,
@@ -371,6 +411,33 @@ def create_embedder(
)
+async def openai_embedding_access(
+ api_key: Optional[str],
+ endpoint: str,
+ auth_service: Any,
+) -> tuple[Optional[str], Optional[GatewayRoute]]:
+ """The key for OpenAI embeddings, and the gateway route when it is the
+ OpenCompany Gateway's.
+
+ Own keys first: the given key, else the install's stored OpenAI key.
+ Without either, and only on OpenAI's own endpoint (an ``endpoint``
+ override names a server of the user's), the gateway serves the calls
+ when it covers OpenAI.
+ """
+
+ key = str(api_key or "").strip() or None
+ if key is None and auth_service is not None:
+ stored = await auth_service.resolve_api_key("openai", "default")
+ key = str(stored or "").strip() or None
+ if key is None and not endpoint:
+ from services.gateway.routing import gateway_route
+
+ route = await gateway_route("openai")
+ if route is not None:
+ return route.api_key, route
+ return key, None
+
+
class NativeMemoryVectorStore:
"""Concurrency-safe async cosine index with the historical tiny API."""
@@ -508,26 +575,28 @@ async def get_memory_vector_store(
model_name = str(model or "").strip() or default_embedding_model(normalized)
normalized_endpoint = _normalize_endpoint(endpoint)
resolved_key = str(api_key or "").strip() or None
- if normalized == "openai" and resolved_key is None:
- if auth_service is not None:
- try:
- auth_key = await auth_service.resolve_api_key(
- "openai",
- "default",
- )
- resolved_key = str(auth_key or "").strip() or None
- except Exception as exc: # noqa: BLE001 - best-effort memory
- logger.warning(
- "[Memory] Could not resolve OpenAI embedding credential: %s",
- exc,
- )
- return None
+ route: Optional[GatewayRoute] = None
+ if normalized == "openai":
+ try:
+ resolved_key, route = await openai_embedding_access(
+ resolved_key,
+ normalized_endpoint,
+ auth_service,
+ )
+ except Exception as exc: # noqa: BLE001 - best-effort memory
+ logger.warning(
+ "[Memory] Could not resolve OpenAI embedding credential: %s",
+ exc,
+ )
+ return None
if not resolved_key:
logger.warning(
"[Memory] OpenAI long-term memory is disabled because no "
"OpenAI credential is configured"
)
return None
+ if route is not None:
+ normalized_endpoint = route.base_url
cache_key = MemoryVectorStoreKey(
session_id=str(session_id),
@@ -546,6 +615,7 @@ async def get_memory_vector_store(
model=model_name,
api_key=resolved_key,
endpoint=normalized_endpoint,
+ route=route,
)
except EmbedderUnavailableError as exc:
logger.warning("[Memory] Vector store not available: %s", exc)
diff --git a/server/tests/gateway/test_embeddings_route.py b/server/tests/gateway/test_embeddings_route.py
new file mode 100644
index 00000000..1a70e4ae
--- /dev/null
+++ b/server/tests/gateway/test_embeddings_route.py
@@ -0,0 +1,113 @@
+"""OpenAI embeddings run on the gateway only with no own key and on OpenAI's
+own endpoint (services/memory/vector_store.py, the Embedding Generator)."""
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+import structlog
+
+from services.gateway.routing import GatewayRoute
+from services.memory import vector_store
+from services.plugin import NodeContext
+
+from .test_errors import _gateway_error
+
+ROUTE = GatewayRoute(api_key="sk-gateway", base_url="https://gw.example/v1", model_prefix="openai/")
+
+
+def _auth(stored=None):
+ auth = MagicMock()
+ auth.resolve_api_key = AsyncMock(return_value=stored)
+ return auth
+
+
+def gateway(monkeypatch):
+ route = AsyncMock(return_value=ROUTE)
+ monkeypatch.setattr("services.gateway.routing.gateway_route", route)
+ return route
+
+
+def openai_sdk(monkeypatch):
+ """A fake AsyncOpenAI: records the client options and every request."""
+ seen = SimpleNamespace(client={}, calls=[], error=None)
+
+ class FakeEmbeddings:
+ async def create(self, **kwargs):
+ seen.calls.append(kwargs)
+ if seen.error is not None:
+ raise seen.error
+ return SimpleNamespace(
+ data=[SimpleNamespace(index=i, embedding=[0.1, 0.2]) for i, _ in enumerate(kwargs["input"])]
+ )
+
+ class FakeClient:
+ def __init__(self, **kwargs):
+ seen.client = kwargs
+ self.embeddings = FakeEmbeddings()
+
+ monkeypatch.setattr("openai.AsyncOpenAI", FakeClient)
+ return seen
+
+
+async def test_own_keys_come_first(gateway):
+ access = vector_store.openai_embedding_access
+ assert await access("sk-given", "", _auth("sk-stored")) == ("sk-given", None)
+ assert await access(None, "", _auth("sk-stored")) == ("sk-stored", None)
+ gateway.assert_not_awaited()
+
+
+async def test_without_an_own_key_the_gateway_serves(gateway):
+ assert await vector_store.openai_embedding_access(None, "", _auth()) == ("sk-gateway", ROUTE)
+
+
+async def test_an_endpoint_override_never_gets_the_gateway_key(gateway):
+ assert await vector_store.openai_embedding_access(None, "http://localhost:8000/v1", _auth()) == (None, None)
+ gateway.assert_not_awaited()
+
+
+async def test_gateway_calls_carry_the_prefix_and_the_runs_attribution(openai_sdk):
+ embedder = vector_store.create_embedder("openai", route=ROUTE)
+ with structlog.contextvars.bound_contextvars(chat_run_id="run-1"):
+ assert await embedder.embed_query("hello") == [0.1, 0.2]
+ assert openai_sdk.client == {"api_key": "sk-gateway", "base_url": "https://gw.example/v1"}
+ assert openai_sdk.calls[0]["model"] == "openai/text-embedding-3-small"
+ assert openai_sdk.calls[0]["extra_headers"]["x-litellm-session-id"] == "run-1"
+
+
+async def test_a_spent_allowance_is_worded_as_the_opencompany_allowance(openai_sdk):
+ openai_sdk.error = _gateway_error(429, "budget_exceeded")
+ embedder = vector_store.create_embedder("openai", route=ROUTE)
+ with pytest.raises(vector_store.EmbeddingProviderError, match="Your OpenCompany allowance"):
+ await embedder.embed_query("hello")
+
+
+def test_an_embedder_never_falls_back_to_the_environment_key(monkeypatch):
+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env")
+ with pytest.raises(vector_store.EmbedderUnavailableError):
+ vector_store.OpenAIEmbedder("text-embedding-3-small", api_key=None)
+
+
+async def test_memory_builds_a_gateway_store_with_no_own_key(gateway, openai_sdk):
+ store = await vector_store.get_memory_vector_store("gw-session", provider="openai", auth_service=_auth())
+ try:
+ assert store is not None
+ assert openai_sdk.client["base_url"] == "https://gw.example/v1"
+ (key,) = [k for k in vector_store._memory_vector_stores if getattr(k, "session_id", None) == "gw-session"]
+ assert key.endpoint == "https://gw.example/v1"
+ finally:
+ await vector_store.clear_memory_vector_stores("gw-session")
+
+
+async def test_the_embedding_node_runs_on_the_gateway_without_a_key(gateway, openai_sdk, monkeypatch):
+ from nodes.document.embedding_generator import EmbeddingGeneratorNode, EmbeddingGeneratorParams
+
+ monkeypatch.setattr("services.plugin.deps.get_auth_service", lambda: _auth())
+ out = await EmbeddingGeneratorNode().embed(
+ NodeContext(node_id="embed-1", node_type="embeddingGenerator"),
+ EmbeddingGeneratorParams(provider="openai", model="text-embedding-3-small", chunks=[{"content": "hi"}]),
+ )
+ assert out.embedding_count == 1
+ assert openai_sdk.calls[0]["model"] == "openai/text-embedding-3-small"
diff --git a/server/tests/gateway/test_errors.py b/server/tests/gateway/test_errors.py
index 9ae13cd1..452f9293 100644
--- a/server/tests/gateway/test_errors.py
+++ b/server/tests/gateway/test_errors.py
@@ -5,7 +5,7 @@ import httpx
import openai
import pytest
-from services.gateway.errors import explain
+from services.gateway.errors import describe, explain
from services.llm.protocol import LLMError, LLMErrorCategory, error_type
@@ -73,3 +73,40 @@ def test_public_hint_replaces_the_category_hint():
error = LLMError(message="x", provider="openai", category=LLMErrorCategory.BILLING, public_hint="custom")
assert error.hint == "custom"
assert LLMError(message="x", provider="openai", category=LLMErrorCategory.BILLING).hint.startswith("Review the provider")
+
+
+def _http_error(status: int, error_type_: str) -> httpx.HTTPStatusError:
+ """What a speech provider's ``raise_for_status`` raises for a LiteLLM error body."""
+ request = httpx.Request("POST", "https://gw.example/v1/audio/speech")
+ response = httpx.Response(
+ status, request=request, json={"error": {"message": "refused", "type": error_type_, "code": str(status)}}
+ )
+ return httpx.HTTPStatusError("refused", request=request, response=response)
+
+
[email protected]("make", [_gateway_error, _http_error], ids=["sdk", "httpx"])
+def test_described_failures_read_the_body_type_of_sdk_and_httpx_errors(make):
+ """Embeddings and vision raise SDK errors; speech raises httpx ones."""
+ assert describe(make(429, "budget_exceeded")).startswith("Your OpenCompany allowance for this month is used up.")
+ assert describe(make(403, "model_cost_map_missing")).startswith("This model isn't available")
+
+
+ "status, expected",
+ [
+ (401, "The OpenCompany Gateway rejected the gateway key. Reconnect"),
+ (403, "The OpenCompany Gateway denied this request. Reconnect"),
+ (429, "The OpenCompany Gateway is rate-limiting requests."),
+ (503, "The OpenCompany Gateway is temporarily unavailable."),
+ ],
+)
+def test_described_failures_fall_back_to_the_status(status, expected):
+ assert describe(_http_error(status, "other")).startswith(expected)
+
+
+def test_an_unreachable_gateway_and_a_rejected_request_are_told_apart():
+ request = httpx.Request("POST", "https://gw.example/v1/embeddings")
+ assert describe(httpx.ConnectError("refused", request=request)) == "Could not reach the OpenCompany Gateway."
+ # A request the gateway rejected for its content keeps the caller's wording.
+ assert describe(_http_error(400, "invalid_request_error")) is None
+ assert describe(ValueError("bad")) is None
diff --git a/server/tests/gateway/test_routing.py b/server/tests/gateway/test_routing.py
index c27115e7..a39f9da9 100644
--- a/server/tests/gateway/test_routing.py
+++ b/server/tests/gateway/test_routing.py
@@ -1,12 +1,13 @@
"""services/gateway/routing.py: own keys first; the gateway serves only the
providers llm_defaults.json marks, and only when enabled and connected."""
+from contextlib import asynccontextmanager
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
-from services.gateway.routing import GATEWAY_CREDENTIAL_ID, GatewayRoute, GatewayRouter, gateway_covers
+from services.gateway.routing import GATEWAY_CREDENTIAL_ID, GatewayRoute, GatewayRouter, gateway_covers, key_or_route
DEFAULTS = {
"providers": {
@@ -85,6 +86,54 @@ async def test_gateway_covers_trusts_only_a_real_router():
assert await gateway_covers("openai") is False
+ROUTE = GatewayRoute(api_key="sk-gateway", base_url="https://gw.example/v1", model_prefix="openai/")
+
+
+def _node_ctx(*, key=None, reason="missing"):
+ """A NodeContext stand-in whose connection yields ``key`` or raises the
+ annotated PermissionError ApiKeyCredential.resolve raises."""
+
+ async def credentials():
+ if key:
+ return {"api_key": key}
+ error = PermissionError("No API key for 'openai'.")
+ error.reason = reason
+ raise error
+
+ @asynccontextmanager
+ async def connection(credential_id):
+ yield SimpleNamespace(credentials=credentials)
+
+ return SimpleNamespace(connection=connection)
+
+
+async def test_a_node_uses_its_own_key_without_asking_the_gateway(monkeypatch):
+ route = AsyncMock(return_value=ROUTE)
+ monkeypatch.setattr("services.gateway.routing.gateway_route", route)
+ assert await key_or_route(_node_ctx(key="sk-own"), "openai", "openai") == ("sk-own", None)
+ route.assert_not_awaited()
+
+
+async def test_a_node_without_its_own_key_gets_the_gateway_key_and_route(monkeypatch):
+ monkeypatch.setattr("services.gateway.routing.gateway_route", AsyncMock(return_value=ROUTE))
+ assert await key_or_route(_node_ctx(), "openai", "openai") == ("sk-gateway", ROUTE)
+
+
+async def test_a_node_without_any_key_still_raises_the_missing_key_error(monkeypatch):
+ monkeypatch.setattr("services.gateway.routing.gateway_route", AsyncMock(return_value=None))
+ with pytest.raises(PermissionError) as raised:
+ await key_or_route(_node_ctx(), "openai", "openai")
+ assert raised.value.reason == "missing"
+
+
+async def test_only_a_missing_key_falls_back_to_the_gateway(monkeypatch):
+ route = AsyncMock(return_value=ROUTE)
+ monkeypatch.setattr("services.gateway.routing.gateway_route", route)
+ with pytest.raises(PermissionError):
+ await key_or_route(_node_ctx(reason="expired"), "openai", "openai")
+ route.assert_not_awaited()
+
+
async def test_activity_key_lookup_returns_no_key_when_the_gateway_covers(monkeypatch):
"""The gateway key never reaches activity code or a Temporal payload:
the activity passes no key and ChatUnifier routes the call."""
diff --git a/server/tests/gateway/test_speech_route.py b/server/tests/gateway/test_speech_route.py
new file mode 100644
index 00000000..3db1696f
--- /dev/null
+++ b/server/tests/gateway/test_speech_route.py
@@ -0,0 +1,140 @@
+"""Speech runs on the gateway only for providers speech_defaults.json marks
+("gateway": true), and only with no key of the install's own (nodes/speech)."""
+
+import io
+import json
+import struct
+import wave
+from contextlib import contextmanager
+from types import SimpleNamespace
+from unittest.mock import MagicMock
+
+import httpx
+import respx
+from structlog.contextvars import bound_contextvars
+
+from nodes.speech import _handlers
+from nodes.speech._base import gateway_serves
+from services.gateway.routing import GatewayRouter
+from services.llm.config import LLM_DEFAULTS
+from tests.nodes._mocks import patched_container, patched_pricing
+
+GATEWAY = "https://gw.example"
+
+
+@contextmanager
+def gateway_container(keys):
+ """The node test container with a real, enabled GatewayRouter."""
+ with patched_container(auth_api_keys=keys) as container, patched_pricing():
+ settings = SimpleNamespace(gateway_enabled=True, distributed_mode=False, gateway_url=GATEWAY)
+ router = GatewayRouter(settings=settings, auth_service=container.auth_service(), defaults=LLM_DEFAULTS)
+ container.gateway_router = MagicMock(return_value=router)
+ yield container
+
+
+def _clip(harness, tmp_path):
+ buffer = io.BytesIO()
+ with wave.open(buffer, "wb") as handle:
+ handle.setnchannels(1)
+ handle.setsampwidth(2)
+ handle.setframerate(8000)
+ handle.writeframes(struct.pack("<h", 0) * 4000)
+ (tmp_path / "a.wav").write_bytes(buffer.getvalue())
+ return harness.build_context(workspace_dir=str(tmp_path))
+
+
+def test_only_openai_speech_is_served():
+ """Groq's would bill $0 on LiteLLM 1.103.4 (gateway/tests/test_gateway_contract.py)."""
+ assert gateway_serves("openai")
+ assert not any(gateway_serves(p) for p in ("groq", "elevenlabs", "deepgram", "sarvam"))
+
+
+async def test_speech_without_an_own_key_runs_on_the_gateway(harness, tmp_path):
+ route = respx.post(f"{GATEWAY}/v1/audio/speech").mock(return_value=httpx.Response(200, content=b"ID3audio"))
+ # BaseNode.execute binds the run's ids, but tests/conftest.py stubs
+ # log_context out, so the test binds them.
+ with gateway_container({"opencompany_gateway": "sk-gateway"}), bound_contextvars(workflow_id="wf-1"):
+ result = await harness.execute(
+ "textToSpeech",
+ {"provider": "openai", "text": "hello"},
+ context=harness.build_context(workspace_dir=str(tmp_path)),
+ )
+ harness.assert_envelope(result, success=True)
+ request = route.calls.last.request
+ assert request.headers["authorization"] == "Bearer sk-gateway"
+ assert json.loads(request.content)["model"] == "openai/gpt-4o-mini-tts"
+ assert "workflow:wf-1" in request.headers["x-litellm-tags"]
+ # The prefix stays on the wire.
+ assert result["result"]["tts_model"] == "gpt-4o-mini-tts"
+
+
+async def test_an_own_key_wins_over_the_gateway(harness, tmp_path):
+ route = respx.post("https://api.openai.com/v1/audio/speech").mock(
+ return_value=httpx.Response(200, content=b"ID3audio")
+ )
+ with gateway_container({"openai": "sk-own", "opencompany_gateway": "sk-gateway"}):
+ result = await harness.execute(
+ "textToSpeech",
+ {"provider": "openai", "text": "hello"},
+ context=harness.build_context(workspace_dir=str(tmp_path)),
+ )
+ harness.assert_envelope(result, success=True)
+ request = route.calls.last.request
+ assert request.headers["authorization"] == "Bearer sk-own"
+ assert json.loads(request.content)["model"] == "gpt-4o-mini-tts"
+ assert "x-litellm-tags" not in request.headers
+
+
+async def test_gateway_transcription_never_asks_for_a_translation(harness, tmp_path):
+ """LiteLLM has no /audio/translations route: translate falls back to
+ in-language transcription, as for a model that cannot translate."""
+ route = respx.post(f"{GATEWAY}/v1/audio/transcriptions").mock(
+ return_value=httpx.Response(200, json={"text": "hi", "duration": 3.0})
+ )
+ with gateway_container({"opencompany_gateway": "sk-gateway"}):
+ result = await harness.execute(
+ "speechToText",
+ {"provider": "openai", "audio_file": "a.wav", "stt_model": "whisper-1", "translate": True},
+ context=_clip(harness, tmp_path),
+ )
+ harness.assert_envelope(result, success=True)
+ assert b'name="model"\r\n\r\nopenai/whisper-1\r\n' in route.calls.last.request.content
+
+
+async def test_a_spent_allowance_is_worded_as_the_opencompany_allowance(harness, tmp_path):
+ respx.post(f"{GATEWAY}/v1/audio/speech").mock(
+ return_value=httpx.Response(429, json={"error": {"type": "budget_exceeded", "message": "Budget has been exceeded!"}})
+ )
+ with gateway_container({"opencompany_gateway": "sk-gateway"}):
+ result = await harness.execute(
+ "textToSpeech",
+ {"provider": "openai", "text": "hello"},
+ context=harness.build_context(workspace_dir=str(tmp_path)),
+ )
+ harness.assert_envelope(result, success=False)
+ assert "Your OpenCompany allowance for this month is used up." in result["error"]
+
+
+async def test_unmarked_providers_never_run_on_the_gateway(harness, tmp_path):
+ """Groq serves chat on the gateway, but not speech: no own key is still no key."""
+ route = respx.post(f"{GATEWAY}/v1/audio/transcriptions").mock(return_value=httpx.Response(200, json={"text": "hi"}))
+ with gateway_container({"opencompany_gateway": "sk-gateway"}):
+ result = await harness.execute(
+ "speechToText", {"provider": "groq", "audio_file": "a.wav"}, context=_clip(harness, tmp_path)
+ )
+ harness.assert_envelope(result, success=False)
+ assert not route.called
+
+
+async def test_dictation_falls_back_to_the_gateway_after_every_own_key():
+ with gateway_container({"deepgram": "dg-own", "opencompany_gateway": "sk-gateway"}):
+ assert await _handlers.dictation_provider() == "deepgram"
+ with gateway_container({"opencompany_gateway": "sk-gateway"}):
+ assert await _handlers.dictation_provider() == "openai"
+ with gateway_container({}):
+ assert await _handlers.dictation_provider() is None
diff --git a/server/tests/gateway/test_vision_route.py b/server/tests/gateway/test_vision_route.py
new file mode 100644
index 00000000..67fa7d19
--- /dev/null
+++ b/server/tests/gateway/test_vision_route.py
@@ -0,0 +1,128 @@
+"""The vision tool runs on the gateway with no own key for its provider
+(nodes/vision/vision_analyze)."""
+
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+import structlog
+from PIL import Image
+
+from nodes.vision.vision_analyze import VisionAnalyzeNode, VisionAnalyzeParams, VisionAnalyzeToolInput
+from services.gateway.routing import GatewayRoute
+from services.plugin import NodeContext, NodeUserError
+from services.plugin.base import _make_connection_factory
+
+from .test_errors import _gateway_error
+
+ROUTES = {
+ "openai": GatewayRoute(api_key="sk-gateway", base_url="https://gw.example/v1", model_prefix="openai/"),
+ "anthropic": GatewayRoute(api_key="sk-gateway", base_url="https://gw.example", model_prefix=""),
+ "gemini": GatewayRoute(api_key="sk-gateway", base_url="https://gw.example", model_prefix=""),
+}
+
+
+def _ctx(tmp_path: Path, provider: str) -> NodeContext:
+ workspace = tmp_path / "workspace"
+ workspace.mkdir(parents=True, exist_ok=True)
+ Image.new("RGB", (64, 48), color=(200, 40, 40)).save(workspace / "chart.png")
+ ctx = NodeContext(node_id="vision-1", node_type="visionAnalyze", workflow_id="wf-vision", workspace_dir=str(workspace))
+ ctx.raw["_tool_config"] = VisionAnalyzeParams(provider=provider, vision_model="vision-model")
+ return ctx
+
+
+def gateway_key(monkeypatch):
+ async def key_or_route(ctx, credential_id, provider):
+ return "sk-gateway", ROUTES[provider]
+
+ monkeypatch.setattr("services.gateway.routing.key_or_route", key_or_route)
+
+
+async def _ask(tmp_path, provider):
+ with structlog.contextvars.bound_contextvars(workflow_id="wf-vision"):
+ return await VisionAnalyzeNode().vision(
+ _ctx(tmp_path, provider), VisionAnalyzeToolInput(operation="describe", image="chart.png")
+ )
+
+
+def test_every_provider_key_resolves_through_a_declared_credential():
+ """It never declared any, so every real run failed at ctx.connection."""
+ factory = _make_connection_factory(VisionAnalyzeNode, {})
+ for provider in ("openai", "anthropic", "gemini"):
+ assert factory(provider).credential_id == provider
+
+
+async def test_openai_vision_on_the_gateway(tmp_path, monkeypatch, gateway_key):
+ seen = {}
+
+ class FakeCompletions:
+ async def create(self, **kwargs):
+ seen["model"] = kwargs["model"]
+ return SimpleNamespace(choices=[SimpleNamespace(message=SimpleNamespace(content="a red chart"))])
+
+ class FakeClient:
+ def __init__(self, **kwargs):
+ seen["client"] = kwargs
+ self.chat = SimpleNamespace(completions=FakeCompletions())
+
+ monkeypatch.setattr("openai.AsyncOpenAI", FakeClient)
+ result = await _ask(tmp_path, "openai")
+ assert result.text == "a red chart"
+ assert seen["model"] == "openai/vision-model"
+ assert (seen["client"]["api_key"], seen["client"]["base_url"]) == ("sk-gateway", "https://gw.example/v1")
+ assert "workflow:wf-vision" in seen["client"]["default_headers"]["x-litellm-tags"]
+ assert result.vision_model == "vision-model"
+
+
+async def test_anthropic_vision_on_the_gateway_keeps_bare_ids(tmp_path, monkeypatch, gateway_key):
+ seen = {}
+
+ class FakeMessages:
+ async def create(self, **kwargs):
+ seen["model"] = kwargs["model"]
+ return SimpleNamespace(content=[SimpleNamespace(type="text", text="chart")])
+
+ class FakeClient:
+ def __init__(self, **kwargs):
+ seen["client"] = kwargs
+ self.messages = FakeMessages()
+
+ monkeypatch.setattr("anthropic.AsyncAnthropic", FakeClient)
+ assert (await _ask(tmp_path, "anthropic")).text == "chart"
+ assert seen["model"] == "vision-model"
+ assert seen["client"]["base_url"] == "https://gw.example"
+
+
+async def test_gemini_vision_on_the_gateway_uses_the_native_route(tmp_path, monkeypatch, gateway_key):
+ seen = {}
+
+ class FakeModels:
+ async def generate_content(self, **kwargs):
+ seen["model"] = kwargs["model"]
+ return SimpleNamespace(text="gemini sees a chart")
+
+ class FakeClient:
+ def __init__(self, api_key, http_options=None):
+ seen["http_options"] = http_options
+ self.aio = SimpleNamespace(models=FakeModels())
+
+ monkeypatch.setattr("google.genai.Client", FakeClient)
+ assert (await _ask(tmp_path, "gemini")).text == "gemini sees a chart"
+ assert seen["model"] == "vision-model"
+ assert seen["http_options"].base_url == "https://gw.example"
+ assert "x-litellm-tags" in seen["http_options"].headers
+
+
+async def test_a_spent_allowance_is_worded_as_the_opencompany_allowance(tmp_path, monkeypatch, gateway_key):
+ class FakeCompletions:
+ async def create(self, **kwargs):
+ raise _gateway_error(429, "budget_exceeded")
+
+ class FakeClient:
+ def __init__(self, **kwargs):
+ self.chat = SimpleNamespace(completions=FakeCompletions())
+
+ monkeypatch.setattr("openai.AsyncOpenAI", FakeClient)
+ with pytest.raises(NodeUserError, match="Your OpenCompany allowance"):
+ await _ask(tmp_path, "openai")
diff --git a/server/tests/nodes/test_vision_analyze.py b/server/tests/nodes/test_vision_analyze.py
index 958b08da..ca9d5b2c 100644
--- a/server/tests/nodes/test_vision_analyze.py
+++ b/server/tests/nodes/test_vision_analyze.py
@@ -128,7 +128,7 @@ def _ctx(tmp_path: Path) -> NodeContext:
@pytest.fixture
def no_api_key_lookup(monkeypatch):
async def fake_key(ctx, provider):
- return "test-key"
+ return "test-key", None
monkeypatch.setattr("nodes.vision.vision_analyze._api_key", fake_key)
@@ -229,8 +229,9 @@ class TestProviderRequestShapes:
return SimpleNamespace(text="gemini sees a chart")
class FakeClient:
- def __init__(self, api_key):
+ def __init__(self, api_key, http_options=None):
captured["api_key"] = api_key
+ captured["http_options"] = http_options
self.aio = SimpleNamespace(models=FakeModels())
monkeypatch.setattr("google.genai.Client", FakeClient)
@@ -249,6 +250,7 @@ class TestProviderRequestShapes:
)
assert result.text == "gemini sees a chart"
assert captured["model"] == "gemini-3.6-flash"
+ assert captured["http_options"] is None
# Custom question wins over the canned prompt.
assert captured["contents"][1] == "What color is it?"