Self-improving AI Employees for everyone
LLM Gateway (LiteLLM)
OpenCompany's hosted LLM gateway is the stock open-source LiteLLM Proxy, deployed from gateway/ with BerriAI's official Terraform module. Invited people call LLMs on platform-owned vendor keys, each with a monthly allowance, and spend is recorded per person and per session. The repository holds only configuration and tests; there is no custom gateway code. The operator runbook (deployment, adding people, upgrades) is gateway/README.md.
Status: the gateway is in gateway/; OpenCompany's side is server/services/gateway/ and covers chat through ChatUnifier (agents, chat runs, employee setup, translate), OpenAI embeddings (long-term memory, the Embedding Generator), OpenAI speech (Text to Speech, Speech to Text, chat dictation), the vision tool, the RLM agent and Claude Code. Codex and the Gemini CLI stay on their own logins. gateway/tests/test_gateway_contract.py pins the contract below against the local stack; server/tests/gateway/ covers OpenCompany's side.
OpenCompany's side
On by default.
OPENCOMPANY_GATEWAY_ENABLED(gateway_enabled, defaulttrue) lists the Gateway card, by OpenCompany, under AI models in Connectors. It is gated byrequires_settinginserver/config/credential_providers.json, honoured byCredentialRegistry.get_all_providers.- Where it appears: in normal mode, the Get Started guide's Connect an AI model step and Connectors; in dev mode, API Credentials. Both shells use the same
CredentialsBrowser. OPENCOMPANY_GATEWAY_URL(gateway_url) points at the gateway.falsehides the card and turns the routing off, so behaviour is exactly as without the gateway.- No 1Password: the router reads the key on every call, which a 1Password lookup can't serve. So the card offers no 1Password source (
catalogue_extras), and distributed mode keeps the gateway off.
The key. One gateway key per install, saved like any API key (OpenCompanyGatewayCredential, services/gateway/credential.py, default validate). Its probe calls the gateway's /user/info with the key: 401 means a wrong key; 403 means an LLM-API key, and the person is told to create a Default one. The card's ws.status hook (gateway_status, services/gateway/handlers.py) returns the allowance as one line, which the API-key panel shows under the key (statusSummary in useCredentialPanel).
Routing: own keys first (services/gateway/routing.py). A call whose provider has a key of the install's own goes to that vendor as always. Otherwise, if the gateway is enabled, its key is stored and the provider has a "gateway" block in server/config/llm_defaults.json, ChatUnifier.chat builds the client with the gateway key and base URL (instead of the {ref}_proxy row) and passes the block's model_prefix to OpenAIProvider, which applies it to the wire model only. The gateway key stays inside the router and ChatUnifier: callers pass no key (_resolve_activity_api_key returns ""), and it never enters node parameters or Temporal payloads. ChatUnifier.fetch_models never routes (it validates a key against its own vendor); AIService.fetch_models lists a gateway-covered provider's curated models with no network call. The checks that used to refuse "no key configured" accept gateway coverage (GatewayRouter.covers, or gateway_covers outside the DI graph): the three in services/ai.py, _resolve_activity_api_key and prepare_agent_payload, Connections.ai_providers (employee setup and the chat model picker), the global model list, and translate.
Default model (gateway_default, the "gateway" entry in server/config/llm_defaults.json). Where the owner has set no global model, an install that relies on the gateway uses Gemini's default_model, Gemini 3.8 Flash:
- chat Auto (
services/chat/choice.py); - employee setup, before the first usable provider (
services/employees/llm.py); - the global model the editor shows (
get_validated_ai_providers).
It applies only while the gateway serves Gemini, and the owner's own choice always wins.
Embeddings, speech and vision call the gateway from their own clients, with the same rule and the provider's same "gateway" block: the gateway's key, its URL plus the block's base_path, the model_prefix on the wire model only, and the attribution headers built per call.
- The key.
key_or_route(ctx, credential_id, provider)gives a plugin node its key: its own throughctx.connection, else the gateway's with the route. With neither, the annotated missing-key error still reaches the Credentials modal. - Embeddings (
services/memory/vector_store.py):openai_embedding_accesstries the given key, then the stored OpenAI key, then the gateway. It routes only on OpenAI's own endpoint: anendpointoverride names a server of the user's, which never gets the gateway key.OpenAIEmbeddernever falls back toOPENAI_API_KEYfrom the environment. - Speech (
nodes/speech/): only providers marked"gateway": trueinspeech_defaults.jsonare routed (provider_key,gateway_serves). That is OpenAI, both directions; Groq is not, because 1.103.4 bills its transcription at $0. The gateway has no/audio/translationsroute, so Translate falls back to in-language transcription there. Dictation takes the first provider with an own key, else the first the gateway serves. - Vision (
nodes/vision/vision_analyze): routes all three SDK clients. The node now declares its OpenAI, Anthropic and Gemini credentials. It never did, so every real run failed atctx.connection. - Error wording. Failures of these calls get gateway wording from
describe(services/gateway/errors.py), which reads LiteLLM's error type from an SDK orhttpxerror.
The RLM agent (services/rlm/) uses the rlms library, which reaches a custom URL only in OpenAI's format.
- Route: with no own key,
BackendAdaptergives it the gateway's/v1route (gateway_route(provider, openai_format=True)) for every provider. The model goes under LiteLLM's name: the block'smodel_prefix, or for Anthropic and Gemini, which OpenCompany otherwise calls on native routes, the block'sopenai_prefix. Its client sends the run's attribution headers asdefault_headers. - Scope: the same applies to a connected small model. The supported providers are unchanged; there is no replay cache.
Claude Code (nodes/agent/claude_code_agent/_oauth.py, api_access).
- When the gateway serves Anthropic: own access still comes first: the CLI's login (
claude auth status), then the install's Anthropic key (ANTHROPIC_API_KEY), then the gateway. On the gateway, the spawn getsANTHROPIC_BASE_URL,ANTHROPIC_AUTH_TOKENand the run's attribution headers inANTHROPIC_CUSTOM_HEADERS. - Without the gateway, or with API settings in the server's own environment, nothing changes.
- No reuse: a gateway process carries its run's headers, so it is terminated after its turn instead of staying warm. The next turn resumes the conversation from the memory node's session id, as after an idle eviction.
- Credential tool: the MCP
getCredentialtool never returns the gateway key, allowlisted or not.
Attribution (services/gateway/attribution.py). Ids come from the structlog log context: BaseNode.execute binds the workflow, execution, root execution, chat run, employee job and user ids; the Temporal activity interceptor binds temporal_workflow_id, temporal_run_id, temporal_activity_id and temporal_attempt; the LLM-step and compaction activities bind the attribution dict their payload carries (_attribution_ids in agent_workflow.py). log_context restores outer values on exit, so a tool node run inside an agent keeps the agent's ids. Headers: session = chat run, else root execution, else execution, else Temporal workflow; tags source:, workflow:, user:; every allowlisted id as metadata. Inside a Temporal activity, calls send a replay-cache namespace from the key's fingerprint and the Temporal workflow, run and activity ids (not the attempt), so a retried activity replays at $0 and a re-run never replays another run's answer. Gemini's native route is the exception: LiteLLM 1.103.4 records its session id and tags but drops the metadata header, and ignores the cache body, so OpenCompany sends no cache body there and a retried Gemini step is billed again.
Errors (services/gateway/errors.py, services/llm/protocol.py). _classify_error maps the budget_exceeded type to BILLING before any status check, so the run fails once with no retry loop, and a deployment-spawned run pauses its deployment. Failures of gateway calls get gateway wording through public_message and the new public_hint (allowance used up; model not on the gateway, from model_cost_map_missing; reconnect the gateway key). Compaction inside the agent workflow handles them like an LLM step (raise_llm_errors), so they fail once instead of retrying.
Layout
| Path | Contents |
|---|---|
gateway/litellm/config.yaml |
Production proxy config: provider wildcards, budget defaults, unpriced-model refusal, opt-in Redis cache, spend-log retention |
gateway/local/ |
Local stack: Postgres 16, Redis 7, LiteLLM with mock models, and fake_upstream.py, fake Anthropic, Gemini and OpenAI-audio upstreams. Mocks reuse one response id on /v1/messages (LiteLLM keys spend rows by it and drops repeats), carry no HTTP response on the native Gemini route (LiteLLM's logging drops those) and don't apply on the audio routes, so those tests use the fake upstreams |
gateway/tests/test_gateway_contract.py |
Contract tests against the local stack (skipped when it isn't running) |
gateway/tests/test_pins.py |
Offline checks: version and price-list pins agree; every served default chat, speech and embedding model is priced; every vendor key the config reads has a Terraform secret; Gemini goes to Vertex with the Agent Platform key |
gateway/infra/terraform/ |
Wrapper around BerriAI/litellm/aws: ECS Fargate, Aurora Postgres (IAM auth), ElastiCache, ALB, ACM, Route 53 records in the delegated gw.opencompany.sh zone |
.github/workflows/gateway-ci.yml |
Terraform checks, cosign verification of the pinned images, locked and audited test dependencies, contract and pin tests |
Pins
- LiteLLM 1.103.4, in four places that
test_pins.pykeeps in agreement:local.litellm_versionand the moduleversioninmain.tf, the compose image tag, andLITELLM_VERSIONin CI. 1.103.x is the only stable line with the streamed-alias billing fix (LiteLLM PR #44341). - The price list:
LITELLM_MODEL_COST_MAP_URLpoints at one reviewed commit of LiteLLM'smodel_prices_and_context_window.json, inmain.tfand the compose file. Unpinned, the gateway downloads upstreammainat every start, so prices would change unannounced; the list bundled with 1.103.4 predatesgpt-6-soland the Claude 5.5 models.GET /model/cost_map/sourceshows the list a process loaded (on the deployed stack it is routed to the backend service, not the gateway tasks).
Served providers
OpenAI, Anthropic, Gemini, Groq, Cerebras, xAI, Moonshot (OpenCompany's kimi) and Mistral for chat (a "gateway" block in server/config/llm_defaults.json). Gemini runs on Vertex AI express mode with an Agent Platform key (GOOGLE_AGENT_API_KEY, an AQ. key billed to its Google Cloud project): the Gemini Developer API refuses these keys, so gateway/litellm/config.yaml points the gemini provider's api_base at https://aiplatform.googleapis.com/v1/publishers/google, keeping Gemini's request format and price-list entries. OpenAI speech ("gateway": true in server/config/speech_defaults.json); OpenAI embeddings. test_pins.py checks each one's default models against the pinned price list, counting any cost_per field as a price, as LiteLLM does when it refuses unpriced models.
Not served: OpenRouter, DeepSeek and Sarvam (no price for their default models), Groq speech (1.103.4 bills its transcription at $0), ElevenLabs and Deepgram (OpenCompany speaks their native formats), and every tool API.
Price override: gpt-4o-mini-tts, OpenCompany's Text to Speech default.
- Why: the speech route returns no usage for the audio, so LiteLLM would bill the input text only, roughly 1% of the cost.
- How:
gateway/litellm/config.yamlgives the model its own deployment, priced per input token (input_cost_per_token: 0.000075). That is about $0.015 per minute of speech, at roughly 200 text tokens a minute. - Checks:
test_pins.pykeeps the local config at the same price, and the contract test checks the row.
tts-1 and tts-1-hd are billed per character, and transcription per second of audio, measured from the file.
Contract OpenCompany relies on
| Behaviour | Detail | Test |
|---|---|---|
| Wire model names | OpenAI-format calls use <litellm prefix>/<model> on /v1 (openai/, groq/, cerebras/, xai/, moonshot/, mistral/). Anthropic SDK calls and Claude Code send bare claude-* ids to /v1/messages; Gemini SDK calls use the native /v1beta/models/<id>:generateContent route with the key in x-goog-api-key. Both are also served in OpenAI's format as anthropic/ and gemini/ (the RLM agent) |
test_anthropic_messages_route_accepts_bare_model_ids, test_gemini_native_route_accepts_virtual_key, test_native_providers_answer_in_openai_format |
| Spent allowance | Refused with error type budget_exceeded on all three routes; HTTP 429 on 1.103.x, 402 from 1.104. Recognise it by type |
test_budget_refusal_is_typed_on_every_route |
| Unpriced model | HTTP 403, error type model_cost_map_missing |
test_unpriced_model_is_refused |
| Allowance | GET /user/info with the person's key returns user_info.spend, max_budget and budget_reset_at. Needs a Default-type key; LLM-API keys are refused; another user's id is refused |
test_user_info_needs_a_default_key_and_stays_scoped |
| Attribution | x-litellm-session-id, x-litellm-tags (comma-separated, low cardinality: workflow:, source:, user:) and x-litellm-spend-logs-metadata (JSON) land on the spend row, queryable with /spend/logs/v2?session_id=. On the native Gemini route only the session id and tags land: there the router's metadata goes to litellm_metadata, which the spend row prefers, and only key fields are copied across |
test_attribution_headers_reach_spend_logs, test_messages_route_writes_a_priced_row_for_the_session, test_gemini_native_route_writes_a_priced_row_for_the_session |
| Retry replay | Request body cache: {"use-cache": true, "namespace": <stable per activity>, "ttl": 172800} (OpenAI format through extra_body). A repeated call with the same namespace is a cache hit billed at $0, streamed or not, on /v1/chat/completions and /v1/messages. The native Gemini route ignores it |
test_cache_namespace_replays_retry_for_free, test_streamed_retry_replays_for_free, test_anthropic_messages_cache_namespace |
| Streamed pricing | A real upstream stream through a dotted alias is billed at the deployment's price | test_streamed_alias_is_priced_at_the_deployment |
| Embeddings and speech | /v1/embeddings, /v1/audio/speech and /v1/audio/transcriptions with the openai/ prefix bill the person and keep all three attribution headers; gpt-4o-mini-tts at the override price. There is no /v1/audio/translations |
test_embeddings_route_writes_an_attributed_row, test_speech_routes_write_attributed_rows, test_the_gateway_has_no_audio_translation_route, test_price_overrides_agree |
| Known gaps | Groq transcription billed at $0; on the native Gemini route, metadata dropped and the cache body ignored. Strict xfail: each test fails the suite once a LiteLLM upgrade closes its gap, so the workaround can be lifted | test_groq_transcription_is_billed, test_gemini_native_route_keeps_the_spend_logs_metadata, test_gemini_native_route_replays_retry_for_free |
Keys
People create one Default-type key per install in the console. A key draws on its person's monthly allowance, and LiteLLM reserves the worst-case cost of each call in Redis before admitting it (the model's full context plus its maximum output), so a request with a large max_tokens can be refused near the cap. An allowance smaller than a model's worst case refuses that model outright, even unspent: with $1, gemini-3.5-flash, gemini-3.1-pro-preview and gemini-2.5-pro were refused, while the $50 default admits every curated model. A Default key can also create and delete that person's own keys (not raise the allowance); a leaked key is revoked in the console.