Self-improving AI Employees for everyone
OpenCompany Gateway
The hosted LLM gateway for OpenCompany: invited people use LLMs on platform-owned vendor keys, each with a monthly allowance, and spend is tracked per person and per session. It is the stock LiteLLM Proxy deployed with BerriAI's official Terraform module. Everything in this folder is configuration and tests; there is no custom gateway code.
| What | Where |
|---|---|
| Proxy config (models, budgets, cache) | litellm/config.yaml |
| AWS deployment (ECS Fargate, Aurora Postgres, ElastiCache, ALB) | infra/terraform/ |
| Local stack with mock models | local/ |
| Contract and pin tests | tests/ |
| CI | .github/workflows/gateway-ci.yml |
| How OpenCompany uses it | docs-internal/llm_gateway.md |
How it works
- People are LiteLLM internal users. Each has a monthly USD allowance (
max_budgetwithbudget_duration: 1mo, reset on the 1st, UTC). - Keys: each person creates one Default-type key per OpenCompany install in the console and pastes it into Connectors. Every key of a person draws on the same allowance. OpenCompany needs a Default key to show the allowance: LLM-API keys can't read
/user/info. - Budgets are checked before every call: LiteLLM reserves the worst-case cost in Redis, then settles the real cost. A spent allowance is refused with error type
budget_exceeded(HTTP 429 on 1.103.x; 402 from 1.104, set bybudget_exceeded_status_code). OpenCompany reads the type, not the status. - Prices come from a pinned copy of LiteLLM's price list (
LITELLM_MODEL_COST_MAP_URL). A model without a price is refused (HTTP 403,model_cost_map_missing) rather than billed at $0. - Attribution: OpenCompany sends
x-litellm-session-id,x-litellm-tagsandx-litellm-spend-logs-metadata, so spend shows per person, per run and per workflow. - Retries: OpenCompany sends an opt-in cache namespace on Temporal LLM steps (
cache: {"use-cache": true, "namespace": ...}), so a retried call replays from Redis at $0.
Served providers
| Kind | Providers |
|---|---|
| Chat | OpenAI, Anthropic, Gemini, Groq, Cerebras, xAI, Moonshot (Kimi), Mistral |
| Speech | OpenAI (speech and transcription) |
| Embeddings | OpenAI |
Gemini runs on Vertex AI express mode with a Google Cloud Agent Platform key (GOOGLE_AGENT_API_KEY, a key starting AQ.), billed to that key's project. The Gemini Developer API refuses these keys, so litellm/config.yaml points the gemini provider's api_base at https://aiplatform.googleapis.com/v1/publishers/google. Gemini is also OpenCompany's default model for installs that rely on the gateway ("gateway" in server/config/llm_defaults.json).
Not served: OpenRouter, DeepSeek and Sarvam (their default models have no price in the list), Groq speech (1.103.4 bills Groq transcription at $0), ElevenLabs and Deepgram (OpenCompany uses their native formats), and every tool API (Brave, Serper, TikHub, Google Maps, Perplexity, DeepL, Apify and the rest). Those stay on each install's own keys.
To serve another provider:
- Add its secret name to
secret_env_namesand its wildcard tolitellm/config.yaml. - Give it a
"gateway"block inserver/config/llm_defaults.json(chat), and"gateway": trueinserver/config/speech_defaults.jsonif its speech is served too. - Run the tests.
tests/test_pins.pychecks that its default models are priced; check its routes bill the person on the local stack. - Apply.
Local stack and tests
Needs Docker. Mock models plus fake Anthropic, Gemini and OpenAI-audio upstreams (local/fake_upstream.py) for what mocks can't show (spend rows on /v1/messages, the native Gemini route and the audio routes, streamed pricing): nothing is billed and no vendor key is needed. Three contract tests are strict expected failures that pin LiteLLM gaps OpenCompany works around. When an upgrade closes a gap, its test passes and fails the suite: lift the workaround and drop the marker.
docker compose -f gateway/local/docker-compose.yml up -d --wait
uv run --project gateway pytest gateway/tests -v
docker compose -f gateway/local/docker-compose.yml down -v
tests/test_pins.py needs no stack. It checks that:
- the version and price-list pins agree across files;
- every served provider's default model is priced;
- every vendor key the config reads has a Terraform secret;
- Gemini goes to Vertex with the Agent Platform key.
To try real Gemini on the local stack, mount a copy of local/config.local.yaml whose gemini-* and gemini/* entries match litellm/config.yaml, through a compose override that reads GOOGLE_AGENT_API_KEY from an env file outside the repository. That costs a fraction of a cent per call, billed to the key's project. The console runs at http://127.0.0.1:4000/ui (user admin, password sk-local-master-key).
First deployment
Prerequisites: AWS credentials for the target account, an S3 bucket for Terraform state (fill REPLACE_WITH_STATE_BUCKET in infra/terraform/envs/<env>.backend.hcl), Terraform 1.14, and the aws CLI (the module runs the database bootstrap and migrations through it).
DNS, once for both environments.
opencompany.shis on Cloudflare, so delegate a subzone to Route 53:aws route53 create-hosted-zone --name gw.opencompany.sh --caller-reference "gw-$(date +%s)"Add the four NS records it returns in Cloudflare as
gwNS records (DNS only). Wait untildig NS gw.opencompany.sh +shortreturns them, then put the zone id ininfra/terraform/envs/<env>.tfvars(REPLACE_WITH_GW_ZONE_ID). Hostnames:api.gw.opencompany.sh(prod) andstaging.gw.opencompany.sh.Secrets. Create the empty secrets, then fill every one, including
SLACK_WEBHOOK_URL: ECS tasks only start once each secret has a value.cd gateway/infra/terraform terraform init -backend-config=envs/staging.backend.hcl terraform apply -var-file=envs/staging.tfvars -target=aws_secretsmanager_secret.vendor aws secretsmanager put-secret-value --secret-id opencompany-gateway-staging/env/openai-api-key --secret-string 'sk-...'Give Gemini its own Agent Platform key: in the Google Cloud project, restricted to the Vertex AI API and bound to the project's service account. Create it in the console or with gcloud, then pass its string straight to the secret, so it never sits in a chat or a file:
gcloud services api-keys create --display-name=opencompany-gateway \ --api-target=service=aiplatform.googleapis.com --service-account=<service-account-email> aws secretsmanager put-secret-value --secret-id opencompany-gateway-staging/env/google-agent-api-key \ --secret-string "$(gcloud services api-keys get-key-string <key-name> --format='value(keyString)')"Full apply.
terraform apply -var-file=envs/staging.tfvars. The module bootstraps the database and runs the migrations before the services start.SpendLogs indexes, once. LiteLLM no longer creates them in migrations (1.103.3+). Run the command from
terraform output -raw spendlogs_index_command; it reuses the module's bootstrap task with the index SQL.Price list check. Call a current model through the gateway, for example
gpt-6-sol. A 403model_cost_map_missingmeans the task couldn't fetch the pinned price list at start and fell back to LiteLLM's bundled one: check the task logs, then restart the service.Admin access.
- Read the master key from the
master_key_secret_arnoutput and sign in to<gateway_url>/uiasadminwith it. - Create named admin accounts (Internal Users, role Admin) and sign in with one.
- Set
disable_env_credential_login: trueingeneral_settingsand apply, so the master key no longer opens the console.
- Read the master key from the
Adding a person
- In the console, create an internal user (or
POST /user/new) with their monthlymax_budget. New people default tomax_internal_user_budget($50). - Create an invitation (
POST /invitation/new). Send them the link yourself: email delivery is an Enterprise feature. Links expire after 7 days. - They set a password, open Virtual Keys, and create one key per install with the Default key type (the console's default). They paste it into OpenCompany's Connectors panel.
Avoid team keys: they skip the person's allowance unless apply_user_budget_to_team_keys is set.
Upgrades
LiteLLM ships a minor release every week and supports only the latest four minor lines, so upgrade every 2-4 weeks. Apply security releases within 72 hours (advisories).
- Bump the version together in
local.litellm_versionand the moduleversion(infra/terraform/main.tf), the image tag inlocal/docker-compose.yml, andLITELLM_VERSIONin the CI workflow.tests/test_pins.pyfails if they disagree. - To move the price list, pick a reviewed commit of
model_prices_and_context_window.jsonand changeLITELLM_MODEL_COST_MAP_URLin bothmain.tfandlocal/docker-compose.yml. - Open a PR: CI verifies the image signatures and runs the contract and pin tests on the new version.
- Take an Aurora snapshot, then apply to staging. The module runs the database migration before the services roll.
- Smoke-test staging with a real cheap model per served provider, then apply to prod.
Why 1.103.x: it is the only stable line with the streamed-alias billing fix (LiteLLM PR #44341, also in 1.106.0-dev). test_streamed_alias_is_priced_at_the_deployment guards streamed pricing through an alias on every upgrade. Moving to 1.104 or later also turns budget refusals into 402 (budget_exceeded_status_code).
Operating notes
- Alerts go to Slack (
SLACK_WEBHOOK_URL): budget thresholds, outages, database errors. - Spend logs are kept for 90 days (
maximum_spend_logs_retention_period); the daily rollups that drive the usage views are kept. - Redis must never evict keys: evictions silently drop budget and rate-limit state. The module's ElastiCache cluster has a replica with automatic failover.
- The salt key (
LITELLM_SALT_KEY) encrypts credentials stored in the database and can never change. Terraform generates it once and ignores later changes. - The price list is fetched when a task starts. If GitHub is unreachable then, LiteLLM serves its bundled list (which refuses newer models) and retries in the background.
- Rust: LiteLLM's Rust path is beta and opt-in per model. To trial it, set
rust: trueon the Anthropic models in staging, confirm responses carryx-litellm-rust: trueand spend logs are unchanged, then promote. The standalone Rust gateway is not production-ready yet.
Known limits
- Replay covers completed LLM calls only; an abandoned stream is billed for what it produced.
- A small allowance refuses expensive models outright. LiteLLM reserves each call's worst case (the model's full context plus its maximum output) before admitting it, so an allowance below that refuses the model with
budget_exceeded, even unspent. With $1,gemini-3.5-flash,gemini-3.1-pro-previewandgemini-2.5-prowere refused; the $50 default admits them. - Gemini SDK calls (the native
generateContentroute) carry the session id and tags, but 1.103.4 drops their per-call metadata and doesn't replay them from the cache, so a retried Gemini step is billed again. gpt-4o-mini-tts(OpenCompany's Text to Speech default) carries a price override inlitellm/config.yaml. The speech route returns no usage for the audio, so LiteLLM would otherwise bill the input text only, about 1% of the cost. The override's per-token rate assumes about 200 text tokens per minute of speech; revisit it if OpenAI's price changes.- Groq speech is not served: 1.103.4 bills Groq transcription at $0. The
groq/*wildcard still lets a person call Groq Whisper directly with their key, billed $0 to them; Groq charges about $0.11 per hour of audio. - A Default key can create and delete that person's own keys, but not raise their allowance. Revoke a leaked key in the console.
- Claude Code on the gateway holds the install's gateway key in its process environment, where its own tools (Bash) can read it, as with any API key a CLI agent runs on. The allowance caps what a leaked key can spend.
- The open-source tier has no SSO beyond 5 users, no audit log, no automatic key rotation, and no route permissions for tool pass-through. Those need a LiteLLM Enterprise license.