Skip to content

Self-improving AI Employees for everyone

12.4 KB
Raw Blame

OpenCompany Gateway

The hosted LLM gateway for OpenCompany: invited people use LLMs on platform-owned vendor keys, each with a monthly allowance, and spend is tracked per person and per session. It is the stock LiteLLM Proxy deployed with BerriAI's official Terraform module. Everything in this folder is configuration and tests; there is no custom gateway code.

What Where
Proxy config (models, budgets, cache) litellm/config.yaml
AWS deployment (ECS Fargate, Aurora Postgres, ElastiCache, ALB) infra/terraform/
Local stack with mock models local/
Contract and pin tests tests/
CI .github/workflows/gateway-ci.yml
How OpenCompany uses it docs-internal/llm_gateway.md

How it works

  • People are LiteLLM internal users. Each has a monthly USD allowance (max_budget with budget_duration: 1mo, reset on the 1st, UTC).
  • Keys: each person creates one Default-type key per OpenCompany install in the console and pastes it into Connectors. Every key of a person draws on the same allowance. OpenCompany needs a Default key to show the allowance: LLM-API keys can't read /user/info.
  • Budgets are checked before every call: LiteLLM reserves the worst-case cost in Redis, then settles the real cost. A spent allowance is refused with error type budget_exceeded (HTTP 429 on 1.103.x; 402 from 1.104, set by budget_exceeded_status_code). OpenCompany reads the type, not the status.
  • Prices come from a pinned copy of LiteLLM's price list (LITELLM_MODEL_COST_MAP_URL). A model without a price is refused (HTTP 403, model_cost_map_missing) rather than billed at $0.
  • Attribution: OpenCompany sends x-litellm-session-id, x-litellm-tags and x-litellm-spend-logs-metadata, so spend shows per person, per run and per workflow.
  • Retries: OpenCompany sends an opt-in cache namespace on Temporal LLM steps (cache: {"use-cache": true, "namespace": ...}), so a retried call replays from Redis at $0.

Served providers

Kind Providers
Chat OpenAI, Anthropic, Gemini, Groq, Cerebras, xAI, Moonshot (Kimi), Mistral
Speech OpenAI (speech and transcription)
Embeddings OpenAI

Gemini runs on Vertex AI express mode with a Google Cloud Agent Platform key (GOOGLE_AGENT_API_KEY, a key starting AQ.), billed to that key's project. The Gemini Developer API refuses these keys, so litellm/config.yaml points the gemini provider's api_base at https://aiplatform.googleapis.com/v1/publishers/google. Gemini is also OpenCompany's default model for installs that rely on the gateway ("gateway" in server/config/llm_defaults.json).

Not served: OpenRouter, DeepSeek and Sarvam (their default models have no price in the list), Groq speech (1.103.4 bills Groq transcription at $0), ElevenLabs and Deepgram (OpenCompany uses their native formats), and every tool API (Brave, Serper, TikHub, Google Maps, Perplexity, DeepL, Apify and the rest). Those stay on each install's own keys.

To serve another provider:

  1. Add its secret name to secret_env_names and its wildcard to litellm/config.yaml.
  2. Give it a "gateway" block in server/config/llm_defaults.json (chat), and "gateway": true in server/config/speech_defaults.json if its speech is served too.
  3. Run the tests. tests/test_pins.py checks that its default models are priced; check its routes bill the person on the local stack.
  4. Apply.

Local stack and tests

Needs Docker. Mock models plus fake Anthropic, Gemini and OpenAI-audio upstreams (local/fake_upstream.py) for what mocks can't show (spend rows on /v1/messages, the native Gemini route and the audio routes, streamed pricing): nothing is billed and no vendor key is needed. Three contract tests are strict expected failures that pin LiteLLM gaps OpenCompany works around. When an upgrade closes a gap, its test passes and fails the suite: lift the workaround and drop the marker.

docker compose -f gateway/local/docker-compose.yml up -d --wait
uv run --project gateway pytest gateway/tests -v
docker compose -f gateway/local/docker-compose.yml down -v

tests/test_pins.py needs no stack. It checks that:

  • the version and price-list pins agree across files;
  • every served provider's default model is priced;
  • every vendor key the config reads has a Terraform secret;
  • Gemini goes to Vertex with the Agent Platform key.

To try real Gemini on the local stack, mount a copy of local/config.local.yaml whose gemini-* and gemini/* entries match litellm/config.yaml, through a compose override that reads GOOGLE_AGENT_API_KEY from an env file outside the repository. That costs a fraction of a cent per call, billed to the key's project. The console runs at http://127.0.0.1:4000/ui (user admin, password sk-local-master-key).

First deployment

Prerequisites: AWS credentials for the target account, an S3 bucket for Terraform state (fill REPLACE_WITH_STATE_BUCKET in infra/terraform/envs/<env>.backend.hcl), Terraform 1.14, and the aws CLI (the module runs the database bootstrap and migrations through it).

  1. DNS, once for both environments. opencompany.sh is on Cloudflare, so delegate a subzone to Route 53:

    aws route53 create-hosted-zone --name gw.opencompany.sh --caller-reference "gw-$(date +%s)"
    

    Add the four NS records it returns in Cloudflare as gw NS records (DNS only). Wait until dig NS gw.opencompany.sh +short returns them, then put the zone id in infra/terraform/envs/<env>.tfvars (REPLACE_WITH_GW_ZONE_ID). Hostnames: api.gw.opencompany.sh (prod) and staging.gw.opencompany.sh.

  2. Secrets. Create the empty secrets, then fill every one, including SLACK_WEBHOOK_URL: ECS tasks only start once each secret has a value.

    cd gateway/infra/terraform
    terraform init -backend-config=envs/staging.backend.hcl
    terraform apply -var-file=envs/staging.tfvars -target=aws_secretsmanager_secret.vendor
    aws secretsmanager put-secret-value --secret-id opencompany-gateway-staging/env/openai-api-key --secret-string 'sk-...'
    

    Give Gemini its own Agent Platform key: in the Google Cloud project, restricted to the Vertex AI API and bound to the project's service account. Create it in the console or with gcloud, then pass its string straight to the secret, so it never sits in a chat or a file:

    gcloud services api-keys create --display-name=opencompany-gateway \
      --api-target=service=aiplatform.googleapis.com --service-account=<service-account-email>
    aws secretsmanager put-secret-value --secret-id opencompany-gateway-staging/env/google-agent-api-key \
      --secret-string "$(gcloud services api-keys get-key-string <key-name> --format='value(keyString)')"
    
  3. Full apply. terraform apply -var-file=envs/staging.tfvars. The module bootstraps the database and runs the migrations before the services start.

  4. SpendLogs indexes, once. LiteLLM no longer creates them in migrations (1.103.3+). Run the command from terraform output -raw spendlogs_index_command; it reuses the module's bootstrap task with the index SQL.

  5. Price list check. Call a current model through the gateway, for example gpt-6-sol. A 403 model_cost_map_missing means the task couldn't fetch the pinned price list at start and fell back to LiteLLM's bundled one: check the task logs, then restart the service.

  6. Admin access.

    1. Read the master key from the master_key_secret_arn output and sign in to <gateway_url>/ui as admin with it.
    2. Create named admin accounts (Internal Users, role Admin) and sign in with one.
    3. Set disable_env_credential_login: true in general_settings and apply, so the master key no longer opens the console.

Adding a person

  1. In the console, create an internal user (or POST /user/new) with their monthly max_budget. New people default to max_internal_user_budget ($50).
  2. Create an invitation (POST /invitation/new). Send them the link yourself: email delivery is an Enterprise feature. Links expire after 7 days.
  3. They set a password, open Virtual Keys, and create one key per install with the Default key type (the console's default). They paste it into OpenCompany's Connectors panel.

Avoid team keys: they skip the person's allowance unless apply_user_budget_to_team_keys is set.

Upgrades

LiteLLM ships a minor release every week and supports only the latest four minor lines, so upgrade every 2-4 weeks. Apply security releases within 72 hours (advisories).

  1. Bump the version together in local.litellm_version and the module version (infra/terraform/main.tf), the image tag in local/docker-compose.yml, and LITELLM_VERSION in the CI workflow. tests/test_pins.py fails if they disagree.
  2. To move the price list, pick a reviewed commit of model_prices_and_context_window.json and change LITELLM_MODEL_COST_MAP_URL in both main.tf and local/docker-compose.yml.
  3. Open a PR: CI verifies the image signatures and runs the contract and pin tests on the new version.
  4. Take an Aurora snapshot, then apply to staging. The module runs the database migration before the services roll.
  5. Smoke-test staging with a real cheap model per served provider, then apply to prod.

Why 1.103.x: it is the only stable line with the streamed-alias billing fix (LiteLLM PR #44341, also in 1.106.0-dev). test_streamed_alias_is_priced_at_the_deployment guards streamed pricing through an alias on every upgrade. Moving to 1.104 or later also turns budget refusals into 402 (budget_exceeded_status_code).

Operating notes

  • Alerts go to Slack (SLACK_WEBHOOK_URL): budget thresholds, outages, database errors.
  • Spend logs are kept for 90 days (maximum_spend_logs_retention_period); the daily rollups that drive the usage views are kept.
  • Redis must never evict keys: evictions silently drop budget and rate-limit state. The module's ElastiCache cluster has a replica with automatic failover.
  • The salt key (LITELLM_SALT_KEY) encrypts credentials stored in the database and can never change. Terraform generates it once and ignores later changes.
  • The price list is fetched when a task starts. If GitHub is unreachable then, LiteLLM serves its bundled list (which refuses newer models) and retries in the background.
  • Rust: LiteLLM's Rust path is beta and opt-in per model. To trial it, set rust: true on the Anthropic models in staging, confirm responses carry x-litellm-rust: true and spend logs are unchanged, then promote. The standalone Rust gateway is not production-ready yet.

Known limits

  • Replay covers completed LLM calls only; an abandoned stream is billed for what it produced.
  • A small allowance refuses expensive models outright. LiteLLM reserves each call's worst case (the model's full context plus its maximum output) before admitting it, so an allowance below that refuses the model with budget_exceeded, even unspent. With $1, gemini-3.5-flash, gemini-3.1-pro-preview and gemini-2.5-pro were refused; the $50 default admits them.
  • Gemini SDK calls (the native generateContent route) carry the session id and tags, but 1.103.4 drops their per-call metadata and doesn't replay them from the cache, so a retried Gemini step is billed again.
  • gpt-4o-mini-tts (OpenCompany's Text to Speech default) carries a price override in litellm/config.yaml. The speech route returns no usage for the audio, so LiteLLM would otherwise bill the input text only, about 1% of the cost. The override's per-token rate assumes about 200 text tokens per minute of speech; revisit it if OpenAI's price changes.
  • Groq speech is not served: 1.103.4 bills Groq transcription at $0. The groq/* wildcard still lets a person call Groq Whisper directly with their key, billed $0 to them; Groq charges about $0.11 per hour of audio.
  • A Default key can create and delete that person's own keys, but not raise their allowance. Revoke a leaked key in the console.
  • Claude Code on the gateway holds the install's gateway key in its process environment, where its own tools (Bash) can read it, as with any API key a CLI agent runs on. The allowance caps what a leaked key can spend.
  • The open-source tier has no SSO beyond 5 users, no audit log, no automatic key rotation, and no route permissions for tool pass-through. Those need a LiteLLM Enterprise license.