Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(gateway): run Gemini on Vertex AI with an Agent Platform key
trohitg3acea3beecf8
diff --git a/docs-internal/llm_gateway.md b/docs-internal/llm_gateway.mdindex 0f518ee3..405890dc 100644--- a/docs-internal/llm_gateway.md+++ b/docs-internal/llm_gateway.md@@ -56,7 +56,7 @@ OpenCompany's hosted LLM gateway is the stock open-source LiteLLM Proxy, deploye## Served providers-OpenAI, Anthropic, Gemini, Groq, Cerebras, xAI, Moonshot (OpenCompany's `kimi`) and Mistral for chat (a `"gateway"` block in `server/config/llm_defaults.json`); OpenAI speech (`"gateway": true` in `server/config/speech_defaults.json`); OpenAI embeddings. `test_pins.py` checks each one's default models against the pinned price list, counting any `cost_per` field as a price, as LiteLLM does when it refuses unpriced models.+OpenAI, Anthropic, Gemini, Groq, Cerebras, xAI, Moonshot (OpenCompany's `kimi`) and Mistral for chat (a `"gateway"` block in `server/config/llm_defaults.json`). Gemini runs on Vertex AI express mode with an Agent Platform key (`GOOGLE_AGENT_API_KEY`, an `AQ.` key billed to its Google Cloud project): the Gemini Developer API refuses these keys, so `gateway/litellm/config.yaml` points the `gemini` provider's `api_base` at `https://aiplatform.googleapis.com/v1/publishers/google`, keeping Gemini's request format and price-list entries. OpenAI speech (`"gateway": true` in `server/config/speech_defaults.json`); OpenAI embeddings. `test_pins.py` checks each one's default models against the pinned price list, counting any `cost_per` field as a price, as LiteLLM does when it refuses unpriced models.Not served: OpenRouter, DeepSeek and Sarvam (no price for their default models), Groq speech (1.103.4 bills its transcription at $0), ElevenLabs and Deepgram (OpenCompany speaks their native formats), and every tool API.diff --git a/gateway/README.md b/gateway/README.mdindex 8a65d438..a391b6e4 100644--- a/gateway/README.md+++ b/gateway/README.md@@ -28,6 +28,8 @@ The hosted LLM gateway for OpenCompany: invited people use LLMs on platform-owne| Speech | OpenAI (speech and transcription) || Embeddings | OpenAI |+Gemini runs on Vertex AI express mode with a Google Cloud Agent Platform key (`GOOGLE_AGENT_API_KEY`, a key starting `AQ.`), billed to that key's project. The Gemini Developer API refuses these keys, so `litellm/config.yaml` points the `gemini` provider's `api_base` at `https://aiplatform.googleapis.com/v1/publishers/google`.+Not served: OpenRouter, DeepSeek and Sarvam (their default models have no price in the list), Groq speech (1.103.4 bills Groq transcription at $0), ElevenLabs and Deepgram (OpenCompany uses their native formats), and every tool API (Brave, Serper, TikHub, Google Maps, Perplexity, DeepL, Apify and the rest). Those stay on each install's own keys.To serve another provider:diff --git a/gateway/infra/terraform/variables.tf b/gateway/infra/terraform/variables.tfindex b91ed1bb..8bc709b3 100644--- a/gateway/infra/terraform/variables.tf+++ b/gateway/infra/terraform/variables.tf@@ -53,7 +53,7 @@ variable "secret_env_names" {default = ["OPENAI_API_KEY","ANTHROPIC_API_KEY",- "GEMINI_API_KEY",+ "GOOGLE_AGENT_API_KEY","GROQ_API_KEY","CEREBRAS_API_KEY","XAI_API_KEY",diff --git a/gateway/litellm/config.yaml b/gateway/litellm/config.yamlindex c89e23a9..b7f42df1 100644--- a/gateway/litellm/config.yaml+++ b/gateway/litellm/config.yaml@@ -27,15 +27,21 @@ model_list:litellm_params:model: anthropic/claude-*api_key: os.environ/ANTHROPIC_API_KEY+ # Gemini runs on Vertex AI express mode with an Agent Platform key (an AQ.+ # key, billed to its Google Cloud project), which the Gemini Developer API+ # refuses. The gemini provider keeps Gemini's request format and prices;+ # api_base points it at Vertex, where it sends the key as x-goog-api-key.- model_name: gemini/*litellm_params:model: gemini/*- api_key: os.environ/GEMINI_API_KEY+ api_key: os.environ/GOOGLE_AGENT_API_KEY+ api_base: https://aiplatform.googleapis.com/v1/publishers/google# The Gemini SDK puts a bare model id in the /v1beta/models/<id> path.- model_name: gemini-*litellm_params:model: gemini/gemini-*- api_key: os.environ/GEMINI_API_KEY+ api_key: os.environ/GOOGLE_AGENT_API_KEY+ api_base: https://aiplatform.googleapis.com/v1/publishers/google- model_name: groq/*litellm_params:model: groq/*diff --git a/gateway/tests/test_pins.py b/gateway/tests/test_pins.pyindex 6e542a89..9c4d1be1 100644--- a/gateway/tests/test_pins.py+++ b/gateway/tests/test_pins.py@@ -23,6 +23,7 @@ import yamlREPO = Path(__file__).resolve().parents[2]TERRAFORM = (REPO / "gateway/infra/terraform/main.tf").read_text(encoding="utf-8")+VARIABLES = (REPO / "gateway/infra/terraform/variables.tf").read_text(encoding="utf-8")GATEWAY_CONFIG = yaml.safe_load((REPO / "gateway/litellm/config.yaml").read_text(encoding="utf-8"))LOCAL_CONFIG = yaml.safe_load((REPO / "gateway/local/config.local.yaml").read_text(encoding="utf-8"))COMPOSE = yaml.safe_load((REPO / "gateway/local/docker-compose.yml").read_text(encoding="utf-8"))@@ -69,6 +70,28 @@ def test_price_overrides_agree() -> None:assert _price_overrides(LOCAL_CONFIG) == overrides+def test_every_vendor_key_has_a_secret() -> None:+ """ECS injects only the secrets Terraform creates (secret_env_names)."""+ block = re.search(r'variable "secret_env_names".*?default\s*=\s*\[(.*?)\]', VARIABLES, re.DOTALL)+ assert block, "secret_env_names not found in gateway/infra/terraform/variables.tf"+ secrets = set(re.findall(r'"([A-Z0-9_]+)"', block.group(1)))+ keys = {+ str(deployment["litellm_params"]["api_key"]).removeprefix("os.environ/")+ for deployment in GATEWAY_CONFIG["model_list"]+ if str(deployment["litellm_params"].get("api_key", "")).startswith("os.environ/")+ }+ assert keys <= secrets, f"vendor keys with no secret: {sorted(keys - secrets)}"+++def test_gemini_runs_on_vertex_with_the_agent_platform_key() -> None:+ """An Agent Platform key (AQ.) works on Vertex AI only; the Gemini Developer API refuses it."""+ gemini = [d["litellm_params"] for d in GATEWAY_CONFIG["model_list"] if d["model_name"] in ("gemini/*", "gemini-*")]+ assert len(gemini) == 2, gemini+ for params in gemini:+ assert params["api_key"] == "os.environ/GOOGLE_AGENT_API_KEY", params+ assert params["api_base"] == "https://aiplatform.googleapis.com/v1/publishers/google", params++@pytest.fixture(scope="module")def prices() -> dict:url = LITELLM["environment"]["LITELLM_MODEL_COST_MAP_URL"]