Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

AI triage could not reach a model, and the variable that would have fixed it was read by nothing (#829)

Beenu Aroraad775e8df38f
diff --git a/.env.example b/.env.example
index b881e5d1d..f612be8a6 100644
--- a/.env.example
+++ b/.env.example
@@ -105,10 +105,20 @@ QDRANT_URL=http://localhost:6333
# ─── AI / LLM ─────────────────────────────────────────────────────────────────
OPENAI_API_KEY=sk-your-openai-api-key-here
-# OPENAI_MODEL now only affects the "explain this alert" / BYOK path. Task
-# workloads (triage, recon, investigation, copilot, summary, report, NL) request
-# a logical alias instead — see the LLM gateway block below.
-OPENAI_MODEL=gpt-4-turbo-preview
+# OPENAI_MODEL affects the "explain this alert" / BYOK path only. Task workloads
+# (triage, recon, investigation, copilot, summary, report, NL) request a logical
+# alias instead — see the LLM gateway block below.
+#
+# It ships as an alias, not a concrete model, because the default deployment
+# routes through the bundled gateway and the gateway 400s anything
+# infra/litellm/config.yaml does not define. It used to read
+# `gpt-4-turbo-preview`, which no gateway config here has ever defined, and the
+# auto-triage worker treated it as a per-tenant BYOK override — so it replaced
+# `aisoc-triage` on the highest-volume path in the product.
+# scripts/check_llm_model_routing.py fails the build if this names a model the
+# gateway does not define. Set a concrete model only when calling a provider
+# directly (see "ENABLE LIVE LLM" option (b) below).
+OPENAI_MODEL=aisoc-summary
# ─── LLM gateway (LiteLLM) — issue #478 ───────────────────────────────────────
# Since #478 AiSOC asks for a logical task alias (aisoc-triage,
@@ -119,19 +129,30 @@ OPENAI_MODEL=gpt-4-turbo-preview
#
# Key that AiSOC uses to authenticate to the gateway (also LiteLLM's admin key):
LITELLM_MASTER_KEY=sk-aisoc-local
-# In-network URL of the gateway (already set per-service in docker-compose.yml):
+# In-network URL of the gateway. docker-compose.yml sets this on the api and
+# agents services, and both resolvers READ it — `resolve_base_url()` returns it
+# for any aisoc-<role> alias, and `resolve_api_key()` returns LITELLM_MASTER_KEY
+# to pair with it. That wire was missing: compose set this variable, the
+# resolvers deliberately ignored it in favour of OPENAI_BASE_URL (which nothing
+# set), and so the product's headline capability could not reach a model in the
+# default deployment.
LLM_GATEWAY_URL=http://litellm:4000/v1
#
-# ENABLE LIVE LLM — pick one:
-# (a) Route through the gateway (recommended). Put your real provider key in
-# OPENAI_API_KEY above (LiteLLM reads it), then uncomment:
-# OPENAI_BASE_URL=http://litellm:4000/v1
-# OPENAI_API_KEY=${LITELLM_MASTER_KEY}
-# (b) Call a provider directly without the gateway — pin each task role to a
-# concrete model (escape hatch), e.g.:
+# ENABLE LIVE LLM:
+# (a) Through the gateway — the default, and already wired. Put your real
+# provider key in OPENAI_API_KEY above (the gateway reads it) and bring up
+# the `full` profile, which is where the litellm service lives:
+# docker compose --profile full up -d
+# Nothing else to set. To point at a gateway somewhere else, or to send a
+# different bearer, set OPENAI_BASE_URL/OPENAI_API_KEY and they win.
+# (b) Direct to a provider, no gateway — pin each task role to a concrete
+# model. A concrete pin is NOT rerouted to the gateway, so this keeps
+# working with LLM_GATEWAY_URL set:
# AISOC_MODEL_PIN_TRIAGE=gpt-4o-mini
# AISOC_MODEL_PIN_INVESTIGATION=gpt-4o
-# With neither configured, AiSOC uses its deterministic offline path.
+# OPENAI_MODEL=gpt-4o-mini # the BYOK/explain path, direct too
+# With neither configured, AiSOC uses its deterministic offline path — which
+# reports zero model calls and zero tokens rather than pretending to triage.
# ─── Threat Intelligence ──────────────────────────────────────────────────────
# Open-source / community feeds (work out of the box, no key required for some)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 1a12343e9..ce9cd480f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -412,6 +412,22 @@ jobs:
run: python3 scripts/check_connector_profiles.py --self-test
- name: Connector profile drift gate
run: python3 scripts/check_connector_profiles.py
+ # Same question one layer up: a connector name has to reach a profile,
+ # and a model name has to reach a gateway that defines it. AI triage
+ # could not reach a model in the default deployment because
+ # docker-compose.yml set LLM_GATEWAY_URL and both resolvers ignored it,
+ # and because .env.example shipped a model no gateway config here has
+ # ever defined. Neither is visible in a diff, and both are structural.
+ #
+ # The self-test runs first and on the same tree: it injects drift in
+ # each direction and feeds the parsers prose shaped like the thing they
+ # look for — a commented-out alias in the config's Ollama example, a
+ # commented-out pin in .env.example's escape-hatch block, a compose
+ # comment naming a variable two lines above the key it describes.
+ - name: LLM model routing gate — self-test
+ run: python3 scripts/check_llm_model_routing.py --self-test
+ - name: LLM model routing gate
+ run: python3 scripts/check_llm_model_routing.py
# The count claim above says how many connectors exist; this says
# whether a reader can find out how to use them. 35 of 84 had no page
# at all and the sidebar listed 34 of 96, so most connector
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4b9c7c49a..92fcb5032 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -75,8 +75,105 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the breaking-change gate does not see this; it is recorded here because a
dropped enum value is a break whether or not a workflow notices.
+### Fixed
+
+- **AI triage could not reach a model in the default deployment, and it was
+ not a missing key.** `docker-compose.yml` set `LLM_GATEWAY_URL` on the `api`
+ and `agents` services. Nothing read it. Both resolvers honoured only
+ `OPENAI_BASE_URL`, which compose never set, and each carried a comment
+ explaining that the refusal was deliberate — routing through the gateway
+ should be an explicit choice so it is never ambiguous whether the bearer is
+ the gateway master key or a provider key. The reasoning was sound and the
+ result was that every `aisoc-<role>` alias went to `api.openai.com`, which
+ cannot resolve one, and the caller's `except` rendered that as "no LLM
+ available". `preflight_llm()` logged exactly this condition at boot with the
+ remedy, so the diagnosis existed and the wiring did not.
+
+ `LLM_GATEWAY_URL` is now the lowest-precedence base URL, and the ambiguity is
+ resolved rather than avoided. **The model decides the route:** the gateway is
+ adopted only for a gateway alias, which resolves nowhere else, while a
+ concrete `AISOC_MODEL_PIN_<ROLE>` — the documented direct-to-provider escape
+ hatch — is left pointing at its provider. **The route decides the bearer:**
+ when AiSOC picks the gateway itself it sends `LITELLM_MASTER_KEY`, which
+ compose now supplies to both services, and never a provider key. An explicit
+ `OPENAI_BASE_URL` still outranks both. The rule lives once, in
+ `services/agents/app/llm/routing.py`, shared with the BYOK resolver and
+ mirrored in `services/api/app/services/model_aliases.py` for the service that
+ cannot import it.
+
+ Consumers audited and fixed alongside it: the API copilot and the NL-query
+ translator (both already went through the shared helpers); the agents
+ **contextual copilot**, which built `ChatOpenAI(model=…)` with no base URL at
+ all and reported "LLM not configured" from a key check that could not see the
+ gateway's; the **detection builder** (`/nl-detection`), the only consumer that
+ could not be pointed anywhere — it POSTed to a hardcoded
+ `https://api.openai.com/v1/chat/completions` with `OPENAI_MODEL`; the four API
+ endpoints (translation, hunts, knowledge base, phishing) that read
+ `LLM_BASE_URL` and defaulted to OpenAI without consulting `OPENAI_BASE_URL`;
+ the BYOK / "explain this alert" resolvers in both services, which defaulted
+ straight to `https://api.openai.com` and took an alias there; and
+ `GET /llm/status`, so the indicator describes the route the explain path
+ actually takes. The **MITRE RAG embedding** path is a deliberate exclusion —
+ `infra/litellm/config.yaml` declares chat aliases only, so an embedding call
+ routed there would 400 on every batch. It has its own
+ `AISOC_EMBEDDING_BASE_URL` / `AISOC_EMBEDDING_MODEL` pair and the gate checks
+ the exclusion in both directions.
+
+- **`OPENAI_MODEL` replaced the triage role's model on the highest-volume path
+ in the product.** `.env.example` shipped `OPENAI_MODEL=gpt-4-turbo-preview`,
+ a model no gateway config in this tree has ever defined. The auto-triage
+ worker layers a tenant's BYOK configuration over the role pin — but the
+ resolver's `LlmConfig` collapsed provenance into a single `source` word, so
+ the worker could not tell a per-tenant override from the process-wide env
+ baseline and bound all three fields unconditionally. Once traffic reached the
+ gateway that would have produced `Invalid model name` on every triage call:
+ the failure the commercial deployment already hit, where the copilot and every
+ triage agent degraded to empty output.
+
+ `LlmConfig` now carries `model_from_tenant` / `base_url_from_tenant` /
+ `api_key_from_tenant`, computed from state the resolvers already tracked, and
+ only a field the tenant actually set is treated as an override. A real BYOK
+ model still wins. `.env.example` ships `OPENAI_MODEL=aisoc-summary` — an alias
+ the bundled gateway defines — and the variable is documented as the BYOK /
+ explain path only, never a task role.
+
+- **A model with nowhere to go is now an error instead of a fallback.**
+ `make_chat_model` and `chat_completions_url` raise `UnroutableModelError`,
+ naming the alias and the remedy, when a gateway alias is requested with no
+ gateway configured — rather than building a client destined to 404 and
+ letting a caller read that as "the LLM is unavailable". `run_auto_triage`
+ constructs its model inside its own `try` so the failure arrives as
+ `AutoTriageError` like every other LLM failure; `auto_triage_node` catches
+ that specifically, so constructing outside it would have failed a whole graph
+ run over a configuration problem the deterministic path handles. The
+ deterministic floor is unchanged and still reports `call_count=0 models=[]
+ total_tokens=0` with no `recommended_actions`. `preflight_llm()` now reports
+ both directions at boot: an alias with no gateway, and a concrete model
+ pointed at the bundled gateway.
+
### Added
+- **A model name must resolve, and the variable that routes it must be read —
+ gated in both directions.** `scripts/check_llm_model_routing.py` reconciles
+ `infra/litellm/config.yaml`, the role pins, the API-side mirror,
+ `.env.example` and `docker-compose.yml`: every pin's alias must be defined by
+ the gateway and every alias claimed by a pin; every model named in an env or
+ compose file that wires the bundled gateway must be one the gateway defines;
+ every variable compose points at the gateway must be read by *both* resolvers
+ and every gateway variable the resolvers read must be set by compose; and a
+ service handed the gateway URL must be handed the gateway key. Wired into
+ `ci.yml :: python-lint`, with the self-test running first.
+
+ Every parser is AST- or comment-aware, and the self-test proves each refusal
+ rather than asserting it, because all three corpora contain text shaped
+ exactly like what the gate looks for: the gateway config's commented
+ Ollama/vLLM examples repeat `model_name: aisoc-triage` verbatim,
+ `.env.example` documents the escape hatch as a commented
+ `AISOC_MODEL_PIN_TRIAGE=gpt-4o-mini`, and `docker-compose.yml` explains the
+ gateway in prose containing `OPENAI_BASE_URL=http://litellm:4000/v1` two lines
+ above the key it describes. A line regex credits all three. 13 injected
+ defects and 7 parser blind spots, each caught by its own code, plus the shared
+ empty-tree refusal.
- **`GET /api/v1/graph` exists.** The console's Attack Graph view has always
called it and always got a 404, while `/graph/neighbors/…` and
`/graph/mitre-coverage` returned 200 from the same Neo4j instance holding
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index 48e148abf..a44a97c0f 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -33,7 +33,7 @@ and says so.
| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` |
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
-The last row is the one worth reading first. It is the honest index: 129 rows
+The last row is the one worth reading first. It is the honest index: 130 rows
`GATED`, 8 `PARTIAL` with the specific gap named, and a ratchet that refuses
to let a row sit at `NO GATE`.
diff --git a/apps/docs/docs/deployment/env-vars.md b/apps/docs/docs/deployment/env-vars.md
index 6e355f33b..2083c54d7 100644
--- a/apps/docs/docs/deployment/env-vars.md
+++ b/apps/docs/docs/deployment/env-vars.md
@@ -210,8 +210,13 @@ Source: [`services/agents/app/`](https://github.com/beenuar/AiSOC/tree/main/serv
| Variable | Default | Description |
|----------|---------|-------------|
-| `OPENAI_API_KEY` | — | **Required.** OpenAI key used by the investigator and copilot agents |
-| `OPENAI_MODEL` | `gpt-4o` | LLM identifier — set per agent if you need different models |
+| `OPENAI_API_KEY` | — | Provider key. With the bundled gateway running, the *gateway* uses this to reach the upstream model; AiSOC itself authenticates with `LITELLM_MASTER_KEY`. |
+| `LLM_GATEWAY_URL` | `http://litellm:4000/v1` (set by compose) | In-network URL of the bundled LiteLLM gateway. Both resolvers read it for any `aisoc-<role>` alias — an alias resolves nowhere else. A concrete `AISOC_MODEL_PIN_<ROLE>` is left pointing at its provider. |
+| `LITELLM_MASTER_KEY` | `sk-aisoc-local` in compose | Bearer AiSOC sends when it routes to the gateway itself. Resolved *with* the URL, so a provider key is never sent to the gateway (it rejects one). |
+| `OPENAI_BASE_URL` / `LLM_BASE_URL` | — | Explicit override. Outranks `LLM_GATEWAY_URL` for every role; you own the key pairing when you set it. |
+| `AISOC_MODEL_PIN_<ROLE>` | `aisoc-<role>` | Per-role model. Set a concrete provider model to call a provider directly with no gateway. Roles: `triage`, `recon`, `investigation`, `copilot`, `summary`, `report`, `nl`. |
+| `OPENAI_MODEL` | `aisoc-summary` | BYOK / "explain this alert" path **only** — never a task role's model. Ships as an alias because the default deployment routes through the gateway, which 400s anything `infra/litellm/config.yaml` does not define. |
+| `AISOC_EMBEDDING_BASE_URL` / `AISOC_EMBEDDING_MODEL` | — / `text-embedding-3-large` | MITRE RAG embeddings. Deliberately outside the gateway, whose model list is chat aliases only. |
| `DATABASE_URL` | `postgresql+asyncpg://aisoc:aisoc@localhost:5432/aisoc` | Postgres DSN for the Investigation Ledger |
| `QDRANT_URL` | `http://localhost:6333` | Vector store for case memory and RAG |
| `ENRICHMENT_SERVICE_URL` | `http://enrichment:8011` | URL of the enrichment service (Go) |
@@ -449,8 +454,11 @@ AISOC_CORS_ORIGINS=http://localhost:3000
JWT_SECRET=$(openssl rand -hex 32)
# --- Agents ---
+# The gateway reads OPENAI_API_KEY; AiSOC authenticates to the gateway with
+# LITELLM_MASTER_KEY. Routing is already wired — compose sets LLM_GATEWAY_URL
+# on api and agents, and both resolvers read it.
OPENAI_API_KEY=sk-...
-OPENAI_MODEL=gpt-4o
+LITELLM_MASTER_KEY=$(openssl rand -hex 32)
ENRICHMENT_SERVICE_URL=http://enrichment:8011
# --- Realtime ---
diff --git a/apps/docs/docs/operations/faq.md b/apps/docs/docs/operations/faq.md
index d9040aca7..368f8d90c 100644
--- a/apps/docs/docs/operations/faq.md
+++ b/apps/docs/docs/operations/faq.md
@@ -101,7 +101,7 @@ Postgres and Kafka are the natural ceiling; both scale predictably under standar
### Which model does the agent use?
-Whichever you configure. `OPENAI_MODEL` (and per-agent overrides) is read at startup. Default is `gpt-4o`. Anthropic Claude works via the agents service's provider abstraction. Self-hosted Llama / Mistral works via OpenAI-compatible endpoints (vLLM, Ollama, llama.cpp server).
+Whichever you configure, and the choice lives in one file. Each workload asks for a logical alias (`aisoc-triage`, `aisoc-investigation`, …) and `infra/litellm/config.yaml` maps aliases to real models — so you re-point a task without touching AiSOC. Hosted providers, Anthropic Claude, and self-hosted Llama / Mistral behind an OpenAI-compatible endpoint (vLLM, Ollama, llama.cpp server) are all just a different line in that file. To skip the gateway entirely, pin a concrete model per role with `AISOC_MODEL_PIN_<ROLE>`. See [LLM gateway](./llm-gateway.md).
### Will the agent do things without my permission?
diff --git a/apps/docs/docs/operations/llm-gateway.md b/apps/docs/docs/operations/llm-gateway.md
index e2b149e93..a6dc267be 100644
--- a/apps/docs/docs/operations/llm-gateway.md
+++ b/apps/docs/docs/operations/llm-gateway.md
@@ -51,30 +51,73 @@ point is that you change it. The alias names stay constant.
## Enable the gateway
-The `litellm` service is defined in `docker-compose.yml` and starts with the
-stack. To route AiSOC through it, set in your `.env`:
+The `litellm` service is part of the `full` profile in `docker-compose.yml`, and
+routing to it is already wired. Put your provider key in `.env` and bring the
+profile up:
```bash
+OPENAI_API_KEY=<your-real-provider-key> # the gateway uses this to reach the upstream model
LITELLM_MASTER_KEY=<a-strong-key> # AiSOC authenticates to the gateway with this
-OPENAI_API_KEY=<your-real-provider-key> # LiteLLM uses this to reach the upstream model
-OPENAI_BASE_URL=http://litellm:4000/v1 # send AiSOC's calls to the gateway
-# and set AiSOC's client key to the gateway key:
-# OPENAI_API_KEY=${LITELLM_MASTER_KEY} # (in the AiSOC services' environment)
```
-AiSOC now requests a task **alias** for every live call, so an alias only
-resolves when it reaches the gateway. If you don't run the gateway, pin each
-role to a concrete provider model instead (the **escape hatch**):
+```bash
+docker compose --profile full up -d
+```
+
+That is the whole configuration. `docker-compose.yml` sets `LLM_GATEWAY_URL` on
+the `api` and `agents` services, and both services' resolvers read it for any
+`aisoc-<role>` alias — an alias resolves nowhere else, so the gateway is the
+only correct destination for one. The bearer token is resolved with the route:
+when AiSOC picks the gateway itself it sends `LITELLM_MASTER_KEY`, never a
+provider key, which the gateway would reject as an invalid proxy token.
+
+To send AiSOC somewhere else — a gateway on another host, or your own
+OpenAI-compatible endpoint — set `OPENAI_BASE_URL` (or `LLM_BASE_URL`) and it
+outranks `LLM_GATEWAY_URL` for every role.
+
+### Calling a provider directly, with no gateway
+
+Pin each role to a concrete provider model. A concrete pin is **not** rerouted
+to the gateway, so this keeps working with `LLM_GATEWAY_URL` set:
```bash
AISOC_MODEL_PIN_TRIAGE=gpt-4o-mini
AISOC_MODEL_PIN_INVESTIGATION=gpt-4o
# … one per role: triage, recon, investigation, copilot, summary, report, nl
+OPENAI_MODEL=gpt-4o-mini # the BYOK / "explain this alert" path, direct too
```
With neither the gateway nor pin overrides configured, AiSOC uses its
-deterministic offline path. (`OPENAI_MODEL` still applies to the separate
-"explain this alert" / BYOK path.)
+deterministic offline path, which reports zero model calls and zero tokens
+rather than presenting a heuristic as triage.
+
+### A model the gateway does not define is an error, not a fallback
+
+Two failures used to be indistinguishable from "no LLM available", because both
+arrived as an exception a caller turned into a deterministic result:
+
+- An `aisoc-<role>` alias sent to a provider default, which does not know the
+ name. AiSOC now raises `UnroutableModelError` naming the alias and the remedy
+ instead of building a client destined to fail.
+- A concrete model sent to the gateway, which answers `Invalid model name`.
+ `preflight_llm()` reports this at boot, in both directions, and
+ `scripts/check_llm_model_routing.py` fails the build when a model named in
+ `.env.example` or `docker-compose.yml` is not one `infra/litellm/config.yaml`
+ defines — or when the gateway defines an alias no role requests.
+
+`OPENAI_MODEL` applies to the BYOK / "explain this alert" path **only**. It is
+never a task role's model: the auto-triage worker layers a tenant's BYOK
+configuration over the role pin, and only the fields the *tenant* set are
+treated as overrides, so an environment default cannot displace an alias.
+
+### Embeddings
+
+The MITRE RAG path calls an embedding endpoint, and `infra/litellm/config.yaml`
+declares chat aliases only — so embeddings deliberately do **not** follow chat
+traffic to the gateway. They have their own pair of variables,
+`AISOC_EMBEDDING_BASE_URL` and `AISOC_EMBEDDING_MODEL` (default
+`text-embedding-3-large`), and default to the provider. The routing gate checks
+that exclusion in both directions.
## Re-point a task to a local model
diff --git a/docker-compose.yml b/docker-compose.yml
index 7d12268bf..2e57cd11d 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -344,10 +344,17 @@ services:
SECRET_KEY: ${SECRET_KEY:-dev_secret_key_change_in_production}
ENVIRONMENT: development
LOG_LEVEL: info
- # Issue #478 — LiteLLM gateway. Route this service's live LLM endpoints
- # (translation, hunts, KB, phishing, NL-detection) through the gateway by
- # setting OPENAI_BASE_URL=http://litellm:4000/v1 + OPENAI_API_KEY=$LITELLM_MASTER_KEY.
+ # Issue #478 — LiteLLM gateway. This service's live LLM endpoints
+ # (translation, hunts, KB, phishing, NL-detection) route here by default:
+ # `app.services.model_aliases.resolve_base_url` reads LLM_GATEWAY_URL for
+ # any aisoc-<role> alias. It was previously read by nothing, so the
+ # variable was set, documented, and dead — every alias went to the
+ # provider default and 404'd. OPENAI_BASE_URL still overrides it.
LLM_GATEWAY_URL: http://litellm:4000/v1
+ # The bearer that pairs with the URL above. Without it AiSOC would send a
+ # provider key to the gateway, which rejects it as an invalid proxy token.
+ LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:-sk-aisoc-local}
+ OPENAI_API_KEY: ${OPENAI_API_KEY:-}
# services/actions — where an approved response action is executed.
# The API had no pointer to it at all: the one endpoint that called it
# read the URL from os.environ and fell back to `aisoc-actions`, which
@@ -516,13 +523,20 @@ services:
NEO4J_PASSWORD: neo4j_dev_secret
OPENAI_API_KEY: ${OPENAI_API_KEY:-}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
- OPENAI_MODEL: ${OPENAI_MODEL:-gpt-4o-mini}
+ # BYOK / "explain this alert" only — never a task role. A task role asks
+ # `app.llm.model_pins` for its aisoc-<role> alias, and this leaking into
+ # that path is what replaced `aisoc-triage` with a model the gateway does
+ # not define. Defaults to the summary alias so the explain path resolves
+ # at the gateway too; set a concrete model only for direct-to-provider.
+ OPENAI_MODEL: ${OPENAI_MODEL:-aisoc-summary}
# Issue #478 — the LiteLLM gateway (below) is the single entry point for
- # live LLM calls when enabled. To route this service through it, set in
- # .env: OPENAI_BASE_URL=http://litellm:4000/v1 and
- # OPENAI_API_KEY=$LITELLM_MASTER_KEY. Left unset, calls go direct to the
- # provider (unchanged behaviour).
+ # live LLM calls. `app.llm.factory.resolve_base_url` reads this for any
+ # aisoc-<role> alias, so the default deployment reaches the gateway with
+ # no further configuration; OPENAI_BASE_URL still overrides it, and a
+ # concrete AISOC_MODEL_PIN_<ROLE> is left pointing at its provider.
LLM_GATEWAY_URL: http://litellm:4000/v1
+ # The bearer that pairs with the URL above — see the api service.
+ LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:-sk-aisoc-local}
ATTCK_DATA_PATH: /app/data/enterprise-attack.json
# Where to post an approval card when triage raises one. Unset means
# no ChatOps notification; the approval is still durable and visible in
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index fa2c73cb5..9bed5da5a 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -144,11 +144,12 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| What counts as a check is decided by what a script does, not what it is named | `scripts/check_gate_coverage.py` (the inventory it reports on) | `ci.yml` (three structural signals, all read from the tree: a declared verdict flag, an exit status derived from findings the script accumulates, and a workflow job whose output another job branches on. The first two are intrinsic, so an uncalled gate is still inventoried and the `SCRIPT -> WORKFLOW` direction is not vacuous. The classifier is also checked in reverse — a script CI runs with `--check` that the classifier does not inventory fails as `classifier-blind-spot`) + `tests/test_gate_coverage_classifier.py` (10 cases, including a gate planted under a name no convention would reveal, and a generator that aborts on an empty read asserted *not* to be a check) | GATED | - |
| No check reports OK over a repository with no content | every row above (a gate that credits an empty tree proves nothing about a full one) | `ci.yml :: python-test` (`scripts/check_gate_contract.py` — runs all 72 inventoried checks the way a workflow runs each one, inside a git repository holding only `scripts/`, and requires a non-zero exit. Two tree shapes, because they are two questions: **bare** omits the directories a gate judges, **skeleton** creates them empty. The second is the one that happens — a renamed package, a glob that stopped matching — and a gate opening with `if not X.is_dir(): return 2` answers the first for a reason that says nothing about its corpus. Measured before it was added: 64 of 71 checks already refused the skeleton, so the exception list stayed readable rather than doubling. A run that failed for a reason the empty tree did not cause — an argparse usage error, or an import of a module the repository does not supply — is INCONCLUSIVE and fails rather than counting as a refusal, so the gate cannot credit a failure it did not cause. Exceptions are recorded with the disposition they are excused for and checked in both directions: an entry naming nothing, and an entry whose gate has started behaving differently, both fail) + `tests/test_gate_contract.py` (45 cases) | GATED | 5 recorded exceptions, keyed by tree shape; the probe cannot pose the question to a gate whose only input is a path named on the command line, or one that lives inside `scripts/`. One entry exists because its dispositions legitimately differ: `check_logger_kwargs.py` scans every Python file in the repository, so against a skeleton it has a real corpus in `scripts/` and correctly finds nothing wrong |
| Every check resolves its root from git and carries a `--self-test` | `scripts/` (the gates themselves) | `ci.yml :: python-test` (`scripts/check_gate_contract.py` — a check that derives a repository root from `__file__` without consulting git fails as `root-from-file`, and one that declares no `--self-test` fails as `no-self-test`. Both read from the syntax tree: the first spelling matched `repo_root` anywhere in the text and `check_gate_coverage.py` emits it as a JSON key, so a non-compliant gate read as compliant on the strength of a dictionary key in its own output. Both exception lists are empty — all 72 comply — and an entry added to either must carry a reason) | GATED | - |
+| Every model AiSOC sends has a gateway that defines it, and the variable compose supplies is the variable the resolvers read | `apps/docs/docs/operations/llm-gateway.md`, `.env.example`, README (AI triage) | `ci.yml :: python-lint` (`scripts/check_llm_model_routing.py` — both directions on every name: each role pin's alias must be defined by `infra/litellm/config.yaml` and each alias must be claimed by a pin; every model named in `.env.example` or `docker-compose.yml`, in a file that wires the bundled gateway, must be one the gateway defines; every variable compose points at the gateway must be read by *both* resolvers, and every gateway variable the resolvers read must be set by compose — the defect that shipped was compose supplying `LLM_GATEWAY_URL` and both resolvers deliberately ignoring it, so AI triage could not reach a model at all; a service given the gateway URL must also get `LITELLM_MASTER_KEY`, because a provider key sent there is rejected. Every parser is AST- or comment-aware and the self-test proves it: the config's commented Ollama example repeats `model_name: aisoc-triage` verbatim, `.env.example` documents the escape hatch as a commented `AISOC_MODEL_PIN_TRIAGE=`, and compose explains the gateway in prose two lines above the key it describes — a line regex credits all three) + `services/agents/tests/test_llm_factory.py` + `services/api/tests/test_model_aliases_routing.py` | GATED | Embeddings are a recorded exclusion — the gateway's model list is chat aliases only — and the gate checks that in both directions rather than trusting the comment |
| Every test file in the tree is executed by a workflow | `CONTRIBUTING.md` / the suites themselves | `ci.yml :: python-test` (`scripts/check_test_discovery.py` — the corpus is `git ls-files`, so a service added tomorrow is scanned with no edit, and an untracked install tree is out by construction rather than by a name somebody remembered. Reach is modelled the way pytest collects: directory arguments, `--ignore`, `conftest` `collect_ignore`, and `-m` marker deselection, because a file whose every test is deselected runs nothing however plainly it is named. Both directions — an invocation naming a path that is not in the tree fails too. Quarantine is a shrink-only list where each entry carries a reason and an entry that has become reachable fails as stale. Two blind spots in this parser were found by counting what it **credited**: a step that `cd`s into a service and then names `tests/` had its paths resolved against the repository root, which reported all 82 agents files unreached *and* credited the root tree with a package's invocation; and `-m` was first skipped as "a filter, not a collection rule", which would have credited a file in full under `-m "not integration"`) + `tests/test_test_discovery_gate.py` | GATED | - |
## Summary
-- GATED: 129
+- GATED: 130
- PARTIAL: 8
- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)
diff --git a/scripts/check_llm_model_routing.py b/scripts/check_llm_model_routing.py
new file mode 100644
index 000000000..02b05b626
--- /dev/null
+++ b/scripts/check_llm_model_routing.py
@@ -0,0 +1,858 @@
+#!/usr/bin/env python3
+"""A model name must have somewhere to go, and the variable that sends it there must be read.
+
+Why this exists
+---------------
+AI triage is this product's headline capability and it could not reach a model
+in the default deployment. Not for want of a key — for two wiring defects that
+each looked fine in isolation:
+
+``docker-compose.yml`` set ``LLM_GATEWAY_URL`` on the api and agents services.
+Both resolvers deliberately ignored it, honouring only ``OPENAI_BASE_URL``,
+which compose never set. So the one variable the deployment supplied was read by
+nothing, every ``aisoc-<role>`` alias went to ``api.openai.com``, and the 404
+came back through a caller's ``except`` as "no LLM available". The service even
+logged the diagnosis at boot; the diagnosis existed and the wiring did not.
+
+And ``.env.example`` shipped ``OPENAI_MODEL=gpt-4-turbo-preview`` — a model no
+gateway config in this tree has ever defined — which the auto-triage worker
+picked up as a per-tenant BYOK override and sent in place of ``aisoc-triage``.
+The commercial deployment hit the same shape from the other end: the model names
+its LiteLLM did not know produced "Invalid model name" on every call and the
+whole thing degraded to empty output across the copilot and every triage agent.
+
+Both are unreachability, and both are structural: a name that resolves nowhere,
+or a variable nothing reads. Neither is visible in a diff.
+
+What it checks, in both directions
+----------------------------------
+The dominant bug shape in this repository is the one-directional check: it
+compares A against B and never B against A, so drift in the direction things
+actually change slips through while the check prints OK. Every rule below is a
+resolution requirement on a specific name, and both name spaces are enumerated.
+
+ PIN -> GW every role pin's shipped alias is defined by the gateway
+ GW -> PIN every gateway alias is claimed by a role pin; an orphan is a
+ model the operator maintains and nothing requests
+ AGENTS<->API the role set and the routing surface agree across the two
+ packages, which cannot import one another and so drift
+ CONFIG-> GW every router fallback names a defined alias, in both the key
+ and the value position
+ ENV -> GW every model an env/compose file names, in a file that wires
+ the bundled gateway, is one the gateway defines
+ CMP -> CODE every variable compose points at the bundled gateway is read
+ by both resolvers. This is the defect above, stated as a rule
+ CODE -> CMP every gateway variable the resolvers read is set by compose on
+ the services that call models. The same defect, mirrored: a
+ read with nothing to read is as dead as a write nobody reads
+ PAIRING a service handed the gateway URL is handed the gateway's key.
+ Resolving the two independently is what made adopting the URL
+ look ambiguous; a provider key sent to LiteLLM is rejected
+ SCOPE embeddings are recorded as outside the gateway (its model list
+ is seven chat aliases), and the exclusion is checked in both
+ directions rather than asserted in a comment
+
+Usage
+-----
+ python3 scripts/check_llm_model_routing.py # gate
+ python3 scripts/check_llm_model_routing.py --json
+ python3 scripts/check_llm_model_routing.py --self-test # prove it bites
+
+``--repo-root`` overrides the tree under inspection. The resolved root comes
+from ``git rev-parse``, every file read and every count is printed before the
+verdict, and a missing or empty input is a hard error rather than a quiet zero:
+a gate that reports OK about a tree it never opened is worse than no gate.
+
+Stdlib only. The lint job that runs this installs ruff and mypy and nothing
+else, and a gate that needs a ``pip install`` to render a verdict is a gate that
+can be skipped.
+
+Exit codes: 0 clean, 1 findings, 2 the scan itself could not run.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import copy
+import json
+import re
+import sys
+from collections.abc import Callable
+from dataclasses import dataclass, field
+from pathlib import Path
+
+# `scripts/` is on sys.path when this file is run as a program, but not when a
+# test loads it by path with importlib. gate_toolkit sits beside it either way.
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root, self_test_main # noqa: E402
+
+CONFIG_REL = "infra/litellm/config.yaml"
+PINS_REL = "services/agents/app/llm/model_pins.py"
+AGENTS_ROUTING_REL = "services/agents/app/llm/routing.py"
+API_ROUTING_REL = "services/api/app/services/model_aliases.py"
+ENV_EXAMPLE_REL = ".env.example"
+COMPOSE_REL = "docker-compose.yml"
+EMBEDDING_REL = "services/agents/app/tools/mitre_full.py"
+
+#: Env vars whose value is a model name. Read from the tree where possible;
+#: this is the shape that identifies one in a compose or dotenv file, where
+#: there is no code to parse.
+MODEL_VAR_RE = re.compile(r"^(OPENAI_MODEL|LLM_MODEL|AISOC_LLM_MODEL|AISOC_MODEL_PIN_[A-Z0-9_]+)$")
+
+#: How the bundled gateway is recognised in a compose or dotenv value. Matched
+#: on the service name compose gives it, so a port change does not blind this.
+BUNDLED_GATEWAY_RE = re.compile(r"https?://litellm(:\d+)?(/|$)")
+
+#: Compose services that make LLM calls, and so need both halves of the pair.
+#: Derived below from the services that set a gateway variable *or* a model
+#: variable, so adding a third LLM service does not require editing this file.
+GATEWAY_KEY_VAR = "LITELLM_MASTER_KEY"
+
+#: Embeddings are out of the gateway's scope: ``infra/litellm/config.yaml``
+#: declares chat aliases only, so routing an embedding call there 400s on every
+#: batch. Recorded here so the exclusion is a checked property rather than a
+#: comment, and enforced in both directions below.
+EMBEDDING_MODEL_SHAPES = ("embedding", "embed-")
+
+
+class GateError(RuntimeError):
+ """The scan could not be performed — distinct from the scan finding nothing."""
+
+
+@dataclass
+class RoutingModule:
+ """What one routing module defines, declares as the rule, and reads."""
+
+ defined: set[str] = field(default_factory=set)
+ declared: set[str] = field(default_factory=set)
+ reads: dict[str, set[str]] = field(default_factory=dict)
+
+ @property
+ def all_reads(self) -> set[str]:
+ return set().union(*self.reads.values()) if self.reads else set()
+
+
+@dataclass
+class Corpus:
+ """Everything the gate read, named, so a caller cannot lose track of which tree."""
+
+ gateway_aliases: set[str] = field(default_factory=set)
+ fallbacks: dict[str, list[str]] = field(default_factory=dict)
+ role_pins: dict[str, str] = field(default_factory=dict)
+ api_roles: set[str] = field(default_factory=set)
+ agents_routing: RoutingModule = field(default_factory=RoutingModule)
+ api_routing: RoutingModule = field(default_factory=RoutingModule)
+ env_example: dict[str, str] = field(default_factory=dict)
+ compose: dict[str, dict[str, str]] = field(default_factory=dict)
+ embedding_default: str = ""
+
+
+# --------------------------------------------------------------------------
+# Parsing. Every parser below is comment-aware or AST-based, never a bare
+# regex over raw source, because both of this gate's corpora contain text that
+# looks exactly like the thing it is looking for:
+#
+# * ``infra/litellm/config.yaml`` ends with commented Ollama / vLLM /
+# Anthropic examples that repeat ``model_name: aisoc-triage`` verbatim. A
+# line regex credits three aliases the gateway does not serve.
+# * ``.env.example`` documents the direct-to-provider escape hatch as
+# ``# AISOC_MODEL_PIN_TRIAGE=gpt-4o-mini``. A line regex reads that
+# as a live assignment and fails the build over documentation.
+# * ``docker-compose.yml`` explains the gateway in prose containing
+# ``OPENAI_BASE_URL=http://litellm:4000/v1`` two lines above the key it is
+# describing.
+#
+# The self-test injects each of those three shapes and requires the parser to
+# decline to credit it.
+# --------------------------------------------------------------------------
+
+
+def strip_comment(line: str) -> str:
+ """Drop a trailing ``#`` comment, respecting quotes."""
+ out: list[str] = []
+ quote: str | None = None
+ for ch in line:
+ if quote:
+ out.append(ch)
+ if ch == quote:
+ quote = None
+ continue
+ if ch in "\"'":
+ quote = ch
+ out.append(ch)
+ continue
+ if ch == "#":
+ break
+ out.append(ch)
+ return "".join(out).rstrip()
+
+
+def _unquote(value: str) -> str:
+ value = value.strip()
+ if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
+ return value[1:-1]
+ return value
+
+
+def parse_gateway_aliases(text: str) -> set[str]:
+ """Every ``model_name`` the gateway actually serves."""
+ aliases: set[str] = set()
+ for raw in text.splitlines():
+ line = strip_comment(raw)
+ m = re.match(r"^\s*-?\s*model_name:\s*(\S+)\s*$", line)
+ if m:
+ aliases.add(_unquote(m.group(1)))
+ return aliases
+
+
+def parse_gateway_fallbacks(text: str) -> dict[str, list[str]]:
+ """``router_settings.fallbacks`` as ``{alias: [alias, ...]}``."""
+ fallbacks: dict[str, list[str]] = {}
+ in_block = False
+ for raw in text.splitlines():
+ line = strip_comment(raw)
+ if not line.strip():
+ continue
+ if re.match(r"^\s*fallbacks:\s*$", line):
+ in_block = True
+ continue
+ if in_block:
+ m = re.match(r"^\s*-\s*([A-Za-z0-9_.\-]+):\s*\[(.*)\]\s*$", line)
+ if not m:
+ # Any non-item line ends the block; a `fallbacks:` entry is a
+ # flat list of one-key maps in this file.
+ if not re.match(r"^\s*-\s", line):
+ in_block = False
+ continue
+ targets = [_unquote(t) for t in m.group(2).split(",") if t.strip()]
+ fallbacks[m.group(1)] = targets
+ return fallbacks
+
+
+def parse_role_pins(source: str) -> dict[str, str]:
+ """``_DEFAULT_PINS`` as ``{role: shipped primary model}``, read as code.
+
+ AST rather than regex: ``model_pins.py`` names ``aisoc-triage`` in its
+ module docstring and in an inline comment explaining the escape hatch, and
+ a gate that credits a variable mentioned in prose is a gate that credits
+ documentation for the behaviour it describes.
+ """
+ tree = ast.parse(source)
+ pins: dict[str, str] = {}
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.AnnAssign | ast.Assign):
+ continue
+ targets = [node.target] if isinstance(node, ast.AnnAssign) else node.targets
+ if not any(isinstance(t, ast.Name) and t.id == "_DEFAULT_PINS" for t in targets):
+ continue
+ if not isinstance(node.value, ast.Dict):
+ continue
+ for key, value in zip(node.value.keys, node.value.values, strict=False):
+ if not (isinstance(key, ast.Constant) and isinstance(key.value, str)):
+ continue
+ if not isinstance(value, ast.Call) or len(value.args) < 2:
+ continue
+ primary = value.args[1]
+ if isinstance(primary, ast.Constant) and isinstance(primary.value, str):
+ pins[key.value] = primary.value
+ return pins
+
+
+def parse_api_roles(source: str) -> set[str]:
+ """``ROLES`` from the API-side mirror, read as code."""
+ tree = ast.parse(source)
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Assign):
+ continue
+ if not any(isinstance(t, ast.Name) and t.id == "ROLES" for t in node.targets):
+ continue
+ for sub in ast.walk(node.value):
+ if isinstance(sub, ast.Set):
+ return {e.value for e in sub.elts if isinstance(e, ast.Constant) and isinstance(e.value, str)}
+ return set()
+
+
+def _env_reads(node: ast.AST) -> set[str]:
+ """Env var names read by an actual call under ``node``, not mentioned in prose."""
+ names: set[str] = set()
+ for sub in ast.walk(node):
+ if isinstance(sub, ast.Call):
+ func = sub.func
+ target = None
+ if isinstance(func, ast.Attribute) and func.attr in ("getenv", "get"):
+ owner = func.value
+ if isinstance(owner, ast.Name) and owner.id == "os":
+ target = sub.args[0] if sub.args else None
+ elif isinstance(owner, ast.Attribute) and owner.attr == "environ":
+ target = sub.args[0] if sub.args else None
+ if isinstance(target, ast.Constant) and isinstance(target.value, str):
+ names.add(target.value)
+ elif isinstance(sub, ast.Subscript):
+ owner = sub.value
+ if isinstance(owner, ast.Attribute) and owner.attr == "environ" and isinstance(sub.slice, ast.Constant):
+ if isinstance(sub.slice.value, str):
+ names.add(sub.slice.value)
+ return names
+
+
+def parse_routing_module(source: str) -> RoutingModule:
+ """Names defined, names declared in ``__all__``, and the env vars each function reads."""
+ tree = ast.parse(source)
+ functions: dict[str, set[str]] = {}
+ defined: set[str] = set()
+ declared: set[str] = set()
+ for node in tree.body:
+ if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef):
+ functions[node.name] = _env_reads(node)
+ defined.add(node.name)
+ elif isinstance(node, ast.ClassDef):
+ defined.add(node.name)
+ elif isinstance(node, ast.Assign):
+ names = {t.id for t in node.targets if isinstance(t, ast.Name)}
+ defined |= names
+ if "__all__" in names and isinstance(node.value, ast.List | ast.Tuple):
+ declared = {e.value for e in node.value.elts if isinstance(e, ast.Constant) and isinstance(e.value, str)}
+ return RoutingModule(defined=defined, declared=declared, reads=functions)
+
+
+def parse_env_file(text: str) -> dict[str, str]:
+ """Live ``KEY=VALUE`` assignments. A commented example is documentation."""
+ out: dict[str, str] = {}
+ for raw in text.splitlines():
+ line = raw.strip()
+ if not line or line.startswith("#"):
+ continue
+ m = re.match(r"^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)=(.*)$", line)
+ if m:
+ out[m.group(1)] = _unquote(strip_comment(m.group(2)))
+ return out
+
+
+def parse_compose_env(text: str) -> dict[str, dict[str, str]]:
+ """``{service: {VAR: value}}`` from a compose file's ``environment:`` blocks."""
+ services: dict[str, dict[str, str]] = {}
+ in_services = False
+ service: str | None = None
+ in_env = False
+ for raw in text.splitlines():
+ line = strip_comment(raw)
+ if not line.strip():
+ continue
+ if re.match(r"^services:\s*$", line):
+ in_services, service, in_env = True, None, False
+ continue
+ if re.match(r"^[A-Za-z_]", line): # another top-level key
+ in_services, service, in_env = False, None, False
+ continue
+ if not in_services:
+ continue
+ m = re.match(r"^ ([A-Za-z0-9_.\-]+):\s*$", line)
+ if m:
+ service, in_env = m.group(1), False
+ services.setdefault(service, {})
+ continue
+ if service is None:
+ continue
+ if re.match(r"^ environment:\s*$", line):
+ in_env = True
+ continue
+ if re.match(r"^ [A-Za-z0-9_.\-]+:", line): # sibling of environment:
+ in_env = False
+ continue
+ if not in_env:
+ continue
+ m = re.match(r"^ ([A-Z_][A-Z0-9_]*):\s*(.*)$", line)
+ if m:
+ services[service][m.group(1)] = _unquote(m.group(2))
+ continue
+ m = re.match(r"^ -\s*([A-Z_][A-Z0-9_]*)=(.*)$", line)
+ if m:
+ services[service][m.group(1)] = _unquote(m.group(2))
+ return services
+
+
+def parse_embedding_default(source: str) -> str:
+ """The model the RAG embedding path sends when nothing overrides it."""
+ tree = ast.parse(source)
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id == "_DEFAULT_EMBEDDING_MODEL" for t in node.targets):
+ if isinstance(node.value, ast.Constant) and isinstance(node.value.value, str):
+ return node.value.value
+ return ""
+
+
+# --------------------------------------------------------------------------
+# Rules
+# --------------------------------------------------------------------------
+
+
+def _resolve_value(value: str, env: dict[str, str]) -> str:
+ """Resolve ``${VAR:-default}`` / ``${VAR}`` against a dotenv baseline."""
+
+ def sub(m: re.Match[str]) -> str:
+ name, default = m.group(1), m.group(3) or ""
+ return env.get(name) or default
+
+ return re.sub(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(:-([^}]*))?\}", sub, value).strip()
+
+
+def evaluate(c: Corpus) -> list[tuple[str, str]]:
+ """Return ``(code, detail)`` for every unresolvable name. Empty means clean."""
+ out: list[tuple[str, str]] = []
+ gateway_aliases = c.gateway_aliases
+ fallbacks = c.fallbacks
+ role_pins = c.role_pins
+ api_roles = c.api_roles
+ agents_routing = c.agents_routing
+ api_routing = c.api_routing
+ env_example = c.env_example
+ compose = c.compose
+ embedding_default = c.embedding_default
+
+ # PIN -> GW / GW -> PIN.
+ claimed = set(role_pins.values())
+ for role, primary in sorted(role_pins.items()):
+ if primary not in gateway_aliases:
+ out.append(("pin-alias-undefined", f"role '{role}' ships primary '{primary}', which {CONFIG_REL} does not define"))
+ for alias in sorted(gateway_aliases - claimed):
+ out.append(("gateway-alias-unclaimed", f"{CONFIG_REL} defines '{alias}', which no role pin in {PINS_REL} requests"))
+
+ # AGENTS <-> API.
+ for role in sorted(set(role_pins) - api_roles):
+ out.append(("role-set-drift", f"'{role}' is pinned in {PINS_REL} but absent from ROLES in {API_ROUTING_REL}"))
+ for role in sorted(api_roles - set(role_pins)):
+ out.append(("role-set-drift", f"'{role}' is in ROLES in {API_ROUTING_REL} but has no pin in {PINS_REL}"))
+
+ # The mirror compared against the rule the agents module declares, in both
+ # directions. Equality of *every* name would be wrong — the API module also
+ # carries alias resolution, which the agents side gets from model_pins — so
+ # the contract is ``__all__``: the mirror may not lose a rule, and may not
+ # grow a routing rule the agents side does not have. Growth is detected by
+ # the reads, not by the name, because a new rule is one that consults the
+ # environment.
+ declared = agents_routing.declared
+ routing_vars = agents_routing.all_reads
+ api_defined = api_routing.defined
+ agents_defined = agents_routing.defined
+ for name in sorted(declared - agents_defined):
+ out.append(("routing-surface-drift", f"{AGENTS_ROUTING_REL} declares '{name}' in __all__ and does not define it"))
+ for name in sorted(declared - api_defined):
+ out.append(
+ ("routing-surface-drift", f"{AGENTS_ROUTING_REL} declares '{name}' as a routing rule and the {API_ROUTING_REL} mirror lacks it")
+ )
+ for name, reads in sorted(api_routing.reads.items()):
+ if reads & routing_vars and name.lstrip("_") not in declared:
+ out.append(
+ (
+ "routing-surface-drift",
+ f"{API_ROUTING_REL} defines '{name}', which reads {', '.join(sorted(reads & routing_vars))}, "
+ f"and {AGENTS_ROUTING_REL} does not declare it — the two services would route differently",
+ )
+ )
+
+ agents_reads = agents_routing.all_reads
+ api_reads = api_routing.all_reads
+ for name in sorted(agents_reads ^ api_reads):
+ side = AGENTS_ROUTING_REL if name in agents_reads else API_ROUTING_REL
+ other = API_ROUTING_REL if name in agents_reads else AGENTS_ROUTING_REL
+ out.append(("routing-read-drift", f"'{name}' is read by {side} and not by {other}; the two services would route differently"))
+
+ # CONFIG -> GW.
+ for key, targets in sorted(fallbacks.items()):
+ if key not in gateway_aliases:
+ out.append(("fallback-undefined", f"router fallback is keyed on '{key}', which {CONFIG_REL} does not define"))
+ for target in targets:
+ if target not in gateway_aliases:
+ out.append(("fallback-undefined", f"router fallback '{key}' falls back to '{target}', which {CONFIG_REL} does not define"))
+
+ # ENV -> GW. Only in a file that wires the bundled gateway: an operator
+ # pointing at their own vLLM with concrete model names is correct, and only
+ # this gateway's contents are knowable from here.
+ env_wires_gateway = any(BUNDLED_GATEWAY_RE.search(v) for v in env_example.values())
+ if env_wires_gateway:
+ for name, value in sorted(env_example.items()):
+ if MODEL_VAR_RE.match(name) and value and value not in gateway_aliases:
+ out.append(
+ (
+ "env-model-undefined",
+ f"{ENV_EXAMPLE_REL} sets {name}={value} while wiring the bundled gateway, which does not define it",
+ )
+ )
+ for service, env in sorted(compose.items()):
+ if not any(BUNDLED_GATEWAY_RE.search(v) for v in env.values()):
+ continue
+ for name, value in sorted(env.items()):
+ if not MODEL_VAR_RE.match(name):
+ continue
+ resolved = _resolve_value(value, env_example)
+ if resolved and resolved not in gateway_aliases:
+ out.append(
+ (
+ "env-model-undefined",
+ f"{COMPOSE_REL} service '{service}' sets {name}={value} (resolves to '{resolved}') "
+ f"while wiring the bundled gateway, which does not define it",
+ )
+ )
+
+ # CMP -> CODE and CODE -> CMP. The variable compose points at the gateway
+ # and the variable the resolvers read have to be the same variable.
+ read_everywhere = agents_reads & api_reads
+ gateway_services = {s for s, env in compose.items() if any(BUNDLED_GATEWAY_RE.search(v) for v in env.values())}
+ compose_gateway_vars: set[str] = set()
+ for service in gateway_services:
+ for name, value in compose[service].items():
+ if BUNDLED_GATEWAY_RE.search(value):
+ compose_gateway_vars.add(name)
+ for name in sorted(compose_gateway_vars - read_everywhere):
+ out.append(
+ (
+ "compose-var-unread",
+ f"{COMPOSE_REL} points {name} at the bundled gateway and neither resolver reads it; "
+ "the deployment supplies a route nothing can follow",
+ )
+ )
+ # The mirror. Every gateway variable the resolvers read has to be supplied,
+ # or the read is as dead as the write was.
+ code_gateway_vars = {n for n in read_everywhere if n.endswith("GATEWAY_URL")}
+ if not code_gateway_vars:
+ out.append(("code-var-unset", f"no *_GATEWAY_URL variable is read by both {AGENTS_ROUTING_REL} and {API_ROUTING_REL}"))
+ for name in sorted(code_gateway_vars):
+ if not any(name in compose[s] for s in compose):
+ out.append(("code-var-unset", f"both resolvers read {name} and no {COMPOSE_REL} service sets it"))
+
+ # PAIRING. The route decides the bearer; a provider key sent to LiteLLM is
+ # rejected as an invalid proxy token.
+ for service in sorted(gateway_services):
+ if GATEWAY_KEY_VAR not in compose[service]:
+ out.append(
+ (
+ "gateway-without-key",
+ f"{COMPOSE_REL} service '{service}' is pointed at the bundled gateway without {GATEWAY_KEY_VAR}; "
+ "it would authenticate with a provider key the gateway rejects",
+ )
+ )
+
+ # SCOPE. Embeddings are excluded from the gateway, in both directions.
+ if not embedding_default:
+ out.append(("embedding-scope", f"{EMBEDDING_REL} declares no default embedding model to classify"))
+ elif embedding_default.startswith("aisoc-"):
+ out.append(
+ (
+ "embedding-scope",
+ f"{EMBEDDING_REL} defaults to '{embedding_default}', a gateway alias — but {CONFIG_REL} "
+ "declares chat models only, so every embedding batch would 400",
+ )
+ )
+ for alias in sorted(gateway_aliases):
+ if any(shape in alias for shape in EMBEDDING_MODEL_SHAPES):
+ out.append(
+ (
+ "embedding-scope",
+ f"{CONFIG_REL} defines '{alias}', which looks like an embedding model; the exclusion "
+ f"recorded in {EMBEDDING_REL} says the gateway carries chat aliases only",
+ )
+ )
+ return out
+
+
+# --------------------------------------------------------------------------
+# I/O
+# --------------------------------------------------------------------------
+
+
+def load(root: Path) -> Corpus:
+ paths = {
+ "config": root / CONFIG_REL,
+ "pins": root / PINS_REL,
+ "agents_routing": root / AGENTS_ROUTING_REL,
+ "api_routing": root / API_ROUTING_REL,
+ "env_example": root / ENV_EXAMPLE_REL,
+ "compose": root / COMPOSE_REL,
+ "embedding": root / EMBEDDING_REL,
+ }
+ for name, path in paths.items():
+ if not path.is_file():
+ raise GateError(f"expected input does not exist: {path} ({name})")
+
+ config_text = paths["config"].read_text(encoding="utf-8")
+ corpus = Corpus(
+ gateway_aliases=parse_gateway_aliases(config_text),
+ fallbacks=parse_gateway_fallbacks(config_text),
+ role_pins=parse_role_pins(paths["pins"].read_text(encoding="utf-8")),
+ api_roles=parse_api_roles(paths["api_routing"].read_text(encoding="utf-8")),
+ agents_routing=parse_routing_module(paths["agents_routing"].read_text(encoding="utf-8")),
+ api_routing=parse_routing_module(paths["api_routing"].read_text(encoding="utf-8")),
+ env_example=parse_env_file(paths["env_example"].read_text(encoding="utf-8")),
+ compose=parse_compose_env(paths["compose"].read_text(encoding="utf-8")),
+ embedding_default=parse_embedding_default(paths["embedding"].read_text(encoding="utf-8")),
+ )
+ # An empty parse means the format moved, not that the tree is clean. Found
+ # nothing and scanned nothing print the same word unless one of them refuses.
+ empty = [
+ name for name in ("gateway_aliases", "fallbacks", "role_pins", "api_roles", "env_example", "compose") if not getattr(corpus, name)
+ ]
+ if empty:
+ raise GateError(f"parsed zero {empty[0]} — refusing to report a clean tree from an empty read")
+ for side in ("agents_routing", "api_routing"):
+ module: RoutingModule = getattr(corpus, side)
+ if not module.defined:
+ raise GateError(f"parsed zero definitions from {side} — refusing to report a clean tree from an empty read")
+ if not corpus.agents_routing.declared:
+ raise GateError(f"{AGENTS_ROUTING_REL} declares no __all__ — the routing contract the mirror is checked against is missing")
+ return corpus
+
+
+def _summary(c: Corpus, root: Path) -> list[str]:
+ gateway_services = sorted(s for s, env in c.compose.items() if any(BUNDLED_GATEWAY_RE.search(v) for v in env.values()))
+ reads = ", ".join(sorted(c.agents_routing.all_reads))
+ wired = ", ".join(gateway_services)
+ return [
+ f"repo root {root}",
+ f"gateway config {CONFIG_REL} ({len(c.gateway_aliases)} aliases, {len(c.fallbacks)} router fallbacks)",
+ f"role pins {PINS_REL} ({len(c.role_pins)} roles)",
+ f"agents routing {AGENTS_ROUTING_REL} ({len(c.agents_routing.declared)} declared rules, reads {reads})",
+ f"api routing {API_ROUTING_REL} ({len(c.api_routing.defined)} definitions, {len(c.api_roles)} roles)",
+ f"env example {ENV_EXAMPLE_REL} ({len(c.env_example)} live assignments)",
+ f"compose {COMPOSE_REL} ({len(c.compose)} services, {len(gateway_services)} on the bundled gateway: {wired})",
+ f"embedding scope {EMBEDDING_REL} (default '{c.embedding_default}', excluded from the gateway)",
+ ]
+
+
+# --------------------------------------------------------------------------
+# Self-test
+# --------------------------------------------------------------------------
+
+
+def self_test(root: Path) -> int:
+ """Prove the gate still catches what it claims, and still declines prose.
+
+ Two kinds of case. The first injects a defect into the parsed data and
+ requires the matching code. The second feeds the *parsers* text shaped like
+ the thing they look for but commented out, and requires them not to credit
+ it — because every blind spot found in this tree so far was something a
+ gate credited, not something it failed to flag.
+ """
+ try:
+ base = load(root)
+ except GateError as exc:
+ print(f"self-test cannot run: {exc}", file=sys.stderr)
+ return 2
+
+ baseline = evaluate(base)
+ if baseline:
+ print("self-test cannot run: the clean tree already has findings:", file=sys.stderr)
+ for code, detail in baseline:
+ print(f" [{code}] {detail}", file=sys.stderr)
+ return 2
+
+ def drop_gateway_read(c: Corpus) -> None:
+ for reads in c.api_routing.reads.values():
+ reads.discard("LLM_GATEWAY_URL")
+
+ def add_unset_gateway_var(c: Corpus) -> None:
+ for module in (c.agents_routing, c.api_routing):
+ module.reads.setdefault("gateway_url", set()).add("AISOC_GATEWAY_URL")
+
+ def drop_gateway_key(c: Corpus) -> None:
+ c.compose["agents"].pop(GATEWAY_KEY_VAR, None)
+
+ def use_embedding_alias(c: Corpus) -> None:
+ c.embedding_default = "aisoc-summary"
+
+ # Each case is a defect injected into a deep copy of the real corpus, and
+ # the code it must produce. deepcopy, not a shallow one: the corpora are
+ # nested (compose is service -> var -> value), and the first draft let one
+ # case mutate the baseline every later case was measured against, so every
+ # case "caught" three codes it had inherited. A self-test that passes for
+ # the wrong reason is the defect this gate exists to find, one level up.
+ cases: list[tuple[str, str, Callable[[Corpus], None]]] = [
+ (
+ "PIN -> GW: a role pinned to an alias the gateway does not define",
+ "pin-alias-undefined",
+ lambda c: c.role_pins.update({"triage": "aisoc-triage-v2"}),
+ ),
+ (
+ "GW -> PIN: an alias the gateway serves that no role requests",
+ "gateway-alias-unclaimed",
+ lambda c: c.gateway_aliases.add("aisoc-orphan"),
+ ),
+ (
+ "AGENTS <-> API: a role pinned on one side only",
+ "role-set-drift",
+ lambda c: c.api_roles.discard("triage"),
+ ),
+ (
+ "AGENTS <-> API: the mirror loses a routing function",
+ "routing-surface-drift",
+ lambda c: c.api_routing.defined.discard("resolve_api_key"),
+ ),
+ (
+ "AGENTS <-> API: the mirror stops reading a variable the other reads",
+ "routing-read-drift",
+ drop_gateway_read,
+ ),
+ (
+ "CONFIG -> GW: a router fallback naming an undefined alias",
+ "fallback-undefined",
+ lambda c: c.fallbacks.update({"aisoc-report": ["aisoc-nonexistent"]}),
+ ),
+ (
+ "ENV -> GW: .env.example ships a model the gateway does not define",
+ "env-model-undefined",
+ lambda c: c.env_example.update({"OPENAI_MODEL": "gpt-4-turbo-preview"}),
+ ),
+ (
+ "ENV -> GW: a compose service pins a role to a model the gateway lacks",
+ "env-model-undefined",
+ lambda c: c.compose["agents"].update({"AISOC_MODEL_PIN_TRIAGE": "gpt-4o-mini"}),
+ ),
+ (
+ "CMP -> CODE: compose supplies a gateway route no resolver reads",
+ "compose-var-unread",
+ lambda c: c.compose["agents"].update({"LLM_PROXY_URL": "http://litellm:4000/v1"}),
+ ),
+ (
+ "CODE -> CMP: both resolvers read a gateway variable compose never sets",
+ "code-var-unset",
+ add_unset_gateway_var,
+ ),
+ (
+ "PAIRING: a service given the gateway URL but not the gateway key",
+ "gateway-without-key",
+ drop_gateway_key,
+ ),
+ (
+ "SCOPE: the gateway grows an embedding alias it cannot serve as chat",
+ "embedding-scope",
+ lambda c: c.gateway_aliases.add("aisoc-embedding"),
+ ),
+ (
+ "SCOPE: the RAG path defaults to a gateway alias for an embedding call",
+ "embedding-scope",
+ use_embedding_alias,
+ ),
+ ]
+
+ print(f"self-test against {root}")
+ print("clean tree: 0 findings (the baseline every case below perturbs)\n")
+ results: list[tuple[str, bool]] = []
+ for description, expected, mutate in cases:
+ perturbed = copy.deepcopy(base)
+ mutate(perturbed)
+ codes = {code for code, _ in evaluate(perturbed)}
+ caught = expected in codes
+ results.append((f"{description}\n expected [{expected}] got {sorted(codes) or 'nothing'}", caught))
+
+ # Parser blind spots: text that looks like the thing, and is not it.
+ blind: list[tuple[str, bool]] = [
+ (
+ "PARSER: a commented-out model_name in the Ollama example is not a served alias",
+ "aisoc-never-served"
+ not in parse_gateway_aliases("model_list:\n - model_name: aisoc-triage\n # - model_name: aisoc-never-served\n"),
+ ),
+ (
+ "PARSER: a model_name inside a quoted value is not mistaken for a comment boundary",
+ parse_gateway_aliases('model_list:\n - model_name: "aisoc-triage" # the high-volume path\n') == {"aisoc-triage"},
+ ),
+ (
+ "PARSER: a commented-out .env assignment is documentation, not configuration",
+ parse_env_file("# AISOC_MODEL_PIN_TRIAGE=gpt-4o-mini\nOPENAI_MODEL=aisoc-summary\n")
+ == {"OPENAI_MODEL": "aisoc-summary"},
+ ),
+ (
+ "PARSER: a compose comment naming a variable two lines above is not an assignment",
+ parse_compose_env(
+ "services:\n agents:\n environment:\n"
+ " # set OPENAI_BASE_URL=http://litellm:4000/v1 to route here\n"
+ " LLM_GATEWAY_URL: http://litellm:4000/v1\n"
+ )
+ == {"agents": {"LLM_GATEWAY_URL": "http://litellm:4000/v1"}},
+ ),
+ (
+ "PARSER: a role pin named only in a docstring is not a shipped pin",
+ parse_role_pins('"""_DEFAULT_PINS maps triage to aisoc-triage."""\n_DEFAULT_PINS = {}\n') == {},
+ ),
+ (
+ "PARSER: an env var named only in a docstring is not read",
+ "LLM_GATEWAY_URL"
+ not in parse_routing_module('def gateway_url():\n """Reads LLM_GATEWAY_URL."""\n return None\n').all_reads,
+ ),
+ (
+ "PARSER: an env var read through os.environ[...] counts as read",
+ parse_routing_module('import os\ndef f():\n return os.environ["LLM_GATEWAY_URL"]\n').all_reads == {"LLM_GATEWAY_URL"},
+ ),
+ ]
+ results.extend(blind)
+ results.append(
+ (
+ f"{len(cases)} injected defects and {len(blind)} parser blind spots, each caught by its own code",
+ all(passed for _, passed in results),
+ )
+ )
+ # Delegated so the empty-tree probe is the shared one every gate answers,
+ # run here rather than restated: a gate that refuses nothing is the failure
+ # scripts/check_gate_contract.py exists to make impossible.
+ return self_test_main(Path(__file__).name, [], extra=results)
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("--repo-root", type=Path, default=None)
+ parser.add_argument("--json", action="store_true")
+ parser.add_argument("--self-test", action="store_true", help="prove the gate detects injected drift in each direction")
+ args = parser.parse_args(argv)
+
+ root = (args.repo_root or repo_root()).resolve()
+ if args.self_test:
+ return self_test(root)
+
+ try:
+ corpus = load(root)
+ except GateError as exc:
+ print(f"check_llm_model_routing: FAILED to read the tree: {exc}", file=sys.stderr)
+ return 2
+ except SyntaxError as exc:
+ print(f"check_llm_model_routing: FAILED to parse the tree: {exc}", file=sys.stderr)
+ return 2
+
+ findings = evaluate(corpus)
+
+ if args.json:
+ print(
+ json.dumps(
+ {
+ "repo_root": str(root),
+ "gateway_aliases": sorted(corpus.gateway_aliases),
+ "role_pins": corpus.role_pins,
+ "api_roles": sorted(corpus.api_roles),
+ "agents_reads": sorted(corpus.agents_routing.all_reads),
+ "api_reads": sorted(corpus.api_routing.all_reads),
+ "embedding_default": corpus.embedding_default,
+ "findings": [{"code": c, "detail": d} for c, d in findings],
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ )
+ return 1 if findings else 0
+
+ for line in _summary(corpus, root):
+ print(line)
+ print()
+ if findings:
+ print(f"FAIL: {len(findings)} model name(s) or routing variable(s) that resolve nowhere\n", file=sys.stderr)
+ for code, detail in findings:
+ print(f" [{code}] {detail}", file=sys.stderr)
+ return 1
+ print(
+ f"OK: {len(corpus.role_pins)} role pins, {len(corpus.gateway_aliases)} gateway aliases and "
+ f"every model named in {ENV_EXAMPLE_REL} / {COMPOSE_REL} resolve at the gateway, "
+ "and the variable compose supplies is the variable both resolvers read"
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/services/agents/app/agents/auto_triage_agent.py b/services/agents/app/agents/auto_triage_agent.py
index 068a18c4d..4dd70e29c 100644
--- a/services/agents/app/agents/auto_triage_agent.py
+++ b/services/agents/app/agents/auto_triage_agent.py
@@ -231,10 +231,15 @@ async def run_auto_triage(state: InvestigationState) -> InvestigationState:
alert_context = _build_alert_context(state)
- llm = make_chat_model("triage", temperature=0.0, max_tokens=512)
-
t0 = time.monotonic()
try:
+ # Inside the try: "the model could not be built" and "the call failed"
+ # are the same condition to every caller, and only one of them used to
+ # become an AutoTriageError. An unroutable gateway alias raises here,
+ # and auto_triage_node catches AutoTriageError specifically — so
+ # constructing outside would have failed the whole graph run over a
+ # configuration problem the deterministic path handles fine.
+ llm = make_chat_model("triage", temperature=0.0, max_tokens=512)
response = await safe_ainvoke(
llm,
[
diff --git a/services/agents/app/api/contextual.py b/services/agents/app/api/contextual.py
index aeb8ffb3e..9773f0531 100644
--- a/services/agents/app/api/contextual.py
+++ b/services/agents/app/api/contextual.py
@@ -33,7 +33,6 @@ unset, so the demo path never breaks.
from __future__ import annotations
import json
-import os
import time
import uuid
from collections.abc import AsyncIterator
@@ -46,7 +45,7 @@ from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from app.llm import safe_ainvoke, safe_astream
-from app.llm.factory import resolve_model_alias
+from app.llm.factory import make_chat_model, resolve_api_key, resolve_model_alias
from app.prompt_serialization import summarize_structure_for_llm
from app.security.tenant_scope import require_console_or_service_auth
@@ -369,17 +368,18 @@ async def _call_llm(system: str, user: str, model: str) -> tuple[str, int]:
Falls back to a deterministic stub response when ``OPENAI_API_KEY`` is
missing so the demo never hard-fails.
"""
- if not os.getenv("OPENAI_API_KEY"):
+ if not resolve_api_key(model):
return _fallback_response(system, user), 0
try:
from langchain_core.messages import HumanMessage, SystemMessage
- from langchain_openai import ChatOpenAI
except ImportError as exc:
logger.warning("contextual.llm.import_failed", error=str(exc))
return _fallback_response(system, user), 0
- llm = ChatOpenAI(model=model, temperature=0.2)
+ # Was a bare ``ChatOpenAI(model=model)`` — no base URL, so the copilot sent
+ # `aisoc-copilot` to the provider default and took the 404 as "no LLM".
+ llm = make_chat_model("copilot", temperature=0.2)
response = await safe_ainvoke(llm, [SystemMessage(content=system), HumanMessage(content=user)])
text = response.content if isinstance(response.content, str) else str(response.content)
tokens = 0
@@ -390,7 +390,7 @@ async def _call_llm(system: str, user: str, model: str) -> tuple[str, int]:
async def _stream_llm(system: str, user: str, model: str) -> AsyncIterator[str]:
"""Yield response delta chunks. Used by the NDJSON streaming endpoint."""
- if not os.getenv("OPENAI_API_KEY"):
+ if not resolve_api_key(model):
# Fake-stream the fallback in 8-character chunks for a nice UX in the
# demo path.
text = _fallback_response(system, user)
@@ -400,14 +400,13 @@ async def _stream_llm(system: str, user: str, model: str) -> AsyncIterator[str]:
try:
from langchain_core.messages import HumanMessage, SystemMessage
- from langchain_openai import ChatOpenAI
except ImportError:
text = _fallback_response(system, user)
for i in range(0, len(text), 8):
yield text[i : i + 8]
return
- llm = ChatOpenAI(model=model, temperature=0.2, streaming=True)
+ llm = make_chat_model("copilot", temperature=0.2, streaming=True)
async for chunk in safe_astream(llm, [SystemMessage(content=system), HumanMessage(content=user)]):
if hasattr(chunk, "content") and chunk.content:
yield chunk.content if isinstance(chunk.content, str) else str(chunk.content)
@@ -423,9 +422,9 @@ def _fallback_response(system: str, user: str) -> str:
"so the contextual Copilot cannot reach a language model right now. "
"You're seeing a deterministic placeholder response.\n\n"
"**To enable real contextual answers:**\n\n"
- "1. Set `OPENAI_API_KEY` in your `.env` file\n"
- "2. Point AiSOC at the LiteLLM gateway (`OPENAI_BASE_URL=http://litellm:4000/v1`), "
- "which maps each task alias to a real model\n"
+ "1. Set `OPENAI_API_KEY` in your `.env` file — the gateway reads it\n"
+ "2. Bring up the `full` profile so the LiteLLM gateway runs; "
+ "`LLM_GATEWAY_URL` already points every service at it\n"
"3. Restart the `aisoc-agents` service\n\n"
"See the LLM-gateway docs for self-hosted model alternatives (Ollama, vLLM, Together)."
)
@@ -453,7 +452,10 @@ async def run_action(req: ContextualActionRequest) -> ContextualActionResponse:
system, user = _build_messages(req)
model = resolve_model_alias("copilot")
- fallback = not bool(os.getenv("OPENAI_API_KEY"))
+ # Same question _call_llm asks, so the flag the UI reads and the branch
+ # actually taken cannot disagree — the gateway authenticates with
+ # LITELLM_MASTER_KEY, which OPENAI_API_KEY alone does not see.
+ fallback = not resolve_api_key(model)
try:
content, tokens = await _call_llm(system, user, model)
except Exception as exc: # noqa: BLE001
@@ -512,7 +514,10 @@ async def run_action_stream(req: ContextualActionRequest) -> StreamingResponse:
model = resolve_model_alias("copilot")
title = _TITLES.get((req.page, req.action), f"{req.page} · {req.action}")
suggestions = _FOLLOW_UPS.get((req.page, req.action), [])
- fallback = not bool(os.getenv("OPENAI_API_KEY"))
+ # Same question _call_llm asks, so the flag the UI reads and the branch
+ # actually taken cannot disagree — the gateway authenticates with
+ # LITELLM_MASTER_KEY, which OPENAI_API_KEY alone does not see.
+ fallback = not resolve_api_key(model)
async def gen() -> AsyncIterator[bytes]:
# Header frame so the UI can render the title before tokens arrive.
diff --git a/services/agents/app/api/copilot.py b/services/agents/app/api/copilot.py
index b335bc955..ec325b501 100644
--- a/services/agents/app/api/copilot.py
+++ b/services/agents/app/api/copilot.py
@@ -16,7 +16,6 @@ from __future__ import annotations
import itertools
import json
-import os
import uuid
from collections.abc import AsyncIterator
from datetime import UTC, datetime
@@ -136,13 +135,18 @@ async def _get_openai_reply(
alerts" as real analysis of their environment. The frontend had an honest
fallback of its own that never fired, because the backend reported success.
"""
- api_key = os.getenv("OPENAI_API_KEY", "")
+ from app.llm.factory import resolve_api_key, resolve_model_alias
+
+ model = resolve_model_alias("copilot")
+ # Resolved with the route, not from OPENAI_API_KEY directly: when the call
+ # goes to the bundled gateway the bearer has to be the gateway's master key.
+ api_key = resolve_api_key(model) or ""
if not api_key:
return _synthetic_reply(user_message), "template"
try:
from app.llm.contract import safe_chat_completions_request
- from app.llm.factory import chat_completions_url, resolve_model_alias
+ from app.llm.factory import chat_completions_url
messages: list[dict[str, str]] = [
{
@@ -161,9 +165,9 @@ async def _get_openai_reply(
body = await safe_chat_completions_request(
api_key=api_key,
- model=resolve_model_alias("copilot"),
+ model=model,
messages=messages,
- url=chat_completions_url(),
+ url=chat_completions_url(model),
max_tokens=512,
)
return body["choices"][0]["message"]["content"], "llm"
diff --git a/services/agents/app/llm/factory.py b/services/agents/app/llm/factory.py
index 46493baaf..f732a96c8 100644
--- a/services/agents/app/llm/factory.py
+++ b/services/agents/app/llm/factory.py
@@ -14,10 +14,12 @@ instead of through the gateway can pin a concrete model. When no live LLM is
reachable, callers fall back to their deterministic path exactly as before — the
factory never forces a live call.
-Routing to the gateway is deliberate: set ``OPENAI_BASE_URL`` (or ``LLM_BASE_URL``)
-to the gateway and send ``OPENAI_API_KEY=$LITELLM_MASTER_KEY``. Left unset, the
-client talks to its provider default — where an alias only resolves if the
-operator has pinned a concrete model via ``AISOC_MODEL_PIN_<ROLE>``.
+Which URL a model goes to, and which bearer goes with it, is
+:mod:`app.llm.routing` — shared with the BYOK resolver and mirrored in the API
+service. The short version: the compose-provided ``LLM_GATEWAY_URL`` is adopted
+for an ``aisoc-<role>`` alias (it can resolve nowhere else) and left alone for a
+concrete pinned model (which the gateway would not define), and an explicit
+``OPENAI_BASE_URL`` always wins.
"""
from __future__ import annotations
@@ -33,6 +35,42 @@ from langchain_openai import ChatOpenAI
from app.llm.contract import DEFAULT_OPENAI_CHAT_COMPLETIONS_URL
from app.llm.model_pins import get_pin
+# The routing rule (which URL, which bearer, and when an alias has nowhere to
+# go) lives in one stdlib-only module because the BYOK resolver needs it too
+# and cannot import this one. Re-exported so existing call sites and tests keep
+# importing these names from the factory.
+from app.llm.routing import (
+ BYOK_MODEL_ENV_VARS,
+ GATEWAY_ALIAS_PREFIX,
+ UnroutableModelError,
+ adopted_gateway,
+ assert_routable,
+ at_bundled_gateway,
+ gateway_url,
+ is_gateway_alias,
+ resolve_api_key,
+ resolve_base_url,
+)
+from app.llm.routing import explicit_base_url as _explicit_base_url
+
+__all__ = [
+ "BYOK_MODEL_ENV_VARS",
+ "GATEWAY_ALIAS_PREFIX",
+ "UnroutableModelError",
+ "adopted_gateway",
+ "assert_routable",
+ "at_bundled_gateway",
+ "chat_completions_url",
+ "gateway_url",
+ "is_gateway_alias",
+ "llm_override",
+ "make_chat_model",
+ "preflight_llm",
+ "resolve_api_key",
+ "resolve_base_url",
+ "resolve_model_alias",
+]
+
# Wave 1 — per-tenant BYOK override. The auto-triage / investigation paths
# resolve a tenant's own (base_url, model, api_key) via
# app.security.llm_resolver and bind it here so make_chat_model actually routes
@@ -62,22 +100,16 @@ def resolve_model_alias(role: str) -> str:
return get_pin(role).primary_model
-def resolve_base_url() -> str | None:
- """OpenAI-compatible base URL for live calls, or ``None`` for the client default.
+def chat_completions_url(model: str | None = None) -> str:
+ """Full chat-completions URL for the raw-HTTP path (:func:`safe_chat_completions_request`).
- Honours an explicit ``OPENAI_BASE_URL`` / ``LLM_BASE_URL`` — what an operator
- sets to point AiSOC at the LiteLLM gateway. ``None`` means "use the client's
- provider default" (the direct-to-provider path). We intentionally do **not**
- auto-adopt the compose-provided ``LLM_GATEWAY_URL`` here: routing through the
- gateway is an explicit choice so the bearer token (the gateway master key vs.
- a provider key) is never ambiguous.
+ ``model`` is what the caller is about to send. Passing it lets the same
+ routing rule as :func:`resolve_base_url` apply, and turns an alias with no
+ gateway into :class:`UnroutableModelError` here rather than a 404 later.
"""
- return os.getenv("OPENAI_BASE_URL", "").strip() or os.getenv("LLM_BASE_URL", "").strip() or None
-
-
-def chat_completions_url() -> str:
- """Full chat-completions URL for the raw-HTTP path (:func:`safe_chat_completions_request`)."""
- base = resolve_base_url()
+ base = resolve_base_url(model)
+ if model is not None:
+ assert_routable(model, base)
if base:
return base.rstrip("/") + "/chat/completions"
return DEFAULT_OPENAI_CHAT_COMPLETIONS_URL
@@ -97,39 +129,61 @@ def make_chat_model(
contract. Extra ``kwargs`` pass straight through to ``ChatOpenAI``.
"""
override = _llm_override.get()
- params: dict[str, Any] = {
- "model": (override or {}).get("model") or resolve_model_alias(role),
- "temperature": temperature,
- }
+ model = (override or {}).get("model") or resolve_model_alias(role)
+ params: dict[str, Any] = {"model": model, "temperature": temperature}
if max_tokens is not None:
params["max_tokens"] = max_tokens
- base_url = (override or {}).get("base_url") or resolve_base_url()
+ base_url = (override or {}).get("base_url") or resolve_base_url(model)
+ assert_routable(model, base_url)
if base_url:
params["base_url"] = base_url
- if override and override.get("api_key"):
- params["api_key"] = override["api_key"]
+ api_key = (override or {}).get("api_key") or resolve_api_key(model)
+ if api_key:
+ params["api_key"] = api_key
params.update(kwargs)
return ChatOpenAI(**params)
def preflight_llm(roles: tuple[str, ...] = ("triage", "recon", "investigation", "report", "summary", "copilot", "nl")) -> list[str]:
- """Return startup warnings about unroutable LLM aliases.
+ """Return startup warnings about LLM config that cannot reach a model.
- Catches the LiteLLM 400 gotcha: when no gateway base URL is configured, an
- ``aisoc-<role>`` alias is sent straight to the provider default (e.g.
- api.openai.com), which 400s with "model does not exist" — and the agents
- then silently degrade to heuristics with no operator signal. This surfaces
- the misconfiguration explicitly at boot. Empty list => config looks routable.
+ Checked in both directions, because each one shipped broken:
+
+ * **alias with no gateway** — an ``aisoc-<role>`` alias sent to a provider
+ default 404s, and the caller's ``except`` renders that as a deterministic
+ fallback. This condition was already reported here, correctly, while
+ nothing read the variable that would have fixed it.
+ * **concrete model at the gateway** — the reverse. A BYOK model name
+ (``OPENAI_MODEL``) or a role pin naming something the gateway does not
+ define 400s with "Invalid model name" once traffic is routed through it.
+
+ Empty list => every role has somewhere to send its model.
"""
warnings: list[str] = []
- if resolve_base_url():
- return warnings # a gateway/base URL is set — aliases resolve there.
- unrouted = sorted({resolve_model_alias(r) for r in roles if resolve_model_alias(r).startswith("aisoc-")})
+ aliases = {r: resolve_model_alias(r) for r in roles}
+
+ unrouted = sorted({m for m in aliases.values() if is_gateway_alias(m) and not resolve_base_url(m)})
if unrouted:
warnings.append(
- "No OPENAI_BASE_URL/LLM_BASE_URL is set, so LiteLLM aliases "
- f"({', '.join(unrouted)}) will be sent to the provider default and 400. "
- "Point OPENAI_BASE_URL at the LiteLLM gateway, or pin concrete models "
- "via AISOC_MODEL_PIN_<ROLE>. Agents will otherwise run deterministic-only."
+ f"No LLM gateway is configured, so LiteLLM aliases ({', '.join(unrouted)}) have "
+ "nowhere to resolve and every live call will fail. Set LLM_GATEWAY_URL "
+ "(docker-compose.yml already does) or OPENAI_BASE_URL to the gateway, or pin "
+ "concrete models via AISOC_MODEL_PIN_<ROLE>. Agents run deterministic-only until then."
+ )
+
+ concrete_at_gateway = sorted({m for m in aliases.values() if not is_gateway_alias(m) and at_bundled_gateway(resolve_base_url(m))})
+ if concrete_at_gateway:
+ warnings.append(
+ f"AISOC_MODEL_PIN_<ROLE> names concrete models ({', '.join(concrete_at_gateway)}) while traffic "
+ "is routed at the bundled LiteLLM gateway, which 400s any model it does not define. Either add "
+ "them to infra/litellm/config.yaml or drop the pins so the aisoc-<role> aliases apply."
+ )
+
+ byok = sorted({v for name in BYOK_MODEL_ENV_VARS if (v := os.getenv(name, "").strip()) and not is_gateway_alias(v)})
+ if byok and at_bundled_gateway(_explicit_base_url() or gateway_url()):
+ warnings.append(
+ f"{'/'.join(BYOK_MODEL_ENV_VARS)} names a concrete model ({', '.join(byok)}) the bundled gateway "
+ "does not define, so the BYOK / explain path will 400. These never apply to a task role — "
+ "set one of them to an aisoc-<role> alias, or add the model to infra/litellm/config.yaml."
)
return warnings
diff --git a/services/agents/app/llm/routing.py b/services/agents/app/llm/routing.py
new file mode 100644
index 000000000..49c3df77c
--- /dev/null
+++ b/services/agents/app/llm/routing.py
@@ -0,0 +1,157 @@
+"""Where a model name goes, and which bearer goes with it.
+
+One rule, in one place, because it had been written three times and disagreed
+each time. ``app.llm.factory`` resolves it for the LangChain path,
+``app.security.llm_resolver`` for the BYOK / "explain this alert" path, and
+``app.services.model_aliases`` is the byte-equivalent mirror in the API service
+(a separate package that cannot import this one).
+
+The defect this replaces
+------------------------
+``docker-compose.yml`` set ``LLM_GATEWAY_URL`` on the api and agents services.
+Nothing read it. The resolvers honoured only ``OPENAI_BASE_URL`` /
+``LLM_BASE_URL``, which compose never set, and declined to adopt the gateway
+variable on the grounds that routing through the gateway should be an explicit
+choice so the bearer token is never ambiguous. The reasoning was sound and the
+result was that the product's headline capability could not reach a model in the
+default deployment: an ``aisoc-triage`` alias went to ``api.openai.com``, which
+404s, and the caller's ``except`` rendered that as "no LLM available".
+
+Two rules make the adoption unambiguous rather than merely convenient:
+
+**The model decides the route.** ``LLM_GATEWAY_URL`` is adopted only for a
+gateway alias. An alias is unroutable anywhere else, so sending it to the
+gateway cannot be wrong. A concrete model — ``AISOC_MODEL_PIN_TRIAGE=gpt-4o-mini``,
+the documented direct-to-provider escape hatch — is left pointing at its
+provider, because hijacking it to a gateway that does not define it would break
+a deployment that works today.
+
+**The route decides the bearer.** When AiSOC picks the gateway itself, the
+bearer is the gateway's ``LITELLM_MASTER_KEY``; a provider key sent there is
+rejected as an invalid proxy token. When the operator set ``OPENAI_BASE_URL``
+by hand they own the pairing. That is the ambiguity the old refusal was about,
+resolved instead of avoided.
+
+Stdlib only: the BYOK resolver is imported by explain unit tests that avoid the
+LangChain dependency graph, so this cannot reach for anything heavier.
+"""
+
+from __future__ import annotations
+
+import os
+
+#: The routing rule, named. ``scripts/check_llm_model_routing.py`` requires the
+#: API-service mirror to define every one of these, and requires any function
+#: in the mirror that reads a routing variable to appear here — so the mirror
+#: can neither lose a rule nor grow one this module does not have.
+__all__ = [
+ "BYOK_MODEL_ENV_VARS",
+ "GATEWAY_ALIAS_PREFIX",
+ "UnroutableModelError",
+ "adopted_gateway",
+ "assert_routable",
+ "at_bundled_gateway",
+ "explicit_base_url",
+ "gateway_url",
+ "is_gateway_alias",
+ "resolve_api_key",
+ "resolve_base_url",
+]
+
+#: Prefix of every logical task alias the bundled gateway defines. The alias set
+#: itself lives in ``infra/litellm/config.yaml``;
+#: ``scripts/check_llm_model_routing.py`` reconciles that file against the role
+#: pins in both directions. Only the *shape* is known here, because the service
+#: images do not ship the gateway config.
+GATEWAY_ALIAS_PREFIX = "aisoc-"
+
+#: Env vars holding a model name for the BYOK / "explain this alert" path. They
+#: are not role pins. A task role that adopts one has taken a global default for
+#: a per-tenant override, which is how ``OPENAI_MODEL`` came to replace
+#: ``aisoc-triage`` on the highest-volume path in the product.
+BYOK_MODEL_ENV_VARS = ("OPENAI_MODEL", "LLM_MODEL", "AISOC_LLM_MODEL")
+
+
+class UnroutableModelError(RuntimeError):
+ """A gateway alias was requested with nowhere to send it.
+
+ An ``aisoc-<role>`` alias only means something to the LiteLLM gateway. Sent
+ to a provider default it returns 404, and the caller's ``except`` turns that
+ into a deterministic fallback — a configuration error wearing the costume of
+ a degraded-but-working system. Raising names the actual fault.
+ """
+
+
+def is_gateway_alias(model: str | None) -> bool:
+ """Whether ``model`` is a logical alias that only the gateway can resolve."""
+ return bool(model) and str(model).strip().startswith(GATEWAY_ALIAS_PREFIX)
+
+
+def explicit_base_url() -> str | None:
+ """The base URL an operator set by hand, if any."""
+ return os.environ.get("OPENAI_BASE_URL", "").strip() or os.environ.get("LLM_BASE_URL", "").strip() or None
+
+
+def gateway_url() -> str | None:
+ """The bundled LiteLLM gateway's in-network URL, as compose supplies it."""
+ return os.environ.get("LLM_GATEWAY_URL", "").strip() or None
+
+
+def resolve_base_url(model: str | None = None) -> str | None:
+ """OpenAI-compatible base URL for ``model``, or ``None`` for the client default.
+
+ Precedence, highest first: ``OPENAI_BASE_URL`` / ``LLM_BASE_URL``, then
+ ``LLM_GATEWAY_URL`` when ``model`` is a gateway alias, then nothing.
+
+ ``model=None`` means the caller has not said which model it will send.
+ Every task role defaults to an alias, so the gateway is the right answer.
+ """
+ explicit = explicit_base_url()
+ if explicit:
+ return explicit
+ if model is None or is_gateway_alias(model):
+ return gateway_url()
+ return None
+
+
+def adopted_gateway(model: str | None = None) -> bool:
+ """Whether the base URL for ``model`` came from ``LLM_GATEWAY_URL``, not an operator."""
+ return explicit_base_url() is None and resolve_base_url(model) is not None
+
+
+def at_bundled_gateway(base_url: str | None) -> bool:
+ """Whether ``base_url`` is the gateway this repo ships a model list for.
+
+ Narrower than "some base URL is set" on purpose: an operator pointing at
+ their own vLLM with concrete model names is correct, and warning about it
+ would be noise. Only ``infra/litellm/config.yaml``'s alias set is knowable
+ from here, so only that gateway's contents can be reasoned about.
+ """
+ gateway = gateway_url()
+ if not base_url or not gateway:
+ return False
+ return str(base_url).rstrip("/") == gateway.rstrip("/")
+
+
+def resolve_api_key(model: str | None = None) -> str | None:
+ """Bearer token that pairs with :func:`resolve_base_url`'s answer for ``model``."""
+ if adopted_gateway(model):
+ return os.environ.get("LITELLM_MASTER_KEY", "").strip() or os.environ.get("OPENAI_API_KEY", "").strip() or None
+ return os.environ.get("OPENAI_API_KEY", "").strip() or os.environ.get("LLM_API_KEY", "").strip() or None
+
+
+def assert_routable(model: str, base_url: str | None) -> None:
+ """Raise :class:`UnroutableModelError` if ``model`` can never resolve at ``base_url``.
+
+ One shape, checked at every entry point: a gateway alias with no gateway to
+ send it to. Everything else is the provider's call to make — a wrong
+ concrete model name comes back as a 404 from a host that can answer the
+ question, which is a diagnosable result rather than a silent one.
+ """
+ if is_gateway_alias(model) and not base_url:
+ raise UnroutableModelError(
+ f"model '{model}' is a LiteLLM gateway alias and no gateway is configured. "
+ "Set LLM_GATEWAY_URL (docker-compose.yml already does) or OPENAI_BASE_URL to the "
+ "gateway, or pin a concrete provider model via AISOC_MODEL_PIN_<ROLE>. "
+ "Sending an alias to a provider default returns 404 and degrades silently."
+ )
diff --git a/services/agents/app/nl_query/translator.py b/services/agents/app/nl_query/translator.py
index 5de992be3..98e4f6eb5 100644
--- a/services/agents/app/nl_query/translator.py
+++ b/services/agents/app/nl_query/translator.py
@@ -930,11 +930,12 @@ async def enhance_with_llm(
{"role": "user", "content": prompt},
]
+ resolved_model = model or resolve_model_alias("nl")
payload = await safe_chat_completions_request(
api_key=api_key,
- model=model or resolve_model_alias("nl"),
+ model=resolved_model,
messages=messages,
- url=chat_completions_url(),
+ url=chat_completions_url(resolved_model),
timeout=timeout,
response_format={"type": "json_object"},
temperature=0,
diff --git a/services/agents/app/security/llm_resolver.py b/services/agents/app/security/llm_resolver.py
index 9e6c69489..b8141ccb4 100644
--- a/services/agents/app/security/llm_resolver.py
+++ b/services/agents/app/security/llm_resolver.py
@@ -42,8 +42,9 @@ For each of the three fields we resolve in this order:
1. Tenant row (when present, ``enabled=true``, and successfully
decrypted in the ``api_key_vault`` case).
2. Process env vars (``OPENAI_*`` / ``LLM_*``).
-3. Platform default (``https://api.openai.com`` / ``gpt-4o-mini``)
- so the explain path always has a deterministic config to log
+3. Platform default — the bundled gateway and the ``summary`` alias when a
+ gateway is configured, ``https://api.openai.com`` / ``gpt-4o-mini``
+ otherwise — so the explain path always has a deterministic config to log
against, even when it ultimately decides ``allowed=False``.
Whichever field came from the tenant row is recorded so the resolver
@@ -82,6 +83,7 @@ from urllib.parse import urlparse
import asyncpg
import structlog
+from app.llm.routing import gateway_url, is_gateway_alias, resolve_api_key
from app.security.credential_vault import CredentialVaultError, get_vault
# NOTE: ``app.investigator.ledger`` is imported lazily inside
@@ -106,6 +108,10 @@ logger = structlog.get_logger()
_DEFAULT_OPENAI_BASE = "https://api.openai.com"
_DEFAULT_OPENAI_MODEL = "gpt-4o-mini"
+#: The role the explain path's work belongs to. Its alias is the default model
+#: when a gateway is configured, because ``gpt-4o-mini`` is not a model the
+#: bundled gateway defines and sending it there returns "Invalid model name".
+_EXPLAIN_ROLE_ALIAS = "aisoc-summary"
@dataclass(frozen=True)
@@ -138,6 +144,16 @@ class LlmConfig:
api_key: str | None
source: str
reason: str
+ # Per-field provenance. ``source`` collapses the whole config into one
+ # word, which is enough to log and not enough to act on: the auto-triage
+ # worker binds this config as a per-tenant BYOK *override*, and without
+ # knowing which fields the tenant actually set it overrode the task role's
+ # model with whatever ``OPENAI_MODEL`` happened to be — shipped in
+ # .env.example as `gpt-4-turbo-preview`, which no gateway here defines. An
+ # env baseline is a default, never an override.
+ model_from_tenant: bool = False
+ base_url_from_tenant: bool = False
+ api_key_from_tenant: bool = False
# ---------------------------------------------------------------------------
@@ -157,7 +173,15 @@ def _env_baseline() -> tuple[str, str, str | None]:
"""
base_url = os.getenv("OPENAI_BASE_URL", "").strip() or os.getenv("LLM_BASE_URL", "").strip()
model = os.getenv("OPENAI_MODEL", "").strip() or os.getenv("LLM_MODEL", "").strip() or os.getenv("AISOC_LLM_MODEL", "").strip()
- api_key = os.getenv("OPENAI_API_KEY", "").strip() or os.getenv("LLM_API_KEY", "").strip()
+ if not base_url and gateway_url() and is_gateway_alias(model or _EXPLAIN_ROLE_ALIAS):
+ # The compose-provided gateway, under the same rule every other caller
+ # uses (``app.llm.routing``): adopted for an alias, never for a concrete
+ # model. Without this the explain path was the one consumer that could
+ # not reach the gateway at all — it defaulted straight to
+ # ``https://api.openai.com`` and took an alias there.
+ base_url = gateway_url() or ""
+ model = model or _EXPLAIN_ROLE_ALIAS
+ api_key = resolve_api_key(model) or ""
return base_url, model, (api_key or None)
@@ -388,6 +412,9 @@ async def resolve_llm_config(tenant_ref: str | None) -> LlmConfig:
api_key=None,
source=source if source != "none" else "none",
reason="no API key configured (neither tenant BYOK nor env)",
+ model_from_tenant=tenant_contributed_model,
+ base_url_from_tenant=tenant_contributed_base_url,
+ api_key_from_tenant=tenant_contributed_key,
)
blocked, reason = _airgap_blocks(base_url)
@@ -399,6 +426,9 @@ async def resolve_llm_config(tenant_ref: str | None) -> LlmConfig:
api_key=api_key, # safe: caller does not log this on failure
source=source,
reason=reason,
+ model_from_tenant=tenant_contributed_model,
+ base_url_from_tenant=tenant_contributed_base_url,
+ api_key_from_tenant=tenant_contributed_key,
)
return LlmConfig(
@@ -408,6 +438,9 @@ async def resolve_llm_config(tenant_ref: str | None) -> LlmConfig:
api_key=api_key,
source=source,
reason="",
+ model_from_tenant=tenant_contributed_model,
+ base_url_from_tenant=tenant_contributed_base_url,
+ api_key_from_tenant=tenant_contributed_key,
)
diff --git a/services/agents/app/tools/mitre_full.py b/services/agents/app/tools/mitre_full.py
index 40567a2e4..0db583d07 100644
--- a/services/agents/app/tools/mitre_full.py
+++ b/services/agents/app/tools/mitre_full.py
@@ -34,6 +34,22 @@ ATTCK_DATA_PATH = os.getenv("ATTCK_DATA_PATH", "/data/enterprise-attack.json")
_CACHE_TTL_HOURS = 24
_EMBED_BATCH_SIZE = 50
+#: Embeddings deliberately do NOT follow chat traffic to the bundled LiteLLM
+#: gateway. ``infra/litellm/config.yaml`` declares seven *chat* aliases and no
+#: embedding model, so routing here would 400 on every batch — and a gateway
+#: master key sent to api.openai.com 401s just as surely. This path therefore
+#: keeps its own pair of variables, defaulting to the provider.
+#: ``scripts/check_llm_model_routing.py`` records embeddings as out of the
+#: gateway's scope, so the exclusion is checked rather than assumed.
+_DEFAULT_EMBEDDING_MODEL = "text-embedding-3-large"
+
+
+def _embedding_client(openai_module: Any, api_key: str) -> tuple[Any, str]:
+ """Return ``(client, model)`` for the embedding endpoint."""
+ base_url = os.getenv("AISOC_EMBEDDING_BASE_URL", "").strip() or None
+ model = os.getenv("AISOC_EMBEDDING_MODEL", "").strip() or _DEFAULT_EMBEDDING_MODEL
+ return openai_module.AsyncOpenAI(api_key=api_key, base_url=base_url), model
+
# ─── Data Model ───────────────────────────────────────────────────────────────
@@ -339,8 +355,13 @@ async def embed_techniques_into_qdrant(
logger.warning("Qdrant/OpenAI client not available for ATT&CK embedding", error=str(exc))
return
- oai = openai.AsyncOpenAI(api_key=openai_api_key)
+ oai, embed_model = _embedding_client(openai, openai_api_key)
qdrant = AsyncQdrantClient(url=qdrant_url)
+ logger.info(
+ "ATT&CK embedding target",
+ base_url=os.getenv("AISOC_EMBEDDING_BASE_URL", "").strip() or "provider default",
+ model=embed_model,
+ )
# Ensure collection exists
try:
@@ -362,7 +383,7 @@ async def embed_techniques_into_qdrant(
texts = [f"{t.id} {t.name}: {t.description or ''} Tactics: {', '.join(t.tactic_names or [])}" for t in batch]
try:
resp = await oai.embeddings.create(
- model="text-embedding-3-large",
+ model=embed_model,
input=texts,
dimensions=3072,
)
@@ -412,12 +433,12 @@ async def semantic_technique_search(
except ImportError:
return []
- oai = openai.AsyncOpenAI(api_key=openai_api_key)
+ oai, embed_model = _embedding_client(openai, openai_api_key)
qdrant = AsyncQdrantClient(url=qdrant_url)
try:
emb_resp = await oai.embeddings.create(
- model="text-embedding-3-large",
+ model=embed_model,
input=[query],
dimensions=3072,
)
diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.py
index fd1b7ebcc..6ce70d98c 100644
--- a/services/agents/app/workers/fused_alert_consumer.py
+++ b/services/agents/app/workers/fused_alert_consumer.py
@@ -427,7 +427,23 @@ class FusedAlertTriageWorker:
if use_llm and cfg is not None:
# Route the LLM call through the tenant's BYOK key/model so
# auto-triage actually honours per-tenant credentials.
- with llm_override(api_key=cfg.api_key, base_url=cfg.base_url, model=cfg.model):
+ #
+ # Only the fields the *tenant* set are overrides. The
+ # resolver also carries an env baseline, and passing that
+ # through here replaced the `triage` role's gateway alias
+ # with whatever `OPENAI_MODEL` held — `gpt-4-turbo-preview`
+ # in the shipped .env.example, which the gateway 400s. An
+ # env default is not a per-tenant override.
+ # The key needs the same treatment: the env baseline's key
+ # is a *provider* key, and forcing it here would send it to
+ # the gateway as a bearer token, which 401s. Left unset,
+ # make_chat_model resolves the key that pairs with the route
+ # it actually chose.
+ with llm_override(
+ api_key=cfg.api_key if getattr(cfg, "api_key_from_tenant", False) else None,
+ base_url=cfg.base_url if getattr(cfg, "base_url_from_tenant", False) else None,
+ model=cfg.model if getattr(cfg, "model_from_tenant", False) else None,
+ ):
state, tier = await self._llm_triage(state)
else:
state = await run_triage(state)
@@ -701,10 +717,14 @@ class FusedAlertTriageWorker:
"""Resolve the tenant's LLM config, or None to force deterministic triage."""
try:
cfg = await resolve_llm_config(str(tenant_id))
- return cfg if (cfg.allowed and cfg.api_key) else None
except Exception as exc: # noqa: BLE001 — resolver failure => deterministic
logger.debug("auto_triage_worker.llm_resolve_failed", error=str(exc))
return None
+ # The resolver's env baseline now resolves the bearer that pairs with
+ # the route it chose, so a deployment doing exactly what the compose
+ # file sets up — gateway URL and master key, provider key held by the
+ # gateway — reports a usable key here instead of "none configured".
+ return cfg if (cfg.allowed and cfg.api_key) else None
async def _llm_triage(self, state: InvestigationState) -> tuple[InvestigationState, str]:
try:
diff --git a/services/agents/tests/test_llm_factory.py b/services/agents/tests/test_llm_factory.py
index bed8d2783..3c010a5c8 100644
--- a/services/agents/tests/test_llm_factory.py
+++ b/services/agents/tests/test_llm_factory.py
@@ -4,11 +4,16 @@ from __future__ import annotations
import pytest
from app.llm.factory import (
+ UnroutableModelError,
chat_completions_url,
+ llm_override,
make_chat_model,
+ preflight_llm,
+ resolve_api_key,
resolve_base_url,
resolve_model_alias,
)
+from app.llm.model_pins import all_roles
# app.llm.contract.DEFAULT_OPENAI_CHAT_COMPLETIONS_URL
_DEFAULT_URL = "https://api.openai.com/v1/chat/completions"
@@ -21,8 +26,13 @@ def _clean_llm_env(monkeypatch):
"OPENAI_BASE_URL",
"LLM_BASE_URL",
"LLM_GATEWAY_URL",
- "AISOC_MODEL_PIN_TRIAGE",
- "AISOC_MODEL_PIN_NL",
+ "LITELLM_MASTER_KEY",
+ "OPENAI_API_KEY",
+ "LLM_API_KEY",
+ "OPENAI_MODEL",
+ "LLM_MODEL",
+ "AISOC_LLM_MODEL",
+ *(f"AISOC_MODEL_PIN_{role.upper()}" for role in all_roles()),
):
monkeypatch.delenv(var, raising=False)
@@ -57,11 +67,52 @@ def test_base_url_prefers_openai_then_llm(monkeypatch):
assert resolve_base_url() == "http://gw:4000/v1"
-def test_base_url_ignores_gateway_url_env(monkeypatch):
- # Routing through the gateway must be explicit (OPENAI_BASE_URL), never
- # silently adopted from the compose-provided LLM_GATEWAY_URL.
+def test_base_url_adopts_gateway_url_for_an_alias(monkeypatch):
+ # The compose-provided variable, which nothing read: an aisoc-<role> alias
+ # resolves nowhere else, so the gateway is the only correct destination.
+ # Sending it to the provider default is the 404 this whole module exists to
+ # stop, and the fallback made it look like "no LLM available".
monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
- assert resolve_base_url() is None
+ assert resolve_base_url("aisoc-triage") == "http://litellm:4000/v1"
+ # No model named => every task role defaults to an alias, so same answer.
+ assert resolve_base_url() == "http://litellm:4000/v1"
+
+
+def test_base_url_leaves_a_concrete_pinned_model_at_its_provider(monkeypatch):
+ # The documented direct-to-provider escape hatch. Rerouting it to a gateway
+ # that does not define `gpt-4o-mini` would break a working deployment.
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ assert resolve_base_url("gpt-4o-mini") is None
+
+
+def test_explicit_base_url_outranks_the_gateway(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("OPENAI_BASE_URL", "http://my-vllm:8000/v1")
+ assert resolve_base_url("aisoc-triage") == "http://my-vllm:8000/v1"
+ assert resolve_base_url("gpt-4o-mini") == "http://my-vllm:8000/v1"
+
+
+# ── resolve_api_key ──────────────────────────────────────────────────────────
+
+
+def test_api_key_is_the_master_key_when_we_pick_the_gateway(monkeypatch):
+ # The route decides the bearer. A provider key sent to LiteLLM is rejected
+ # as an invalid proxy token, so resolving the two independently is what
+ # made "adopt the gateway URL" look ambiguous in the first place.
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("LITELLM_MASTER_KEY", "sk-gateway")
+ monkeypatch.setenv("OPENAI_API_KEY", "sk-provider")
+ assert resolve_api_key("aisoc-triage") == "sk-gateway"
+ # …and the provider key for a model that goes straight to the provider.
+ assert resolve_api_key("gpt-4o-mini") == "sk-provider"
+
+
+def test_api_key_is_the_operators_when_they_set_the_base_url(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("LITELLM_MASTER_KEY", "sk-gateway")
+ monkeypatch.setenv("OPENAI_BASE_URL", "http://my-vllm:8000/v1")
+ monkeypatch.setenv("OPENAI_API_KEY", "sk-provider")
+ assert resolve_api_key("aisoc-triage") == "sk-provider"
# ── chat_completions_url ─────────────────────────────────────────────────────
@@ -76,6 +127,17 @@ def test_chat_completions_url_appends_path(monkeypatch):
assert chat_completions_url() == "http://litellm:4000/v1/chat/completions"
+def test_chat_completions_url_refuses_an_alias_with_no_gateway():
+ # The loud failure. Previously this returned api.openai.com and the caller
+ # took the resulting 404 as evidence that no model was available.
+ with pytest.raises(UnroutableModelError, match="aisoc-triage"):
+ chat_completions_url("aisoc-triage")
+
+
+def test_chat_completions_url_allows_a_concrete_model_with_no_gateway():
+ assert chat_completions_url("gpt-4o-mini") == _DEFAULT_URL
+
+
# ── make_chat_model ──────────────────────────────────────────────────────────
@@ -86,3 +148,75 @@ def test_make_chat_model_uses_alias_and_base_url(monkeypatch):
# langchain_openai exposes the model id as ``model_name``.
assert llm.model_name == "aisoc-triage"
assert str(llm.openai_api_base).rstrip("/") == "http://litellm:4000/v1"
+
+
+def test_make_chat_model_reaches_the_gateway_from_compose_alone(monkeypatch):
+ # The default deployment: compose sets LLM_GATEWAY_URL + LITELLM_MASTER_KEY
+ # and nothing else. This is the state that could not reach a model.
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("LITELLM_MASTER_KEY", "sk-aisoc-local")
+ llm = make_chat_model("triage")
+ assert llm.model_name == "aisoc-triage"
+ assert str(llm.openai_api_base).rstrip("/") == "http://litellm:4000/v1"
+ assert llm.openai_api_key.get_secret_value() == "sk-aisoc-local"
+
+
+def test_make_chat_model_refuses_an_alias_with_no_gateway(monkeypatch):
+ monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
+ with pytest.raises(UnroutableModelError, match="aisoc-triage"):
+ make_chat_model("triage")
+
+
+def test_byok_model_override_only_when_the_tenant_set_one(monkeypatch):
+ # The second defect: OPENAI_MODEL reached make_chat_model through the BYOK
+ # override and replaced the role alias with a model the gateway 400s. The
+ # override mechanism itself is fine — a real tenant model still wins.
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("LITELLM_MASTER_KEY", "sk-aisoc-local")
+ monkeypatch.setenv("OPENAI_MODEL", "gpt-4-turbo-preview")
+ assert make_chat_model("triage").model_name == "aisoc-triage"
+ with llm_override(model="tenant-model", base_url="http://tenant:1/v1", api_key="sk-tenant"):
+ llm = make_chat_model("triage")
+ assert llm.model_name == "tenant-model"
+ assert str(llm.openai_api_base).rstrip("/") == "http://tenant:1/v1"
+
+
+# ── preflight_llm ────────────────────────────────────────────────────────────
+
+
+def test_preflight_clean_when_the_gateway_is_wired(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("OPENAI_MODEL", "aisoc-summary")
+ assert preflight_llm() == []
+
+
+def test_preflight_reports_an_alias_with_no_gateway():
+ warnings = preflight_llm()
+ assert len(warnings) == 1
+ assert "aisoc-triage" in warnings[0]
+
+
+def test_preflight_reports_a_concrete_pin_aimed_at_the_bundled_gateway(monkeypatch):
+ # The reverse direction, which is the failure the commercial deployment hit:
+ # every call 400'd with "Invalid model name" and degraded to stub output.
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("OPENAI_BASE_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("AISOC_MODEL_PIN_TRIAGE", "gpt-4-turbo-preview")
+ warnings = preflight_llm()
+ assert any("gpt-4-turbo-preview" in w and "does not define" in w for w in warnings)
+
+
+def test_preflight_reports_a_byok_model_the_gateway_lacks(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("OPENAI_MODEL", "gpt-4-turbo-preview")
+ warnings = preflight_llm()
+ assert any("OPENAI_MODEL" in w and "gpt-4-turbo-preview" in w for w in warnings)
+
+
+def test_preflight_quiet_about_a_third_party_gateway(monkeypatch):
+ # An operator pointing at their own vLLM with concrete model names is
+ # correct; only the bundled gateway's contents are knowable from here.
+ monkeypatch.setenv("OPENAI_BASE_URL", "http://my-vllm:8000/v1")
+ monkeypatch.setenv("AISOC_MODEL_PIN_TRIAGE", "Qwen2.5-32B-Instruct")
+ monkeypatch.setenv("OPENAI_MODEL", "Qwen2.5-32B-Instruct")
+ assert preflight_llm() == []
diff --git a/services/agents/tests/test_llm_hot_path.py b/services/agents/tests/test_llm_hot_path.py
index 5961f63c1..5fa86c014 100644
--- a/services/agents/tests/test_llm_hot_path.py
+++ b/services/agents/tests/test_llm_hot_path.py
@@ -50,8 +50,12 @@ def test_llm_override_routes_to_tenant_model(monkeypatch):
with llm_override(api_key="tenant-byok-key", base_url="https://byok.example/v1", model="gpt-4o-tenant"):
m = make_chat_model("triage")
assert getattr(m, "model_name", None) == "gpt-4o-tenant"
- # Outside the override, back to the alias (platform key from env, as in prod).
+ # Outside the override, back to the alias — which needs the gateway that
+ # docker-compose.yml supplies. A provider key alone is not enough to route
+ # an alias, and building a client that will 404 is what used to look like
+ # "the LLM is configured".
monkeypatch.setenv("OPENAI_API_KEY", "sk-platform-test")
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
m2 = make_chat_model("triage")
assert getattr(m2, "model_name", None) == "aisoc-triage"
diff --git a/services/api/app/_vendor/nl_query/translator.py b/services/api/app/_vendor/nl_query/translator.py
index 5de992be3..98e4f6eb5 100644
--- a/services/api/app/_vendor/nl_query/translator.py
+++ b/services/api/app/_vendor/nl_query/translator.py
@@ -930,11 +930,12 @@ async def enhance_with_llm(
{"role": "user", "content": prompt},
]
+ resolved_model = model or resolve_model_alias("nl")
payload = await safe_chat_completions_request(
api_key=api_key,
- model=model or resolve_model_alias("nl"),
+ model=resolved_model,
messages=messages,
- url=chat_completions_url(),
+ url=chat_completions_url(resolved_model),
timeout=timeout,
response_format={"type": "json_object"},
temperature=0,
diff --git a/services/api/app/api/v1/endpoints/hunts.py b/services/api/app/api/v1/endpoints/hunts.py
index 61cb6e24d..0deb48459 100644
--- a/services/api/app/api/v1/endpoints/hunts.py
+++ b/services/api/app/api/v1/endpoints/hunts.py
@@ -39,7 +39,7 @@ from sqlalchemy import text
from app.api.v1.deps import AuthUser, DBSession
from app.core.airgap import AirgapViolation, enforce_airgap_for_url
from app.services.llm_safety import LLMContractViolation, safe_chat_completions_request
-from app.services.model_aliases import resolve_model_alias
+from app.services.model_aliases import chat_completions_url, resolve_api_key, resolve_model_alias
logger = logging.getLogger(__name__)
@@ -137,16 +137,17 @@ No prose. Just JSON."""
async def _generate_queries(hypothesis: str, mitre: str | None) -> dict[str, str] | None:
- api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")
+ model = os.getenv("LLM_MODEL") or resolve_model_alias("nl")
+ # Resolved together with the route: when the call goes to the bundled
+ # gateway the bearer is the gateway's master key, not a provider key.
+ api_key = resolve_api_key(model)
if not api_key:
return None
- base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")
- model = os.getenv("LLM_MODEL") or resolve_model_alias("nl")
user_msg = f"HYPOTHESIS: {hypothesis}"
if mitre:
user_msg += f"\nMITRE TECHNIQUE: {mitre}"
user_msg += "\nGenerate ES|QL, SPL, and KQL hunt queries."
- completions_url = f"{base_url}/chat/completions"
+ completions_url = chat_completions_url(model)
enforce_airgap_for_url(completions_url)
try:
body = await safe_chat_completions_request( # T2.3
diff --git a/services/api/app/api/v1/endpoints/knowledge_base.py b/services/api/app/api/v1/endpoints/knowledge_base.py
index 1f1a6e133..8ccfdd78c 100644
--- a/services/api/app/api/v1/endpoints/knowledge_base.py
+++ b/services/api/app/api/v1/endpoints/knowledge_base.py
@@ -29,7 +29,7 @@ from app.api.v1.deps import AuthUser, DBSession
from app.core.airgap import AirgapViolation, enforce_airgap_for_url
from app.services.kb_chunking import chunk_text
from app.services.llm_safety import LLMContractViolation, safe_chat_completions_request
-from app.services.model_aliases import resolve_model_alias
+from app.services.model_aliases import chat_completions_url, resolve_api_key, resolve_model_alias
logger = logging.getLogger(__name__)
@@ -112,13 +112,14 @@ If the answer cannot be found in the context, say so clearly."""
async def _synthesise(question: str, chunks: list[KBChunk]) -> str | None:
- api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")
+ model = os.getenv("LLM_MODEL") or resolve_model_alias("nl")
+ # Resolved together with the route: when the call goes to the bundled
+ # gateway the bearer is the gateway's master key, not a provider key.
+ api_key = resolve_api_key(model)
if not api_key:
return None
- base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")
- model = os.getenv("LLM_MODEL") or resolve_model_alias("nl")
context = "\n\n".join(f"[{c.title}] chunk {c.chunk_index}:\n{c.content}" for c in chunks)
- completions_url = f"{base_url}/chat/completions"
+ completions_url = chat_completions_url(model)
enforce_airgap_for_url(completions_url)
try:
# T2.3 — the retrieved KB chunks are the untrusted half here: a
diff --git a/services/api/app/api/v1/endpoints/llm_status.py b/services/api/app/api/v1/endpoints/llm_status.py
index 51f969a50..a758b90e0 100644
--- a/services/api/app/api/v1/endpoints/llm_status.py
+++ b/services/api/app/api/v1/endpoints/llm_status.py
@@ -66,6 +66,7 @@ from app.core.airgap import is_host_allowed_for_airgap
from app.core.config import settings
from app.models.llm_credential import TenantLlmCredential
from app.security.credential_vault import CredentialVaultError, get_vault
+from app.services.model_aliases import gateway_url, is_gateway_alias, resolve_api_key
router = APIRouter(prefix="/llm", tags=["llm"])
@@ -145,7 +146,15 @@ def _env_baseline() -> tuple[str, str, bool]:
"""
base_url = os.getenv("LLM_BASE_URL") or os.getenv("OPENAI_BASE_URL") or ""
model = os.getenv("LLM_MODEL") or os.getenv("OPENAI_MODEL") or os.getenv("AISOC_LLM_MODEL") or ""
- key_set = bool(os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY"))
+ if not base_url and gateway_url() and is_gateway_alias(model or "aisoc-summary"):
+ # The indicator has to describe the route the explain path will
+ # actually take, and that path now adopts the compose-provided gateway
+ # for an alias-shaped model. Reporting "no LLM configured" while the
+ # request goes to the gateway would be the same class of defect one
+ # level up.
+ base_url = gateway_url() or ""
+ model = model or "aisoc-summary"
+ key_set = bool(resolve_api_key(model))
return base_url, model, key_set
diff --git a/services/api/app/api/v1/endpoints/nl_detection.py b/services/api/app/api/v1/endpoints/nl_detection.py
index 1d801dfa7..323e7acc9 100644
--- a/services/api/app/api/v1/endpoints/nl_detection.py
+++ b/services/api/app/api/v1/endpoints/nl_detection.py
@@ -14,7 +14,6 @@ functional in all deployment environments.
from __future__ import annotations
import json
-import os
import uuid
from datetime import UTC, datetime
from typing import Annotated, Any, Literal
@@ -29,6 +28,12 @@ from app.models.detection_proposal import DetectionRuleProposal
from app.services.detection_eval import evaluate_candidate_rule
from app.services.fixture_synth import derive_fixtures_from_sigma
from app.services.llm_safety import LLMContractViolation, safe_chat_completions_request
+from app.services.model_aliases import (
+ UnroutableModelError,
+ chat_completions_url,
+ resolve_api_key,
+ resolve_model_alias,
+)
logger = structlog.get_logger()
@@ -122,9 +127,16 @@ FROM logs-*
async def _llm_translate(request: NLDetectionRequest) -> dict[str, str | None]:
"""Attempt LLM-based translation; fall back to templates on any error."""
- api_key = os.environ.get("OPENAI_API_KEY", "").strip()
+ # The detection builder asked ``OPENAI_MODEL`` for its model and posted to a
+ # hardcoded api.openai.com — so it was the one LLM consumer that could not be
+ # pointed anywhere, and the model it sent (`gpt-4-turbo-preview`, from
+ # .env.example) is not one the bundled gateway defines. It is a
+ # natural-language translation task, so it takes the `nl` role like every
+ # other one.
+ model = resolve_model_alias("nl")
+ api_key = resolve_api_key(model) or ""
if not api_key:
- logger.debug("nl_detection.llm_unavailable", reason="no OPENAI_API_KEY")
+ logger.debug("nl_detection.llm_unavailable", reason="no API key resolved for the nl role")
return _template_fallback(request)
platforms_str = ", ".join(request.target_platforms)
@@ -136,14 +148,17 @@ async def _llm_translate(request: NLDetectionRequest) -> dict[str, str | None]:
f"Return JSON with keys matching the platform names (sigma, kql, spl, esql)."
)
- completions_url = "https://api.openai.com/v1/chat/completions"
+ try:
+ completions_url = chat_completions_url(model)
+ except UnroutableModelError as exc:
+ logger.warning("nl_detection.unroutable_model", model=model, reason=str(exc))
+ return _template_fallback(request)
try:
enforce_airgap_for_url(completions_url)
except AirgapViolation as exc:
logger.info("nl_detection.airgap_block", url=completions_url, reason=str(exc))
return _template_fallback(request)
- model = os.environ.get("OPENAI_MODEL", "gpt-4o-mini")
try:
# T2.3 — the description is analyst-authored but routinely contains a
# pasted log line, which is exactly what the contract refuses to
diff --git a/services/api/app/api/v1/endpoints/phishing.py b/services/api/app/api/v1/endpoints/phishing.py
index a336f34de..92606da71 100644
--- a/services/api/app/api/v1/endpoints/phishing.py
+++ b/services/api/app/api/v1/endpoints/phishing.py
@@ -28,7 +28,7 @@ from sqlalchemy import text
from app.api.v1.deps import AuthUser, DBSession
from app.core.airgap import AirgapViolation, enforce_airgap_for_url
from app.services.llm_safety import LLMContractViolation, safe_chat_completions_request
-from app.services.model_aliases import resolve_model_alias
+from app.services.model_aliases import chat_completions_url, resolve_api_key, resolve_model_alias
logger = logging.getLogger(__name__)
@@ -89,17 +89,18 @@ Analyse the submitted artifact and return ONLY valid JSON with:
async def _triage(artifact_kind: str, content: str, urls: list[str]) -> TriageResult | None:
- api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")
+ model = os.getenv("LLM_MODEL") or resolve_model_alias("investigation")
+ # Resolved together with the route: when the call goes to the bundled
+ # gateway the bearer is the gateway's master key, not a provider key.
+ api_key = resolve_api_key(model)
if not api_key:
return None
- base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")
- model = os.getenv("LLM_MODEL") or resolve_model_alias("investigation")
user_msg = f"ARTIFACT TYPE: {artifact_kind}\n"
if urls:
user_msg += f"URLS: {', '.join(urls[:10])}\n"
if content:
user_msg += f"CONTENT (first 2000 chars):\n{content[:2000]}"
- completions_url = f"{base_url}/chat/completions"
+ completions_url = chat_completions_url(model)
# Air-gap enforcement: refuse the call rather than letting httpx fan out.
# AirgapViolation propagates to the caller so the endpoint can surface 503.
enforce_airgap_for_url(completions_url)
diff --git a/services/api/app/api/v1/endpoints/translation.py b/services/api/app/api/v1/endpoints/translation.py
index 77760b7bd..3814f3046 100644
--- a/services/api/app/api/v1/endpoints/translation.py
+++ b/services/api/app/api/v1/endpoints/translation.py
@@ -24,7 +24,7 @@ from pydantic import BaseModel, Field
from app.api.v1.deps import AuthUser
from app.core.airgap import AirgapViolation, enforce_airgap_for_url
from app.services.llm_safety import LLMContractViolation, safe_chat_completions_request
-from app.services.model_aliases import resolve_model_alias
+from app.services.model_aliases import chat_completions_url, resolve_api_key, resolve_model_alias
logger = logging.getLogger(__name__)
@@ -112,12 +112,13 @@ def _user_prompt(req: TranslateRequest) -> str:
async def _llm_translate(req: TranslateRequest) -> dict[str, Any] | None:
- api_key = os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY")
+ model = os.getenv("LLM_MODEL") or resolve_model_alias("nl")
+ # Resolved together with the route: when the call goes to the bundled
+ # gateway the bearer is the gateway's master key, not a provider key.
+ api_key = resolve_api_key(model)
if not api_key:
return None
- base_url = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")
- model = os.getenv("LLM_MODEL") or resolve_model_alias("nl")
- completions_url = f"{base_url}/chat/completions"
+ completions_url = chat_completions_url(model)
# Air-gap check: refuses the call entirely (rather than silently
# falling back to template-substitution) so misconfigurations are
# loud. The except-Exception below would otherwise swallow this.
diff --git a/services/api/app/services/llm_resolver.py b/services/api/app/services/llm_resolver.py
index 29d2e4df4..2bd0c3c18 100644
--- a/services/api/app/services/llm_resolver.py
+++ b/services/api/app/services/llm_resolver.py
@@ -45,12 +45,17 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.models.llm_credential import TenantLlmCredential
from app.security.credential_vault import CredentialVaultError, get_vault
+from app.services.model_aliases import gateway_url, is_gateway_alias, resolve_api_key
logger = logging.getLogger(__name__)
_DEFAULT_OPENAI_BASE = "https://api.openai.com"
_DEFAULT_OPENAI_MODEL = "gpt-4o-mini"
+#: See the agents-side resolver: the explain path's work is a summary, and its
+#: alias is the default model when a gateway is configured, because the bundled
+#: gateway does not define ``gpt-4o-mini``.
+_EXPLAIN_ROLE_ALIAS = "aisoc-summary"
@dataclass(frozen=True)
@@ -80,6 +85,14 @@ class LlmConfig:
api_key: str | None
source: str
reason: str
+ # Per-field provenance — mirrors the agents-side resolver. ``source``
+ # collapses the whole config into one word, which is enough to log and not
+ # enough to act on: a caller that binds this as a BYOK *override* needs to
+ # know which fields the tenant actually set, or an env baseline silently
+ # overrides a per-role model pin.
+ model_from_tenant: bool = False
+ base_url_from_tenant: bool = False
+ api_key_from_tenant: bool = False
def _env_baseline() -> tuple[str, str, str | None]:
@@ -94,7 +107,14 @@ def _env_baseline() -> tuple[str, str, str | None]:
"""
base_url = os.getenv("OPENAI_BASE_URL", "").strip() or os.getenv("LLM_BASE_URL", "").strip()
model = os.getenv("OPENAI_MODEL", "").strip() or os.getenv("LLM_MODEL", "").strip() or os.getenv("AISOC_LLM_MODEL", "").strip()
- api_key = os.getenv("OPENAI_API_KEY", "").strip() or os.getenv("LLM_API_KEY", "").strip()
+ if not base_url and gateway_url() and is_gateway_alias(model or _EXPLAIN_ROLE_ALIAS):
+ # Same rule as every other caller (``app.services.model_aliases``):
+ # the compose-provided gateway is adopted for an alias and never for a
+ # concrete model. Without this the explain path defaulted straight to
+ # ``https://api.openai.com`` and took an alias there.
+ base_url = gateway_url() or ""
+ model = model or _EXPLAIN_ROLE_ALIAS
+ api_key = resolve_api_key(model) or ""
return base_url, model, (api_key or None)
@@ -256,6 +276,9 @@ async def resolve_llm_config(db: AsyncSession, tenant_id: uuid.UUID) -> LlmConfi
api_key=None,
source=source,
reason="no API key configured (neither tenant BYOK nor env)",
+ model_from_tenant=tenant_contributed_model,
+ base_url_from_tenant=tenant_contributed_base_url,
+ api_key_from_tenant=tenant_contributed_key,
)
blocked, reason = _airgap_blocks(base_url)
@@ -269,6 +292,9 @@ async def resolve_llm_config(db: AsyncSession, tenant_id: uuid.UUID) -> LlmConfi
api_key=None,
source=source,
reason=reason,
+ model_from_tenant=tenant_contributed_model,
+ base_url_from_tenant=tenant_contributed_base_url,
+ api_key_from_tenant=tenant_contributed_key,
)
return LlmConfig(
@@ -278,4 +304,7 @@ async def resolve_llm_config(db: AsyncSession, tenant_id: uuid.UUID) -> LlmConfi
api_key=api_key,
source=source,
reason="",
+ model_from_tenant=tenant_contributed_model,
+ base_url_from_tenant=tenant_contributed_base_url,
+ api_key_from_tenant=tenant_contributed_key,
)
diff --git a/services/api/app/services/model_aliases.py b/services/api/app/services/model_aliases.py
index 9893a09e1..b92132d2a 100644
--- a/services/api/app/services/model_aliases.py
+++ b/services/api/app/services/model_aliases.py
@@ -22,6 +22,24 @@ ROLES = frozenset({"triage", "recon", "investigation", "copilot", "summary", "re
DEFAULT_OPENAI_CHAT_COMPLETIONS_URL = "https://api.openai.com/v1/chat/completions"
+#: See :data:`services.agents.app.llm.routing.GATEWAY_ALIAS_PREFIX`.
+GATEWAY_ALIAS_PREFIX = "aisoc-"
+
+#: See :data:`services.agents.app.llm.routing.BYOK_MODEL_ENV_VARS`.
+BYOK_MODEL_ENV_VARS = ("OPENAI_MODEL", "LLM_MODEL", "AISOC_LLM_MODEL")
+
+
+class UnroutableModelError(RuntimeError):
+ """A gateway alias was requested with nowhere to send it.
+
+ Mirror of :class:`services.agents.app.llm.factory.UnroutableModelError`.
+ """
+
+
+def is_gateway_alias(model: str | None) -> bool:
+ """Whether ``model`` is a logical alias that only the gateway can resolve."""
+ return bool(model) and str(model).strip().startswith(GATEWAY_ALIAS_PREFIX)
+
def resolve_model_alias(role: str) -> str:
"""Return the ``aisoc-<role>`` alias, honouring an ``AISOC_MODEL_PIN_<ROLE>`` override."""
@@ -29,18 +47,77 @@ def resolve_model_alias(role: str) -> str:
return override or f"aisoc-{role}"
-def resolve_base_url() -> str | None:
- """OpenAI-compatible base URL for live calls, or ``None`` for the default.
+def _explicit_base_url() -> str | None:
+ """The base URL an operator set by hand, if any."""
+ return os.environ.get("OPENAI_BASE_URL", "").strip() or os.environ.get("LLM_BASE_URL", "").strip() or None
+
+
+def gateway_url() -> str | None:
+ """The bundled LiteLLM gateway's in-network URL, as compose supplies it."""
+ return os.environ.get("LLM_GATEWAY_URL", "").strip() or None
- Mirrors :func:`services.agents.app.llm.factory.resolve_base_url`, including
- its deliberate refusal to auto-adopt ``LLM_GATEWAY_URL``: routing through
- the gateway is an explicit choice so it is never ambiguous whether the
- bearer token is the gateway master key or a provider key.
+
+def resolve_base_url(model: str | None = None) -> str | None:
+ """OpenAI-compatible base URL for ``model``, or ``None`` for the default.
+
+ Mirrors :func:`services.agents.app.llm.factory.resolve_base_url` exactly,
+ including the rule that ``LLM_GATEWAY_URL`` — the only base-URL variable
+ ``docker-compose.yml`` sets — is adopted for an alias-shaped model and left
+ alone for a concrete pinned one.
"""
- return os.environ.get("OPENAI_BASE_URL", "").strip() or os.environ.get("LLM_BASE_URL", "").strip() or None
+ explicit = _explicit_base_url()
+ if explicit:
+ return explicit
+ if model is None or is_gateway_alias(model):
+ return gateway_url()
+ return None
+
+
+def explicit_base_url() -> str | None:
+ """The base URL an operator set by hand, if any."""
+ return _explicit_base_url()
+
+
+def adopted_gateway(model: str | None = None) -> bool:
+ """Whether the base URL for ``model`` came from ``LLM_GATEWAY_URL`` rather than an operator."""
+ return _explicit_base_url() is None and resolve_base_url(model) is not None
+
+
+def at_bundled_gateway(base_url: str | None) -> bool:
+ """Whether ``base_url`` is the gateway this repo ships a model list for.
+
+ Mirrors :func:`services.agents.app.llm.routing.at_bundled_gateway`.
+ """
+ gateway = gateway_url()
+ if not base_url or not gateway:
+ return False
+ return str(base_url).rstrip("/") == gateway.rstrip("/")
+
+
+def resolve_api_key(model: str | None = None) -> str | None:
+ """Bearer token to send alongside :func:`resolve_base_url`'s answer.
+
+ Mirrors :func:`services.agents.app.llm.factory.resolve_api_key`: when AiSOC
+ picks the bundled gateway itself the bearer is ``LITELLM_MASTER_KEY``, never
+ a provider key.
+ """
+ if adopted_gateway(model):
+ return os.environ.get("LITELLM_MASTER_KEY", "").strip() or os.environ.get("OPENAI_API_KEY", "").strip() or None
+ return os.environ.get("OPENAI_API_KEY", "").strip() or os.environ.get("LLM_API_KEY", "").strip() or None
+
+
+def assert_routable(model: str, base_url: str | None) -> None:
+ """Raise :class:`UnroutableModelError` if ``model`` can never resolve at ``base_url``."""
+ if is_gateway_alias(model) and not base_url:
+ raise UnroutableModelError(
+ f"model '{model}' is a LiteLLM gateway alias and no gateway is configured. "
+ "Set LLM_GATEWAY_URL (docker-compose.yml already does) or OPENAI_BASE_URL to the "
+ "gateway, or pin a concrete provider model via AISOC_MODEL_PIN_<ROLE>. "
+ "Sending an alias to a provider default returns 404 and degrades silently."
+ )
-def chat_completions_url() -> str:
+def chat_completions_url(model: str | None = None) -> str:
"""Full chat-completions URL for the raw-HTTP path.
Exists so the NL-query translator — one file vendored into both services —
@@ -50,7 +127,9 @@ def chat_completions_url() -> str:
``/nl-query`` silently never reached a model and always returned the
deterministic translation: safe by accident, and invisible.
"""
- base = resolve_base_url()
+ base = resolve_base_url(model)
+ if model is not None:
+ assert_routable(model, base)
if base:
return base.rstrip("/") + "/chat/completions"
return DEFAULT_OPENAI_CHAT_COMPLETIONS_URL
diff --git a/services/api/tests/test_llm_status.py b/services/api/tests/test_llm_status.py
index 610c64529..af7f3496d 100644
--- a/services/api/tests/test_llm_status.py
+++ b/services/api/tests/test_llm_status.py
@@ -36,6 +36,11 @@ def _clear_llm_env(monkeypatch: pytest.MonkeyPatch):
"OPENAI_API_KEY",
"OPENAI_MODEL",
"AISOC_LLM_MODEL",
+ # The gateway pair the default deployment supplies. The indicator now
+ # reports the route the explain path would actually take, so leaving
+ # these set would make "nothing configured" describe a live gateway.
+ "LLM_GATEWAY_URL",
+ "LITELLM_MASTER_KEY",
):
monkeypatch.delenv(var, raising=False)
monkeypatch.setattr(airgap_module.settings, "AISOC_AIRGAPPED", False)
diff --git a/services/api/tests/test_model_aliases_routing.py b/services/api/tests/test_model_aliases_routing.py
new file mode 100644
index 000000000..d42ddc0ac
--- /dev/null
+++ b/services/api/tests/test_model_aliases_routing.py
@@ -0,0 +1,82 @@
+"""The API service's half of the gateway routing rule.
+
+``services/api`` is a separate package from ``services/agents`` and cannot
+import its factory, so ``app.services.model_aliases`` mirrors
+``app.llm.routing``. Two copies of a rule drift, and the drift is invisible
+until a deployment behaves differently in one service than the other — so the
+same behaviours are asserted on both sides, and
+``scripts/check_llm_model_routing.py`` compares the two module surfaces.
+"""
+
+from __future__ import annotations
+
+import pytest
+from app.services.model_aliases import (
+ ROLES,
+ UnroutableModelError,
+ chat_completions_url,
+ is_gateway_alias,
+ resolve_api_key,
+ resolve_base_url,
+ resolve_model_alias,
+)
+
+_DEFAULT_URL = "https://api.openai.com/v1/chat/completions"
+
+
[email protected](autouse=True)
+def _clean_llm_env(monkeypatch):
+ for var in (
+ "OPENAI_BASE_URL",
+ "LLM_BASE_URL",
+ "LLM_GATEWAY_URL",
+ "LITELLM_MASTER_KEY",
+ "OPENAI_API_KEY",
+ "LLM_API_KEY",
+ *(f"AISOC_MODEL_PIN_{role.upper()}" for role in ROLES),
+ ):
+ monkeypatch.delenv(var, raising=False)
+
+
+def test_alias_shape_matches_the_roles_the_gateway_defines():
+ for role in ROLES:
+ assert resolve_model_alias(role) == f"aisoc-{role}"
+ assert is_gateway_alias(resolve_model_alias(role))
+
+
+def test_base_url_adopts_the_compose_gateway_for_an_alias(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ assert resolve_base_url("aisoc-nl") == "http://litellm:4000/v1"
+
+
+def test_base_url_leaves_a_concrete_model_at_its_provider(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ assert resolve_base_url("gpt-4o-mini") is None
+
+
+def test_explicit_base_url_outranks_the_gateway(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("LLM_BASE_URL", "http://my-vllm:8000/v1")
+ assert resolve_base_url("aisoc-nl") == "http://my-vllm:8000/v1"
+
+
+def test_api_key_pairs_with_the_route(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1")
+ monkeypatch.setenv("LITELLM_MASTER_KEY", "sk-gateway")
+ monkeypatch.setenv("OPENAI_API_KEY", "sk-provider")
+ assert resolve_api_key("aisoc-nl") == "sk-gateway"
+ assert resolve_api_key("gpt-4o-mini") == "sk-provider"
+
+
+def test_completions_url_refuses_an_alias_with_no_gateway():
+ with pytest.raises(UnroutableModelError, match="aisoc-nl"):
+ chat_completions_url("aisoc-nl")
+
+
+def test_completions_url_allows_a_concrete_model_with_no_gateway():
+ assert chat_completions_url("gpt-4o-mini") == _DEFAULT_URL
+
+
+def test_completions_url_appends_the_path_once(monkeypatch):
+ monkeypatch.setenv("LLM_GATEWAY_URL", "http://litellm:4000/v1/")
+ assert chat_completions_url("aisoc-nl") == "http://litellm:4000/v1/chat/completions"