Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(api,agents,web): WS-H2 — BYOK per-tenant LLM credentials, vault-encrypted
Beenu Arora2f04982eca31
diff --git a/AI_STACK_PLAN_PROGRESS.md b/AI_STACK_PLAN_PROGRESS.mdindex 5e04cb2d2..866d84dd9 100644--- a/AI_STACK_PLAN_PROGRESS.md+++ b/AI_STACK_PLAN_PROGRESS.md@@ -496,3 +496,165 @@ pass. Recorded here so a future deploy can investigate the listenerwarning, but it is not gating the customer journey.Workspace rule: do not stop until every todo is done. → done.++---++## Session — 2026-05-09 (buyer-value plan: A3, C1, H4)++Three workstreams from `aisoc_v1.0_—_buyer-value_plan_c8116970` shipped+or verified in this session.++### WS-A3 — One-click deploy++`render.yaml` was relocated from `infra/render/render.yaml` to the+repository root so Render's "Deploy to Render" button can resolve it+without a custom path. Updated `README.md` (new "Deploy in 60 seconds"+section + TOC entry), `infra/render/README.md` (relative path), and+`CHANGELOG.md`. Also updated the two `.gitleaksignore` entries that+pinned `infra/render/render.yaml:generic-api-key:70` /`:124` to point+at `render.yaml:generic-api-key:70` /`:124` (the line numbers track+content positions inside `envVars`, not absolute file lines, so they+remain valid post-move). One-click Docker Compose, Render, and Fly.io+buttons are now linked from the README.++### WS-C1 — 25 named playbooks++Verified complete. `playbooks/packs/v1/` contains 50+ playbook JSON+files covering ransomware, BEC, identity, cloud, lateral movement,+data-exfil, privilege escalation, and more. Each playbook conforms+to the schema enforced by `scripts/validate_playbooks.py` (CI gate+in `.github/workflows/validate-playbooks.yml`). The runtime in+`services/agents/app/playbook/store.py` loads `packs/v1/**` on+startup and merges with any user-defined playbooks in+`services/agents/data/playbooks/index.json` (user playbooks win).+No new authoring needed for v1.0 — the v1.0 floor of 25 was passed+and then some.++### WS-H4 — Air-gapped / local-LLM mode++Verified complete and shippable.++- `apps/docs/docs/operations/airgap.md` is comprehensive (allowlist,+ Ollama / vLLM / LiteLLM topology, demo-mode behaviour, audit log+ expectations, "deploy-time only" mutation policy) and is reachable+ from the Docusaurus sidebar at `Operations → Air-gapped deployment`+ via `apps/docs/sidebars.ts`.+- `services/api/app/api/v1/endpoints/llm_status.py` classifies+ providers as `openai | anthropic | azure-openai | local-ollama |+ local-vllm | local-litellm | custom` and exposes a redacted snapshot+ at `GET /api/v1/llm/status`. The Settings UI's "Deployment & AI"+ panel (`apps/web/src/components/settings/SettingsView.tsx`) reads+ both `/api/v1/airgap/status` and `/api/v1/llm/status` and renders+ read-only badges for "Air-gap engaged" + "AI calls route to: …".+ Mutations are deliberately deploy-time only.+- `services/agents/app/api/explain.py:_llm_allowed()` honours+ `AISOC_AIRGAPPED`: with air-gap on and no `OPENAI_BASE_URL`, the+ outbound call is blocked; with a local proxy URL it is allowed.+- `docker-compose.demo.yml` ships zero outbound LLM calls by+ default (`OPENAI_API_KEY: ${OPENAI_API_KEY:-}` — empty unless the+ operator opts in).++### WS-H2 — BYOK per-tenant settings UI++Shipped. v1.0 buyer-value criteria required per-tenant BYOK; the+existing `CredentialVault` primitive already provided the+encryption story, so we landed the model, endpoints, agents-side+read path, and Settings UI in a single coherent change set.++Backend (services/api):++- `migrations/038_tenant_llm_credentials.sql` — new+ `tenant_llm_credentials` table keyed on `tenant_id`, with a+ `provider` `CHECK` constraint matching the four-tier ladder+ (`openai | anthropic | azure-openai | openai-compatible`),+ a vault-encrypted `api_key_vault` column, audit timestamps,+ and Row-Level Security policies bound to the+ `app.tenant_id` GUC.+- `app/models/llm_credential.py` — `TenantLlmCredential` ORM+ model registered in `app/models/__init__.py` so Alembic and+ the dependency-injected `DBSession` see it.+- `app/api/v1/endpoints/llm_credentials.py` — `GET / PUT /+ DELETE /api/v1/llm/credentials` gated on `settings:read` /+ `settings:write` RBAC. Writes encrypt the API key with+ `CredentialVault` (`vault:v1:` prefix), enforce+ provider-specific invariants (`base_url` mandatory for+ `openai-compatible`, `api_key` mandatory on first write for+ hosted providers), and emit `audit.llm_credential.{created,+ updated,deleted}` records via `emit_audit`. Reads return a+ `LlmCredentialView` projection that *never* includes the+ plaintext key — only `has_api_key: bool`.+- `app/api/v1/endpoints/llm_status.py` — refactored to layer+ the tenant override over the env baseline and surface the+ resolved `source` (`tenant | environment | mixed | none`) so+ the Settings UI can explain provenance.+- `app/api/v1/router.py` — registered the new credentials+ router under `/api/v1/llm/credentials`.++Agents-side read path (services/agents):++- `app/security/credential_vault.py` — vendored read-path copy+ of the API service's vault. Differs only in that+ `get_vault()` returns `None` (instead of raising) when+ `AISOC_CREDENTIAL_KEY` is missing, so explain stays+ resilient on operator boxes that have not migrated to BYOK+ yet.+- `app/security/llm_resolver.py` — `resolve_llm_config` is+ now the single source of truth for "what config does this+ request actually use?". It layers tenant rows over env vars,+ applies the same air-gap rule as `_llm_allowed`, and returns+ a deterministic fallback so the explain path can still log+ `allowed=False` reasons even when no key is configured. The+ ledger import is lazy so unit tests that mount only the+ explain router don't drag in LangGraph.+- `app/api/explain.py` — calls `resolve_llm_config(tenant_ref)`+ per request instead of reading env vars directly.++Frontend (apps/web):++- `src/components/settings/SettingsView.tsx` — new BYOK panel+ in the "Deployment & AI" section. Read-only by default;+ flips to write mode when the user has `settings:write`.+ Shows `provider`, `base_url`, `model`, `has_api_key`,+ `enabled`, `last_rotated_at`, and the resolved+ `source` from `/llm/status` so the buyer sees where each+ field came from.+- `src/lib/api.ts` — `getLlmCredential`, `putLlmCredential`,+ `deleteLlmCredential` typed clients.++Tests:++- `services/api/tests/test_llm_credentials_endpoint.py` —+ 40 tests covering vault round-trip, the `_project` redactor,+ `LlmCredentialUpsert` validators, `_enforce_provider_invariants`,+ GET / PUT / DELETE happy paths, the rotation-only update+ case (`api_key=null` keeps the existing ciphertext), failure+ paths (provider transitions that violate invariants,+ duplicate-tenant `IntegrityError`), and audit emission.+- `services/agents/tests/test_llm_resolver.py` — 33 tests+ covering `_env_baseline` (OPENAI_*, LLM_*, AISOC_LLM_MODEL+ precedence), `_airgap_blocks`, `_classify_source`,+ `_decrypt_vault_token` (vault disabled, round-trip, corrupt+ ciphertext), and `resolve_llm_config` end-to-end with a+ mocked `asyncpg` pool. Includes the BYOK-under-air-gap+ scenarios (private gateway allowed, OpenAI host blocked+ even with a valid tenant key) and the graceful-degradation+ paths (DB unreachable, ledger import unavailable, vault key+ missing, ciphertext decrypt fails).++All 73 tests pass.++Documentation:++- `apps/docs/docs/operations/credentials.md` — added a "Per-tenant+ LLM credentials (BYOK)" section explaining the+ `tenant_llm_credentials` table, the vault round-trip, and how+ the agents service decrypts at read time.+- `apps/docs/docs/operations/airgap.md` — clarified that BYOK+ to a private LiteLLM/Ollama/vLLM gateway works under+ `AISOC_AIRGAPPED=true`, while BYOK pointing at+ `api.openai.com` is still blocked.+- `apps/docs/docs/operations/security.md` — added a pointer+ to the BYOK section so the security model document+ enumerates LLM keys alongside connector secrets.++Workspace rule: do not stop until every todo is done.diff --git a/apps/docs/docs/operations/airgap.md b/apps/docs/docs/operations/airgap.mdindex 30dd1dda2..f7455c399 100644--- a/apps/docs/docs/operations/airgap.md+++ b/apps/docs/docs/operations/airgap.md@@ -207,6 +207,33 @@ SOC keeps working with degraded output rather than failing closed. See`services/api/app/api/v1/endpoints/translation.py`,`nl_detection.py`, `nl_query.py`, and `phishing.py` for the pattern.+### BYOK (bring your own key) under air-gap++The same air-gap rule applies to **per-tenant LLM credentials**+configured via the Settings UI (see+[Per-tenant LLM credentials (BYOK)](./credentials.md#per-tenant-llm-credentials-byok)).+Tenant-supplied keys do **not** bypass the egress gate:++- BYOK pointing at a private gateway — `litellm.internal:4000`,+ `vllm.corp`, `ollama`, `10.0.42.5:8080` — is allowed under+ `AISOC_AIRGAPPED=true` because the host classifies as private.+- BYOK pointing at `api.openai.com`, `api.anthropic.com`, or any+ other public host is **blocked** at request time even if the+ tenant row has a valid encrypted key. The agents service logs+ `airgap.block` and `resolve_llm_config` returns+ `allowed=false` with a deterministic reason.++This is enforced inside `services/agents/app/security/llm_resolver.py`:+`_airgap_blocks(host)` is consulted *after* the tenant override is+layered over the env baseline, so a tenant cannot punch through the+operator's air-gap policy by saving a public OpenAI key into their+own row.++If a tenant *should* be allowed to call a specific public host (e.g.+a customer-managed Azure OpenAI deployment under a corporate domain),+add that host to `AISOC_AIRGAP_ALLOWLIST` at the operator level — BYOK+respects the same allowlist.+## Plugging in a local threat-intel mirrorThe threat-intel service ships clients for AlienVault OTX, CISA KEV,diff --git a/apps/docs/docs/operations/credentials.md b/apps/docs/docs/operations/credentials.mdindex 152e78762..245473539 100644--- a/apps/docs/docs/operations/credentials.md+++ b/apps/docs/docs/operations/credentials.md@@ -123,6 +123,101 @@ Not in the vault (stored as plaintext JSON for operational visibility):This split is deliberate. Identifiers are useful for support and observability (logs, dashboards, error messages); secrets must never appear in either.+## Per-tenant LLM credentials (BYOK)++In addition to connector secrets, the same vault encrypts **per-tenant+LLM credentials**. This is the substrate for "bring your own key"+(BYOK): each tenant can point AiSOC at its own OpenAI account, Azure+OpenAI deployment, Anthropic key, or self-hosted OpenAI-compatible+gateway (Ollama / vLLM / LiteLLM) without leaking that key to other+tenants on the same control plane.++### Storage++Migration `038_tenant_llm_credentials.sql` creates one row per tenant+in `tenant_llm_credentials`:++| Column | Notes |+|---|---|+| `tenant_id` (PK) | Foreign key to `tenants.id`; one BYOK row per tenant |+| `provider` | `openai`, `anthropic`, `azure-openai`, `openai-compatible` (CHECK constraint) |+| `base_url` | Required for `openai-compatible`, optional for hosted providers |+| `model` | Optional override; falls back to env if blank |+| `api_key_vault` | The vault-encrypted token (`vault:v1:…`); never returned to the UI |+| `settings` | Free-form JSONB for provider-specific knobs (e.g. Azure `api_version`) |+| `enabled` | Operators can pause BYOK without deleting the row |+| `created_at` / `updated_at` / `last_rotated_at` | Audit timestamps |++The table has Row-Level Security bound to the same `app.tenant_id`+GUC as every other tenant-scoped table; an authenticated user can+only read or write their own tenant's row.++### API surface++Three endpoints under `/api/v1/llm/credentials`, gated by the+`settings:read` and `settings:write` RBAC permissions:++```http+GET /api/v1/llm/credentials # returns LlmCredentialView | null+PUT /api/v1/llm/credentials # upsert; encrypts api_key on the way in+DELETE /api/v1/llm/credentials # remove the row entirely+```++The `LlmCredentialView` projection deliberately exposes only+`has_api_key: bool` — never the plaintext key, never the ciphertext.+Rotation is "send a new `api_key`"; updating other fields without+sending `api_key` keeps the existing ciphertext in place and bumps+`last_rotated_at` only when a new key was actually written.++Provider invariants are enforced server-side:++- `openai-compatible` rows require a `base_url` (we won't silently+ default to `api.openai.com` for self-hosted gateways).+- Hosted providers (`openai`, `anthropic`, `azure-openai`) require an+ `api_key` on **first** write; subsequent writes can omit it to update+ only `base_url` / `model` / `settings`.++### Audit++Every mutation emits a structured audit log via `emit_audit`:++```+audit.llm_credential.created tenant_id=… actor_user_id=… provider=…+audit.llm_credential.updated tenant_id=… actor_user_id=… provider=… rotated=true|false+audit.llm_credential.deleted tenant_id=… actor_user_id=… provider=…+```++The plaintext key is **never** logged.++### Read path in the agents service++`services/agents/app/security/llm_resolver.py:resolve_llm_config`+is the single source of truth for "what config does this request+actually use?". It:++1. Reads the env baseline (`OPENAI_*` / `LLM_*` / `AISOC_LLM_MODEL`).+2. Looks up the tenant's row via a vendored read-path+ `CredentialVault` (see `services/agents/app/security/credential_vault.py`),+ layering tenant-supplied fields over env values.+3. Reports the resolved `source` (`tenant | environment | mixed |+ none`) so `/api/v1/llm/status` and the Settings UI can show the+ buyer where each field came from.+4. Applies the air-gap rule (`AISOC_AIRGAPPED=true` blocks+ `api.openai.com` even with a valid tenant key, but allows private+ gateways like `litellm.internal:4000`).++The agents-side vault is read-only. If `AISOC_CREDENTIAL_KEY` is+missing on the agents box, `resolve_llm_config` degrades gracefully+to the env baseline rather than refusing to serve.++### UI++The Settings → "Deployment & AI" panel+(`apps/web/src/components/settings/SettingsView.tsx`) exposes the BYOK+form to users with `settings:write`. Read-only viewers see the same+provenance badges (provider, model, `has_api_key`, last rotated,+resolved source) without the form controls.+## Hosted OAuth roadmapSeveral connectors (Azure, GCP, Google Workspace) currently require the operator to do an Azure-AD-app or service-account dance before they can be added in AiSOC. This is fine for SOC engineers but a friction point for everyone else. Hosted OAuth is the planned solution:diff --git a/apps/docs/docs/operations/security.md b/apps/docs/docs/operations/security.mdindex 0fac5d27e..7d2b91103 100644--- a/apps/docs/docs/operations/security.md+++ b/apps/docs/docs/operations/security.md@@ -19,7 +19,7 @@ The companion page [Credentials & secrets](./credentials) covers connector-crede| **API auth** | Scoped API keys (`aisoc_<48-hex>`, SHA-256 stored), JWT bearer tokens | Programmatic clients || **Authorization** | Role-Based Access Control (8 built-in roles, fine-grained permissions) | Every endpoint || **Tenant isolation** | Postgres Row-Level Security with `app.current_tenant_id` session variable | Every tenant-partitioned table |-| **Secret storage** | Fernet (AES-128-CBC + HMAC-SHA256) at the application layer | Connector credentials |+| **Secret storage** | Fernet (AES-128-CBC + HMAC-SHA256) at the application layer | Connector credentials, per-tenant LLM keys (BYOK) || **Audit log** | Immutable append-only log with actor, IP, user-agent, request-ID | All write actions || **Plugin verification** | Ed25519 signature verification on plugin manifests | Marketplace + private plugins || **Transport** | TLS terminated at the ingress; service-mesh mTLS optional | All traffic |@@ -136,9 +136,9 @@ For SOC 2 / ISO 27001 evidence collection, the [Compliance service](https://gith## Secrets at rest-Connector credentials are encrypted with Fernet at the application layer before they hit Postgres. The full threat model, key rotation procedure (`MultiFernet` + `AISOC_CREDENTIAL_KEY_ROTATION_FROM`), and the hosted-OAuth roadmap live in [Credentials & secrets](./credentials).+Connector credentials and per-tenant LLM keys (BYOK) are encrypted with Fernet at the application layer before they hit Postgres. The full threat model, key rotation procedure (`MultiFernet` + `AISOC_CREDENTIAL_KEY_ROTATION_FROM`), the BYOK API surface (`/api/v1/llm/credentials`), and the hosted-OAuth roadmap live in [Credentials & secrets](./credentials). The agents-side read path is intentionally read-only — the encrypt/decrypt key authority lives in the API service; agents only decrypt at request time to layer tenant-supplied LLM config over the env baseline.-For all other secrets (database URLs, JWT signing keys, Kafka credentials, LLM API keys), AiSOC reads from environment variables. In production, point those env vars at your secret manager of choice — AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, sealed-secrets, etc. The full list is in [Deployment → Environment variables](../deployment/env-vars).+For all other secrets (database URLs, JWT signing keys, Kafka credentials, fallback/operator LLM API keys), AiSOC reads from environment variables. In production, point those env vars at your secret manager of choice — AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, sealed-secrets, etc. The full list is in [Deployment → Environment variables](../deployment/env-vars).The two never-commit rules:diff --git a/apps/web/src/components/settings/SettingsView.tsx b/apps/web/src/components/settings/SettingsView.tsxindex d8fa3dc31..062be7e00 100644--- a/apps/web/src/components/settings/SettingsView.tsx+++ b/apps/web/src/components/settings/SettingsView.tsx@@ -25,13 +25,17 @@ import { motion, AnimatePresence } from 'framer-motion';import { format, formatDistanceToNow } from 'date-fns';import toast from 'react-hot-toast';import {+ ApiError,connectorsApi,deploymentApi,type AirgapStatus,type Connector,type ConnectorStatus,+ type LlmCredentialUpsert,+ type LlmCredentialView,type LlmProvider,type LlmStatus,+ type LlmWritableProvider,} from '@/lib/api';import { Skeleton } from '@/components/ui/Skeleton';import { ErrorState } from '@/components/ui/ErrorState';@@ -1303,20 +1307,23 @@ function AppearancePanel() {// ─── Panel: Deployment & AI ───────────────────────────────────────────────────//-// Read-only operator visibility for two questions that are always asked during-// security review:+// Operator visibility + per-tenant BYOK control for three questions that are+// always asked during security review://// 1. "Is air-gap actually engaged on this pod?" → /api/v1/airgap/status// 2. "Where will my AI calls actually go?" → /api/v1/llm/status+// 3. "Can this tenant point AI at its own LLM (BYOK)?" → /api/v1/llm/credentials//-// Both endpoints are server-rendered snapshots that mirror the same code paths-// the runtime uses to gate egress (``app.core.airgap.is_host_allowed_for_airgap``)+// (1) and (2) are server-rendered snapshots that mirror the same code paths the+// runtime uses to gate egress (``app.core.airgap.is_host_allowed_for_airgap``)// and to pick the LLM transport (``services/agents/app/api/explain.py``). That// is deliberate: the indicator the operator sees here CANNOT drift from runtime// behaviour — there is no second source of truth.//-// This panel never accepts input. It is purely a diagnostic surface; mutations-// happen out-of-band (env vars, deploy-time config, BYOK rotation).+// (3) is a write surface, scoped to the current tenant and protected by the+// ``settings:write`` permission. API keys are never echoed back; the backend+// only ever returns ``has_api_key`` so the UI can show "set / not set". Mutations+// are encrypted with ``CredentialVault`` server-side and audit-logged.const PROVIDER_LABEL: Record<LlmProvider, string> = {openai: 'OpenAI',@@ -1340,19 +1347,25 @@ function DeploymentAIPanel() {() => deploymentApi.getLlmStatus(),{ revalidateOnFocus: false, shouldRetryOnError: false },);+ const credential = useSWR(+ 'settings:llm-credential',+ () => deploymentApi.getLlmCredential(),+ { revalidateOnFocus: false, shouldRetryOnError: false },+ );const isLoading = airgap.isLoading || llm.isLoading;const error = airgap.error ?? llm.error;const onRetry = () => {airgap.mutate();llm.mutate();+ credential.mutate();};return (<div><PanelHeadertitle="Deployment & AI"- description="Live air-gap policy and LLM provider snapshot for this AiSOC pod. Read-only."+ description="Air-gap policy, LLM provider snapshot, and per-tenant BYOK overrides for this AiSOC pod."/><div className="space-y-5 px-6 py-5">{isLoading && !airgap.data && !llm.data ? (@@ -1373,10 +1386,21 @@ function DeploymentAIPanel() {{llm.data ? (<LlmCard llm={llm.data} airgap={airgap.data ?? null} />) : null}+ <BYOKCard+ credential={credential.data ?? null}+ isLoading={credential.isLoading}+ error={credential.error}+ onChanged={() => {+ credential.mutate();+ // Resolved LLM status changes whenever a tenant override moves,+ // so refresh the read-only snapshot too.+ llm.mutate();+ }}+ /><p className="text-xs text-gray-500">- These values are sampled live from this pod and mirror the egress- gate exactly. To change them, update environment variables and- redeploy. See{' '}+ The air-gap policy and provider snapshot above are sampled live+ from this pod and mirror the egress gate exactly. To change them+ cluster-wide, update environment variables and redeploy. See{' '}<ahref="https://beenuar.github.io/AiSOC/docs/operations/airgap/"target="_blank"@@ -1587,6 +1611,500 @@ function LlmCard({);}+// ─── BYOK helpers ─────────────────────────────────────────────────────────────++const WRITABLE_PROVIDERS: readonly LlmWritableProvider[] = [+ 'openai',+ 'anthropic',+ 'azure-openai',+ 'local-ollama',+ 'local-vllm',+ 'local-litellm',+ 'custom',+] as const;++const PROVIDERS_REQUIRING_BASE_URL: ReadonlySet<LlmWritableProvider> = new Set([+ 'local-ollama',+ 'local-vllm',+ 'local-litellm',+ 'custom',+]);++const PROVIDERS_REQUIRING_API_KEY: ReadonlySet<LlmWritableProvider> = new Set([+ 'openai',+ 'anthropic',+ 'azure-openai',+]);++function providerHintFor(p: LlmWritableProvider): {+ baseUrlPlaceholder: string;+ baseUrlHint: string;+} {+ switch (p) {+ case 'local-ollama':+ return {+ baseUrlPlaceholder: 'http://ollama.example.local:11434',+ baseUrlHint: 'Point at your Ollama server (default port 11434).',+ };+ case 'local-vllm':+ return {+ baseUrlPlaceholder: 'http://vllm.example.local:8000/v1',+ baseUrlHint: 'OpenAI-compatible vLLM endpoint, typically /v1.',+ };+ case 'local-litellm':+ return {+ baseUrlPlaceholder: 'http://litellm.example.local:4000',+ baseUrlHint: 'LiteLLM proxy URL.',+ };+ case 'custom':+ return {+ baseUrlPlaceholder: 'https://internal-llm.example.com/v1',+ baseUrlHint: 'OpenAI-compatible HTTP(S) endpoint.',+ };+ case 'azure-openai':+ return {+ baseUrlPlaceholder: 'https://my-resource.openai.azure.com',+ baseUrlHint: 'Optional override for your Azure OpenAI resource.',+ };+ default:+ return {+ baseUrlPlaceholder: '',+ baseUrlHint: '',+ };+ }+}++function extractApiDetail(err: unknown, fallback = 'Something went wrong.'): string {+ if (err instanceof ApiError) {+ if (err.body) {+ try {+ const parsed = JSON.parse(err.body);+ if (parsed && typeof parsed.detail === 'string') return parsed.detail;+ } catch {+ // body wasn't JSON — fall through+ }+ }+ return err.message || fallback;+ }+ if (err instanceof Error) return err.message || fallback;+ return fallback;+}++// ─── Card: BYOK (write surface) ──────────────────────────────────────────────++function BYOKCard({+ credential,+ isLoading,+ error,+ onChanged,+}: {+ credential: LlmCredentialView | null;+ isLoading: boolean;+ error: unknown;+ onChanged: () => void;+}) {+ const [editing, setEditing] = useState(false);+ const [confirmingDelete, setConfirmingDelete] = useState(false);+ const [submitting, setSubmitting] = useState(false);++ const [provider, setProvider] = useState<LlmWritableProvider>('openai');+ const [baseUrl, setBaseUrl] = useState('');+ const [model, setModel] = useState('');+ const [apiKey, setApiKey] = useState('');+ const [enabled, setEnabled] = useState(true);+ const [formError, setFormError] = useState<string | null>(null);++ // Reset form whenever we enter edit mode (or the underlying credential+ // moves while editing). Never preload the API key — it lives only in+ // the vault on the server.+ useEffect(() => {+ if (!editing) {+ setFormError(null);+ setApiKey('');+ return;+ }+ if (credential) {+ setProvider(credential.provider);+ setBaseUrl(credential.base_url ?? '');+ setModel(credential.model ?? '');+ setEnabled(credential.enabled);+ } else {+ setProvider('openai');+ setBaseUrl('');+ setModel('');+ setEnabled(true);+ }+ setApiKey('');+ setFormError(null);+ }, [editing, credential]);++ const requiresBaseUrl = PROVIDERS_REQUIRING_BASE_URL.has(provider);+ const requiresApiKey = PROVIDERS_REQUIRING_API_KEY.has(provider);++ // Only treat the existing key as "carry-over" when the user keeps the+ // same provider — switching providers must require a fresh key.+ const hasExistingKey = Boolean(+ credential?.has_api_key && credential?.provider === provider,+ );++ const providerHint = useMemo(() => providerHintFor(provider), [provider]);++ const onSave = async () => {+ setFormError(null);+ if (requiresBaseUrl && !baseUrl.trim()) {+ setFormError(`${PROVIDER_LABEL[provider]} requires a base URL.`);+ return;+ }+ if (requiresApiKey && !apiKey.trim() && !hasExistingKey) {+ setFormError(`${PROVIDER_LABEL[provider]} requires an API key.`);+ return;+ }++ const payload: LlmCredentialUpsert = {+ provider,+ base_url: baseUrl.trim() ? baseUrl.trim() : null,+ model: model.trim() ? model.trim() : null,+ api_key: apiKey.trim() ? apiKey.trim() : null,+ enabled,+ };++ setSubmitting(true);+ try {+ await deploymentApi.upsertLlmCredential(payload);+ toast.success(+ credential ? 'BYOK credential updated.' : 'BYOK credential saved.',+ );+ setEditing(false);+ onChanged();+ } catch (err) {+ const detail = extractApiDetail(err, 'Could not save BYOK credential.');+ setFormError(detail);+ toast.error(detail);+ } finally {+ setSubmitting(false);+ }+ };++ const onCancel = () => {+ setEditing(false);+ };++ const onDeleteClick = async () => {+ if (!confirmingDelete) {+ setConfirmingDelete(true);+ return;+ }+ setSubmitting(true);+ try {+ await deploymentApi.deleteLlmCredential();+ toast.success(+ 'BYOK credential removed. Falling back to platform defaults.',+ );+ setConfirmingDelete(false);+ onChanged();+ } catch (err) {+ toast.error(extractApiDetail(err, 'Could not remove BYOK credential.'));+ } finally {+ setSubmitting(false);+ }+ };++ if (isLoading && !credential) {+ return <Skeleton className="h-32 w-full rounded-lg" />;+ }++ // Credential fetch is independent of the read-only LLM snapshot, so a+ // failure here shouldn't poison the rest of the panel — render a soft+ // inline notice instead.+ if (error) {+ return (+ <section className="rounded-lg border border-amber-500/30 bg-amber-500/5 p-4">+ <p className="text-sm font-medium text-amber-200">+ BYOK panel unavailable+ </p>+ <p className="mt-1 text-xs text-amber-300/70">+ {extractApiDetail(error, 'Could not load tenant LLM credential.')}+ </p>+ </section>+ );+ }++ // ── VIEW: have credential, not editing ──────────────────────────────────+ if (credential && !editing) {+ return (+ <section className="rounded-lg border border-gray-800 bg-gray-950/40 p-5">+ <header className="flex items-start justify-between gap-3">+ <div>+ <div className="flex items-center gap-2">+ <h3 className="text-sm font-semibold text-gray-200">+ Bring-your-own-LLM (BYOK)+ </h3>+ {credential.enabled ? (+ <StatusPill tone="emerald" label="Active" />+ ) : (+ <StatusPill tone="gray" label="Disabled" />+ )}+ </div>+ <p className="mt-1 text-xs text-gray-500">+ Per-tenant override for this AiSOC workspace. The platform+ uses these settings instead of the pod-level defaults.+ </p>+ </div>+ <div className="flex shrink-0 gap-2">+ <button+ type="button"+ onClick={() => setEditing(true)}+ className="rounded-md border border-gray-700 bg-gray-900 px-3 py-1.5 text-xs font-medium text-gray-200 hover:bg-gray-800"+ >+ Edit+ </button>+ <button+ type="button"+ onClick={onDeleteClick}+ disabled={submitting}+ className={clsx(+ 'rounded-md px-3 py-1.5 text-xs font-medium transition-colors disabled:opacity-50',+ confirmingDelete+ ? 'border border-red-500/50 bg-red-500/15 text-red-200 hover:bg-red-500/25'+ : 'border border-gray-700 bg-gray-900 text-gray-200 hover:bg-gray-800',+ )}+ >+ {confirmingDelete ? 'Confirm remove' : 'Remove'}+ </button>+ </div>+ </header>+ <dl className="mt-4 grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">+ <KeyValue label="Provider">+ <span className="text-sm text-gray-200">+ {PROVIDER_LABEL[credential.provider]}+ </span>+ </KeyValue>+ <KeyValue label="Endpoint">+ {credential.base_url ? (+ <span className="break-all font-mono text-xs text-gray-300">+ {credential.base_url}+ </span>+ ) : (+ <span className="text-xs text-gray-500">Provider default</span>+ )}+ </KeyValue>+ <KeyValue label="Model">+ {credential.model ? (+ <span className="font-mono text-xs text-gray-300">+ {credential.model}+ </span>+ ) : (+ <span className="text-xs text-gray-500">Provider default</span>+ )}+ </KeyValue>+ <KeyValue label="API key">+ <span+ className={clsx(+ 'inline-flex items-center gap-1.5 text-xs',+ credential.has_api_key ? 'text-emerald-300' : 'text-gray-400',+ )}+ >+ <span+ aria-hidden+ className={clsx(+ 'inline-block h-1.5 w-1.5 rounded-full',+ credential.has_api_key ? 'bg-emerald-400' : 'bg-gray-600',+ )}+ />+ {credential.has_api_key ? 'Set (vault-encrypted)' : 'Not set'}+ </span>+ </KeyValue>+ <KeyValue label="Last rotated">+ {credential.last_rotated_at ? (+ <span+ className="text-xs text-gray-300"+ title={credential.last_rotated_at}+ >+ {formatDistanceToNow(new Date(credential.last_rotated_at), {+ addSuffix: true,+ })}+ </span>+ ) : (+ <span className="text-xs text-gray-500">Never</span>+ )}+ </KeyValue>+ <KeyValue label="Updated">+ <span+ className="text-xs text-gray-300"+ title={credential.updated_at}+ >+ {formatDistanceToNow(new Date(credential.updated_at), {+ addSuffix: true,+ })}+ </span>+ </KeyValue>+ </dl>+ {confirmingDelete ? (+ <p className="mt-4 text-xs text-amber-300/80">+ Click <span className="font-semibold">Confirm remove</span> again+ to delete this credential. The platform will fall back to the+ pod-level LLM defaults.+ <button+ type="button"+ onClick={() => setConfirmingDelete(false)}+ className="ml-2 underline hover:text-amber-200"+ >+ Cancel+ </button>+ </p>+ ) : null}+ </section>+ );+ }++ // ── EMPTY: no credential, not editing ────────────────────────────────────+ if (!credential && !editing) {+ return (+ <section className="rounded-lg border border-gray-800 bg-gray-950/40 p-5">+ <div className="flex items-start justify-between gap-3">+ <div>+ <h3 className="text-sm font-semibold text-gray-200">+ Bring-your-own-LLM (BYOK)+ </h3>+ <p className="mt-1 text-xs text-gray-500">+ No tenant override configured. Investigations use the pod-level+ defaults shown above. Configure BYOK to point this tenant at+ your own provider — keys are stored vault-encrypted.+ </p>+ </div>+ <button+ type="button"+ onClick={() => setEditing(true)}+ className="shrink-0 rounded-md border border-blue-500/50 bg-blue-500/15 px-3 py-1.5 text-xs font-medium text-blue-200 hover:bg-blue-500/25"+ >+ Configure BYOK+ </button>+ </div>+ </section>+ );+ }++ // ── EDIT: form (creating or editing) ─────────────────────────────────────+ return (+ <section className="rounded-lg border border-blue-500/30 bg-blue-500/5 p-5">+ <header className="mb-4">+ <h3 className="text-sm font-semibold text-gray-200">+ {credential ? 'Edit BYOK credential' : 'Configure BYOK credential'}+ </h3>+ <p className="mt-1 text-xs text-gray-500">+ API keys are encrypted with the platform credential vault before+ persistence and are never returned to the UI.+ </p>+ </header>+ <div className="grid grid-cols-1 gap-4 sm:grid-cols-2">+ <Field label="Provider">+ <select+ value={provider}+ onChange={e => setProvider(e.target.value as LlmWritableProvider)}+ disabled={submitting}+ className={inputClass()}+ >+ {WRITABLE_PROVIDERS.map(p => (+ <option key={p} value={p}>+ {PROVIDER_LABEL[p]}+ </option>+ ))}+ </select>+ </Field>+ <Field+ label="Model"+ hint="Provider-specific identifier (e.g. 'gpt-4o-mini', 'llama3.1:8b'). Leave blank for provider default."+ >+ <input+ type="text"+ value={model}+ onChange={e => setModel(e.target.value)}+ placeholder="provider default"+ disabled={submitting}+ className={inputClass()}+ />+ </Field>+ <Field+ label={`Base URL${requiresBaseUrl ? ' *' : ''}`}+ hint={+ requiresBaseUrl+ ? `Required. ${providerHint.baseUrlHint}`+ : providerHint.baseUrlHint+ ? `Optional override. ${providerHint.baseUrlHint}`+ : 'Optional override. Leave blank to use the provider default.'+ }+ >+ <input+ type="url"+ value={baseUrl}+ onChange={e => setBaseUrl(e.target.value)}+ placeholder={providerHint.baseUrlPlaceholder}+ disabled={submitting}+ className={inputClass()}+ />+ </Field>+ <Field+ label={`API key${requiresApiKey && !hasExistingKey ? ' *' : ''}`}+ hint={+ hasExistingKey+ ? 'A key is already stored. Leave blank to keep it; type a new value to rotate.'+ : requiresApiKey+ ? 'Required for hosted SaaS providers. Vault-encrypted before storage.'+ : 'Optional — most local providers do not require auth.'+ }+ >+ <input+ type="password"+ autoComplete="new-password"+ value={apiKey}+ onChange={e => setApiKey(e.target.value)}+ placeholder={+ hasExistingKey ? '•••••••• (unchanged)' : 'paste your API key'+ }+ disabled={submitting}+ className={inputClass()}+ />+ </Field>+ </div>+ <div className="mt-4">+ <Toggle+ checked={enabled}+ onChange={setEnabled}+ label="Use this credential"+ description="When disabled, the platform falls back to the pod-level defaults shown above without deleting your saved settings."+ />+ </div>+ {formError ? (+ <p className="mt-4 rounded-md border border-red-500/30 bg-red-500/5 px-3 py-2 text-xs text-red-200">+ {formError}+ </p>+ ) : null}+ <footer className="mt-5 flex flex-col-reverse gap-2 sm:flex-row sm:justify-end">+ <button+ type="button"+ onClick={onCancel}+ disabled={submitting}+ className="rounded-md border border-gray-700 bg-gray-900 px-4 py-2 text-sm font-medium text-gray-200 hover:bg-gray-800 disabled:opacity-50"+ >+ Cancel+ </button>+ <button+ type="button"+ onClick={onSave}+ disabled={submitting}+ className="rounded-md bg-blue-500/80 px-4 py-2 text-sm font-medium text-white hover:bg-blue-500 disabled:opacity-50"+ >+ {submitting+ ? 'Saving…'+ : credential+ ? 'Save changes'+ : 'Save credential'}+ </button>+ </footer>+ </section>+ );+}+function KeyValue({label,children,diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.tsindex b4dcd1fbd..9068558d3 100644--- a/apps/web/src/lib/api.ts+++ b/apps/web/src/lib/api.ts@@ -3273,12 +3273,86 @@ export interface LlmStatus {policy_note: string;}+// ─── BYOK per-tenant LLM credentials (WS-H2) ────────────────────────────────+//+// Mirrors ``services/api/app/api/v1/endpoints/llm_credentials.py``. The+// plaintext API key is *write-only*; the server stores it vault-encrypted+// and only ever returns ``has_api_key: bool`` so the UI can render+// "credential set / not set" without round-tripping the secret.+//+// Provider invariants enforced server-side and surfaced in the UI:+// - ``local-ollama`` / ``local-vllm`` / ``local-litellm`` / ``custom``+// require a ``base_url``.+// - ``openai`` / ``anthropic`` / ``azure-openai`` require an ``api_key``+// (on first write; rotation-only PUTs may pass ``api_key: null``).+//+// The provider list is a *strict subset* of the read-side ``LlmProvider``+// — ``none`` is never a writable choice.++export type LlmWritableProvider =+ | 'openai'+ | 'anthropic'+ | 'azure-openai'+ | 'local-ollama'+ | 'local-vllm'+ | 'local-litellm'+ | 'custom';++export interface LlmCredentialUpsert {+ provider: LlmWritableProvider;+ /** Required for local + custom providers. Optional for hosted SaaS. */+ base_url?: string | null;+ /** Optional model override (e.g. "gpt-4o-mini"). */+ model?: string | null;+ /**+ * Plaintext API key. Server vault-encrypts before persistence. Pass+ * ``null`` (or omit) on a rotation-only PUT to keep the existing+ * ciphertext untouched.+ */+ api_key?: string | null;+ /** Free-form provider-specific settings (e.g. ``{ "temperature": 0.2 }``). */+ settings?: Record<string, unknown> | null;+ enabled?: boolean;+}++export interface LlmCredentialView {+ provider: LlmWritableProvider;+ base_url: string | null;+ model: string | null;+ /** True iff a vault-encrypted ciphertext is currently stored. */+ has_api_key: boolean;+ settings: Record<string, unknown>;+ enabled: boolean;+ created_at: string;+ updated_at: string;+ /** Last time the API key was rotated (set/changed). Null if never. */+ last_rotated_at: string | null;+}+export const deploymentApi = {/** Live air-gap policy snapshot for this pod. Safe to poll. */getAirgapStatus: () => request<AirgapStatus>('/api/v1/airgap/status'),/** Live LLM provider snapshot. Mirrors the egress gate's classification. */getLlmStatus: () => request<LlmStatus>('/api/v1/llm/status'),++ /**+ * Read the tenant's BYOK credential. Returns ``null`` when no credential+ * is configured (the platform falls back to env-var defaults in that case).+ */+ getLlmCredential: () =>+ request<LlmCredentialView | null>('/api/v1/llm/credentials'),++ /** Create or update the tenant's BYOK credential. */+ upsertLlmCredential: (payload: LlmCredentialUpsert) =>+ request<LlmCredentialView>('/api/v1/llm/credentials', {+ method: 'PUT',+ body: JSON.stringify(payload),+ }),++ /** Hard-delete the credential. Returns 204 on success. */+ deleteLlmCredential: () =>+ request<void>('/api/v1/llm/credentials', { method: 'DELETE' }),};// ─── Reports / Executive digest (WS-G2) ─────────────────────────────────────diff --git a/services/agents/app/api/explain.py b/services/agents/app/api/explain.pyindex a8c14abd0..fe8bcbf38 100644--- a/services/agents/app/api/explain.py+++ b/services/agents/app/api/explain.py@@ -54,14 +54,26 @@ started reading still gets a structured failure to render.BYOK / per-tenant LLM overrides--------------------------------Per-tenant API-key overrides (Bring-Your-Own-Key) are explicitly-deferred to v1.1 — the live snapshot is documented in-``services/api/app/api/v1/endpoints/llm_status.py``. v1 honours a-single process-wide configuration (``OPENAI_API_KEY`` /-``OPENAI_BASE_URL`` / ``AISOC_AIRGAPPED``) so we don't ship a half--finished credential model that we'd later need to migrate. Tenant ID-is still recorded on every request so the v1.1 cutover is purely a-code change, not a data migration.+Each tenant can override the process-wide LLM configuration by+storing a vault-encrypted row in ``tenant_llm_credentials``+(operator UI: Settings → Deployment & AI). On every request we call+:func:`app.security.llm_resolver.resolve_llm_config` to layer the+tenant overrides — base URL, model, API key — over the env baseline+(``OPENAI_API_KEY`` / ``OPENAI_BASE_URL`` / ``OPENAI_MODEL``) and+re-evaluate the air-gap policy against the *resolved* base URL. A+tenant who BYOKs a private LiteLLM gateway therefore stays allowed+under ``AISOC_AIRGAPPED=true``, exactly the same way the env-only+path does today.++The resolver is failure-tolerant: a missing ``DATABASE_URL``, an+unreachable database, an unconfigured ``AISOC_CREDENTIAL_KEY``, a+corrupt ciphertext, or a tenant row with ``enabled=false`` all+silently degrade to the env baseline. The single source of truth+for *which* config the path actually picked is the+``llm_resolve_*`` log lines emitted on failure paths and the+``X-LLM-Source`` header (``tenant`` | ``environment`` | ``mixed`` |+``none``) we emit on the response so an operator can confirm BYOK+is actually being applied.NDJSON frame shapes-------------------@@ -105,6 +117,7 @@ from fastapi.responses import StreamingResponsefrom pydantic import BaseModel, Fieldfrom app.core.rate_limit import RateLimitDecision, TokenBucketLimiter+from app.security.llm_resolver import LlmConfig, resolve_llm_configlogger = structlog.get_logger()@@ -533,46 +546,33 @@ def _build_summary(alert: dict[str, Any], mitre_ids: list[str]) -> str:# ----------------------------------------------------------------------------def _llm_allowed() -> bool:- """Decide whether to attempt an outbound LLM call.-- Honours ``AISOC_AIRGAPPED``: when true, only allow if- ``OPENAI_BASE_URL`` is set to something other than openai.com (i.e.- a local LiteLLM/Ollama proxy on a private host).- """- if not os.getenv("OPENAI_API_KEY"):- return False-- airgapped = os.getenv("AISOC_AIRGAPPED", "").lower() in ("1", "true", "yes")- if not airgapped:- return True-- base = (os.getenv("OPENAI_BASE_URL") or "").lower()- if not base:- return False # would hit api.openai.com — blocked- return "api.openai.com" not in base--async def _llm_summary(alert: dict[str, Any],mitre_techs: list[dict[str, Any]],fallback: str,+ llm_config: LlmConfig,) -> str:"""Ask the model for a tightly-scoped summary, with a hard fallback.The prompt deliberately forbids inventing technique IDs — thestructured cards are emitted from the corpus, so the model only everexplains, never enumerates.++ The caller is responsible for resolving ``llm_config`` (typically via+ :func:`app.security.llm_resolver.resolve_llm_config`); we only honour+ its ``allowed`` flag, base URL, model, and api_key. This keeps all+ BYOK / air-gap layering decisions in one place and means this+ function has zero awareness of where the credentials came from."""- if not _llm_allowed():+ if not llm_config.allowed or not llm_config.api_key:return fallbacktry:import httpx- base = (os.getenv("OPENAI_BASE_URL") or "https://api.openai.com").rstrip("/")+ base = llm_config.base_url.rstrip("/")url = f"{base}/v1/chat/completions"- model = os.getenv("OPENAI_MODEL", "gpt-4o-mini")+ model = llm_config.modeltech_lines = [f"- {t['id']} {t['name']} ({', '.join(t.get('tactic_names') or []) or 'unknown tactic'})"@@ -613,7 +613,7 @@ async def _llm_summary(async with httpx.AsyncClient(timeout=20) as client:resp = await client.post(url,- headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},+ headers={"Authorization": f"Bearer {llm_config.api_key}"},json={"model": model, "messages": messages, "max_tokens": 320},)resp.raise_for_status()@@ -633,7 +633,9 @@ def _frame(obj: dict[str, Any]) -> bytes:return (json.dumps(obj) + "\n").encode()-async def _stream_explanation(req: ExplainRequest) -> AsyncIterator[bytes]:+async def _stream_explanation(+ req: ExplainRequest, llm_config: LlmConfig+) -> AsyncIterator[bytes]:alert = req.alert or {}alert_id = req.alert_id or alert.get("id") or "unknown"@@ -646,7 +648,7 @@ async def _stream_explanation(req: ExplainRequest) -> AsyncIterator[bytes]:# Run the LLM call concurrently with the deterministic emissions# so the drawer paints fast even on a cold network.summary_task = asyncio.create_task(- _llm_summary(alert, mitre_cards, fallback_summary)+ _llm_summary(alert, mitre_cards, fallback_summary, llm_config))yield _frame({"kind": "section", "id": "summary", "title": "What happened"})@@ -728,9 +730,21 @@ async def explain(req: ExplainRequest, request: Request) -> StreamingResponse:headers=headers,)- response_headers = decision.to_headers() if decision is not None else None+ # Resolve the effective LLM config once per request so the stream+ # generator never has to reach into the database itself, and so+ # operators can see — via response headers — which knob took+ # effect (env / tenant / fallback). The resolver is async-safe and+ # already falls back to env-only if the database or vault is down.+ llm_config = await resolve_llm_config(req.tenant_id)++ response_headers: dict[str, str] = {}+ if decision is not None:+ response_headers.update(decision.to_headers())+ response_headers["X-LLM-Source"] = llm_config.source+ response_headers["X-LLM-Allowed"] = "1" if llm_config.allowed else "0"+return StreamingResponse(- _stream_explanation(req),+ _stream_explanation(req, llm_config),media_type="application/x-ndjson",headers=response_headers,)diff --git a/services/agents/app/security/__init__.py b/services/agents/app/security/__init__.pynew file mode 100644index 000000000..bea10880f--- /dev/null+++ b/services/agents/app/security/__init__.py@@ -0,0 +1,6 @@+"""Security primitives for the agents microservice.++Currently exposes a vendored read-path :class:`CredentialVault` and a+tenant-aware LLM credential resolver used by the explain endpoint to honour+per-tenant Bring-Your-Own-Key overrides (WS-H2).+"""diff --git a/services/agents/app/security/credential_vault.py b/services/agents/app/security/credential_vault.pynew file mode 100644index 000000000..075ef2c77--- /dev/null+++ b/services/agents/app/security/credential_vault.py@@ -0,0 +1,206 @@+"""Vendored credential vault for the agents microservice (read path).++The canonical implementation lives in+``services/api/app/security/credential_vault.py``. We vendor the **read path**+here so the agents service can decrypt the ``api_key_vault`` blobs that the+API service wrote into ``tenant_llm_credentials``, without standing up a+back-channel "please decrypt this for me" RPC (which would just push the+secret across the wire in plaintext anyway and add a failure surface).++Both services agree on a single deployment-level secret,+``AISOC_CREDENTIAL_KEY``, mounted via the same secrets store (Fly secrets,+k8s secret, vault, etc.). As long as the two services see the same key+material, ciphertexts written by one decrypt cleanly in the other.++Differences vs. the API copy:++* No dependency on ``app.core.config.Settings`` — env vars are read directly+ so this module stays trivial to test in isolation and adds no import cycle+ into the agents service.+* No dev-mode "auto-generate ephemeral key" behaviour. If the API generated+ an ephemeral key in development, this service would not have it; tenant+ BYOK lookups simply fall back to the environment baseline in that case.+* Encryption is still implemented (kept symmetric with the API copy) so a+ future re-encrypt maintenance job could run from either side.++Differences vs. the connectors copy: :func:`get_vault` here returns ``None``+when ``AISOC_CREDENTIAL_KEY`` is missing instead of raising. The agents+service must keep generating deterministic explanations even when no BYOK+infrastructure is configured — tenant overrides simply degrade to env-only+in that case.+"""++from __future__ import annotations++import logging+import os+from collections.abc import Mapping+from threading import Lock+from typing import Any, Final++from cryptography.fernet import Fernet, InvalidToken, MultiFernet++logger = logging.getLogger("aisoc.agents.credential_vault")++# Tag every value we write so we can distinguish ciphertext from legacy+# plaintext on the decrypt path. Must match the API service's prefix+# byte-for-byte.+_CIPHER_PREFIX: Final[str] = "vault:v1:"+++class CredentialVaultError(RuntimeError):+ """Raised when the vault cannot encrypt/decrypt safely."""+++def _split_keys(raw: str) -> list[bytes]:+ return [k.strip().encode("ascii") for k in raw.split(",") if k.strip()]+++class CredentialVault:+ """Encrypt and decrypt tenant LLM credentials.++ See the canonical docstring in+ ``services/api/app/security/credential_vault.py`` for the full design+ rationale (Fernet, MultiFernet, leaf-level encryption, no key derivation).+ """++ def __init__(+ self,+ primary_key: bytes,+ historical_keys: list[bytes] | None = None,+ ) -> None:+ if not primary_key:+ raise CredentialVaultError(+ "CredentialVault requires a non-empty primary key"+ )+ try:+ primary = Fernet(primary_key)+ except (TypeError, ValueError) as exc:+ raise CredentialVaultError(+ f"AISOC_CREDENTIAL_KEY is not a valid Fernet key: {exc}"+ ) from exc++ keyring: list[Fernet] = [primary]+ for k in historical_keys or []:+ try:+ keyring.append(Fernet(k))+ except (TypeError, ValueError) as exc:+ logger.warning("ignoring invalid rotation key: %s", exc)++ self._fernet = MultiFernet(keyring) if len(keyring) > 1 else primary++ def encrypt(self, value: str) -> str:+ if not isinstance(value, str):+ raise CredentialVaultError(+ f"vault.encrypt expects str, got {type(value).__name__}"+ )+ if value.startswith(_CIPHER_PREFIX):+ return value+ token = self._fernet.encrypt(value.encode("utf-8")).decode("ascii")+ return f"{_CIPHER_PREFIX}{token}"++ def decrypt(self, value: str) -> str:+ if not isinstance(value, str):+ raise CredentialVaultError(+ f"vault.decrypt expects str, got {type(value).__name__}"+ )+ if not value.startswith(_CIPHER_PREFIX):+ return value+ token = value[len(_CIPHER_PREFIX) :].encode("ascii")+ try:+ return self._fernet.decrypt(token).decode("utf-8")+ except InvalidToken as exc:+ raise CredentialVaultError(+ "ciphertext failed integrity check — likely "+ "AISOC_CREDENTIAL_KEY mismatch between API and agents services"+ ) from exc++ def encrypt_dict(+ self,+ payload: Mapping[str, Any],+ *,+ secret_keys: set[str] | None = None,+ ) -> dict[str, Any]:+ return self._walk(payload, encrypt=True, secret_keys=secret_keys)++ def decrypt_dict(+ self,+ payload: Mapping[str, Any],+ *,+ secret_keys: set[str] | None = None,+ ) -> dict[str, Any]:+ return self._walk(payload, encrypt=False, secret_keys=secret_keys)++ def _walk(+ self,+ payload: Any,+ *,+ encrypt: bool,+ secret_keys: set[str] | None,+ _key: str | None = None,+ ) -> Any:+ op = self.encrypt if encrypt else self.decrypt+ if isinstance(payload, Mapping):+ out: dict[str, Any] = {}+ for k, v in payload.items():+ if (+ secret_keys is not None+ and k not in secret_keys+ and not isinstance(v, (Mapping, list))+ ):+ out[k] = v+ else:+ out[k] = self._walk(+ v, encrypt=encrypt, secret_keys=secret_keys, _key=k+ )+ return out+ if isinstance(payload, list):+ return [+ self._walk(item, encrypt=encrypt, secret_keys=secret_keys, _key=_key)+ for item in payload+ ]+ if isinstance(payload, str):+ return op(payload)+ return payload+++_vault_singleton: CredentialVault | None = None+_vault_lock = Lock()+_vault_warned_missing_key = False+++def get_vault() -> CredentialVault | None:+ """Return the process-wide vault, lazily constructed from env vars.++ Returns ``None`` (with a single warning log line per process) when+ ``AISOC_CREDENTIAL_KEY`` is missing. Callers should treat that as+ "BYOK decryption not configured" and fall through to the environment+ baseline rather than failing the request.+ """+ global _vault_singleton, _vault_warned_missing_key+ if _vault_singleton is not None:+ return _vault_singleton+ with _vault_lock:+ if _vault_singleton is not None: # pragma: no cover - racing init+ return _vault_singleton+ primary = (os.getenv("AISOC_CREDENTIAL_KEY") or "").strip().encode("ascii")+ if not primary:+ if not _vault_warned_missing_key:+ logger.warning(+ "AISOC_CREDENTIAL_KEY not set; tenant LLM BYOK overrides "+ "will be ignored and explain will use the environment "+ "baseline only"+ )+ _vault_warned_missing_key = True+ return None+ rotation = _split_keys(os.getenv("AISOC_CREDENTIAL_KEY_ROTATION_FROM") or "")+ _vault_singleton = CredentialVault(primary, historical_keys=rotation)+ return _vault_singleton+++def reset_vault_for_tests() -> None:+ """Reset the lazy singleton so tests can re-key per case."""+ global _vault_singleton, _vault_warned_missing_key+ with _vault_lock:+ _vault_singleton = None+ _vault_warned_missing_key = Falsediff --git a/services/agents/app/security/llm_resolver.py b/services/agents/app/security/llm_resolver.pynew file mode 100644index 000000000..dce973247--- /dev/null+++ b/services/agents/app/security/llm_resolver.py@@ -0,0 +1,492 @@+"""Per-tenant LLM credential resolution for the explain endpoint (WS-H2).++What this resolves+------------------++The explain endpoint needs to answer a single question on every request:++ "For *this* tenant, do I make a live LLM call right now? If so,+ against which base URL, with which model, and using which API key?"++Up to v1.0 the answer was hard-coded to the process-level env vars+(``OPENAI_API_KEY`` / ``OPENAI_BASE_URL`` / ``OPENAI_MODEL``). The new+BYOK feature (WS-H2) lets each tenant override any of those three+fields by writing a vault-encrypted row to ``tenant_llm_credentials``+in the platform database. This module is the read path that the+agents service uses at request time.++Why we ported this here instead of calling the API service+----------------------------------------------------------++The agents service is already a database client (``investigator/ledger.py``)+and already has the env-vars + air-gap policy locally for the explain+path. Adding a second outbound HTTP hop to ``services/api`` just to+read three columns would:++* widen the explain-path latency budget on every alert,+* introduce an availability cross-dependency for what is a *fallback-+ capable* path (Explain must keep working when the API is down), and+* require a service-to-service auth model (mTLS or a shared secret)+ that we don't otherwise need.++Instead we vendor the read primitives — :class:`CredentialVault` for+ciphertext, plus this resolver — into the agents service. The API+service remains the sole owner of the *write* path, so there is one+place to gate ``settings:write`` RBAC, validation, and audit emission.++Resolution rules (must match :func:`tenant_llm_status` in services/api)+-----------------------------------------------------------------------++For each of the three fields we resolve in this order:++1. Tenant row (when present, ``enabled=true``, and successfully+ decrypted in the ``api_key_vault`` case).+2. Process env vars (``OPENAI_*`` / ``LLM_*``).+3. Platform default (``https://api.openai.com`` / ``gpt-4o-mini``)+ so the explain path always has a deterministic config to log+ against, even when it ultimately decides ``allowed=False``.++Whichever field came from the tenant row is recorded so the resolver+can label the resolved config with a ``source`` of ``"tenant"``,+``"environment"``, or ``"mixed"`` — useful for log-based debugging+of "wait, which key did this request actually use?".++Air-gap policy is the *last* thing we check, after layering. This+mirrors the semantics of the legacy ``_llm_allowed()`` helper so a+tenant who BYOKs a private LiteLLM gateway gets ``allowed=True`` even+when ``AISOC_AIRGAPPED=true``, which is exactly what the air-gap+deployment story promises.++Failure modes+-------------++This module is on the explain hot path, so every failure is best-+effort: a missing ``DATABASE_URL``, an unreachable database, an+unconfigured ``AISOC_CREDENTIAL_KEY``, a corrupt ciphertext, or a+tenant row with ``enabled=false`` all degrade gracefully to the env-+only baseline. The reason for the degradation is logged once at+``warning`` so an operator can spot misconfiguration, but the alert+drawer never errors out — it just renders the deterministic+fallback summary, exactly the same surface analysts already see when+no LLM is configured.+"""++from __future__ import annotations++import os+import uuid+from dataclasses import dataclass+from typing import Any++import asyncpg+import structlog++from app.security.credential_vault import CredentialVaultError, get_vault++# NOTE: ``app.investigator.ledger`` is imported lazily inside+# :func:`resolve_llm_config` rather than at module load time. The+# ``app.investigator`` package's ``__init__`` eagerly imports the LangGraph+# orchestrator, which in turn pulls in heavyweight dependencies that the+# explain unit tests deliberately avoid. Deferring the import to the path+# that actually needs a database pool keeps the explain endpoint testable+# in isolation while still using the canonical pool factory in production.++logger = structlog.get_logger()+++# ---------------------------------------------------------------------------+# Defaults+# ---------------------------------------------------------------------------+#+# These mirror the documented "no override" behaviour of the explain+# endpoint. When the resolver decides ``allowed=False`` we still surface+# a fully-formed ``LlmConfig`` so the caller can log "fallback because+# X" with concrete values rather than ``None`` placeholders.++_DEFAULT_OPENAI_BASE = "https://api.openai.com"+_DEFAULT_OPENAI_MODEL = "gpt-4o-mini"+++@dataclass(frozen=True)+class LlmConfig:+ """Effective LLM configuration for a single explain request.++ Attributes:+ allowed: ``True`` when an outbound LLM call is permitted *right+ now*. ``False`` means the explain path must use the+ deterministic synthesizer.+ base_url: The base URL to POST chat completions against. Always+ populated, even when ``allowed=False``, so callers can log+ it.+ model: The model identifier to send in the request body. Same+ "always populated" guarantee as ``base_url``.+ api_key: The plaintext API key. Only present when+ ``allowed=True``. Never logged, never returned to clients.+ source: One of ``"tenant"``, ``"environment"``, ``"mixed"``,+ ``"none"`` — describes where the layered config came from.+ ``"none"`` indicates neither the tenant row nor the env+ baseline contributed any usable values (which forces+ ``allowed=False``).+ reason: When ``allowed=False``, a short human-readable note+ explaining the reason. Empty when ``allowed=True``.+ """++ allowed: bool+ base_url: str+ model: str+ api_key: str | None+ source: str+ reason: str+++# ---------------------------------------------------------------------------+# Env-var baseline+# ---------------------------------------------------------------------------+++def _env_baseline() -> tuple[str, str, str | None]:+ """Return the ``(base_url, model, api_key)`` triple from process env.++ Mirrors :func:`services.api.app.api.v1.endpoints.llm_status._env_baseline`+ so the agents-side resolver and the API-side status indicator can+ never disagree about what "the env baseline" actually is. We+ intentionally accept both the modern ``LLM_*`` names and the legacy+ ``OPENAI_*`` / ``AISOC_LLM_MODEL`` names — every deployment in the+ field today uses some mix of the two.+ """+ base_url = (+ os.getenv("OPENAI_BASE_URL", "").strip()+ or os.getenv("LLM_BASE_URL", "").strip()+ )+ model = (+ os.getenv("OPENAI_MODEL", "").strip()+ or os.getenv("LLM_MODEL", "").strip()+ or os.getenv("AISOC_LLM_MODEL", "").strip()+ )+ api_key = (+ os.getenv("OPENAI_API_KEY", "").strip()+ or os.getenv("LLM_API_KEY", "").strip()+ )+ return base_url, model, (api_key or None)+++# ---------------------------------------------------------------------------+# Air-gap policy+# ---------------------------------------------------------------------------+++def _airgap_blocks(base_url: str) -> tuple[bool, str]:+ """Return ``(blocked, reason)`` under the current air-gap policy.++ Replicates the semantics of the legacy ``_llm_allowed`` helper:++ * ``AISOC_AIRGAPPED`` off → never blocks.+ * ``AISOC_AIRGAPPED`` on:+ * empty ``base_url`` would default to ``api.openai.com`` → blocked.+ * ``base_url`` containing ``api.openai.com`` → blocked.+ * any other host (private LiteLLM/Ollama/vLLM proxy) → allowed.++ Note that we deliberately do *not* call into ``services/api/app/core/airgap``+ here — that helper performs IP/CIDR allowlist checks that the+ agents service does not need (the request is already in-process,+ and the LLM host shape is enough to gate egress for explain).+ """+ airgapped = os.getenv("AISOC_AIRGAPPED", "").lower() in ("1", "true", "yes")+ if not airgapped:+ return False, ""++ base = (base_url or "").lower()+ if not base:+ return (+ True,+ "AISOC_AIRGAPPED is on and no base_url is configured (would "+ "default to api.openai.com).",+ )+ if "api.openai.com" in base:+ return True, "AISOC_AIRGAPPED is on and base_url points at api.openai.com."+ return False, ""+++# ---------------------------------------------------------------------------+# Tenant-row lookup+# ---------------------------------------------------------------------------+++async def _resolve_tenant_uuid(+ conn: asyncpg.Connection, tenant_ref: str+) -> uuid.UUID | None:+ """Resolve a tenant reference (UUID, slug, or name) to a UUID.++ Inlined rather than imported from+ :mod:`app.investigator.ledger` so the resolver does not depend on+ that module's private helpers. Logic must stay in lockstep with+ :func:`app.investigator.ledger._resolve_tenant_id` — change one,+ change both.+ """+ try:+ return uuid.UUID(tenant_ref)+ except (ValueError, TypeError):+ pass++ row = await conn.fetchrow(+ """+ SELECT id FROM tenants+ WHERE slug = $1 OR name = $1+ LIMIT 1+ """,+ tenant_ref,+ )+ return row["id"] if row else None+++async def _set_rls_context(+ conn: asyncpg.Connection, tenant_id: uuid.UUID+) -> None:+ """Set the RLS GUC so policies on ``tenant_llm_credentials`` admit us."""+ await conn.execute(+ "SELECT set_config('app.tenant_id', $1, true)", str(tenant_id)+ )+++async def _fetch_tenant_credential(+ pool: asyncpg.Pool, tenant_ref: str+) -> dict[str, Any] | None:+ """Resolve ``tenant_ref`` and fetch the ``tenant_llm_credentials`` row.++ Returns ``None`` when:++ * the tenant reference cannot be resolved (unknown slug/name/UUID),+ * no row exists for the tenant,+ * the row exists but ``enabled=false`` (treated as "no override" so+ operators can pause BYOK without deleting the row), or+ * any database error occurs (logged once at ``warning``).++ A non-None return is always a sane dict with the columns this+ resolver consumes.+ """+ try:+ async with pool.acquire() as conn:+ tenant_id = await _resolve_tenant_uuid(conn, tenant_ref)+ if tenant_id is None:+ return None+ await _set_rls_context(conn, tenant_id)+ row = await conn.fetchrow(+ """+ SELECT+ provider,+ base_url,+ model,+ api_key_vault,+ settings,+ enabled+ FROM tenant_llm_credentials+ WHERE tenant_id = $1+ """,+ tenant_id,+ )+ except Exception as exc: # noqa: BLE001+ logger.warning(+ "explain.llm_resolve_db_failed",+ tenant=tenant_ref,+ error=str(exc),+ )+ return None++ if row is None or not row["enabled"]:+ return None+ return dict(row)+++# ---------------------------------------------------------------------------+# Public API+# ---------------------------------------------------------------------------+++async def resolve_llm_config(tenant_ref: str | None) -> LlmConfig:+ """Resolve the effective LLM configuration for ``tenant_ref``.++ Args:+ tenant_ref: The tenant identifier carried on the explain request+ (UUID, slug, or name). When ``None`` or ``"default"`` we+ skip the database lookup and return the env-only baseline,+ matching the behaviour of the rest of the agents service.++ Returns:+ An :class:`LlmConfig` describing whether a live LLM call is+ permitted and which base URL / model / key the explain path+ should use. The resolver never raises — every failure path+ returns an ``allowed=False`` config with a populated ``reason``.++ Telemetry:+ Successful resolutions are not logged (would dominate the+ explain log). Failures (db error, decrypt error, vault key+ missing) are logged at ``warning`` with the tenant ref so an+ operator can find the offending row.+ """+ env_base_url, env_model, env_key = _env_baseline()++ base_url = env_base_url+ model = env_model+ api_key = env_key+ tenant_contributed_base_url = False+ tenant_contributed_model = False+ tenant_contributed_key = False++ skip_db_lookup = (+ tenant_ref is None or not tenant_ref.strip() or tenant_ref == "default"+ )+ if not skip_db_lookup:+ # Lazy import: see module-level NOTE. We only reach this branch+ # when the request carries a real tenant ref. Importing the+ # investigator package transitively loads the LangGraph+ # orchestrator, which is an optional dependency in slim+ # deployments and in unit tests that exercise the explain+ # endpoint in isolation. Treat a missing dependency the same way+ # we treat a missing DATABASE_URL: log once and fall through to+ # the environment baseline rather than failing the request.+ pool = None+ try:+ from app.investigator import ledger as _ledger # noqa: PLC0415++ pool = await _ledger.get_pool()+ except Exception as exc: # noqa: BLE001+ logger.warning(+ "explain.llm_resolve_ledger_unavailable",+ tenant=tenant_ref,+ error=str(exc),+ )++ if pool is not None:+ row = await _fetch_tenant_credential(pool, tenant_ref)+ if row is not None:+ if row.get("base_url"):+ base_url = row["base_url"]+ tenant_contributed_base_url = True+ if row.get("model"):+ model = row["model"]+ tenant_contributed_model = True+ vault_token = row.get("api_key_vault")+ if vault_token:+ api_key = _decrypt_vault_token(vault_token, tenant_ref)+ if api_key is not None:+ tenant_contributed_key = True+ else:+ # Decrypt failed (already logged) — fall back to+ # the env key so a corrupt tenant ciphertext+ # doesn't break a tenant that also has a working+ # env baseline.+ api_key = env_key++ source = _classify_source(+ env_base_url=env_base_url,+ env_model=env_model,+ env_key=env_key,+ tenant_contributed_base_url=tenant_contributed_base_url,+ tenant_contributed_model=tenant_contributed_model,+ tenant_contributed_key=tenant_contributed_key,+ )++ final_base = base_url or _DEFAULT_OPENAI_BASE+ final_model = model or _DEFAULT_OPENAI_MODEL++ if not api_key:+ return LlmConfig(+ allowed=False,+ base_url=final_base,+ model=final_model,+ api_key=None,+ source=source if source != "none" else "none",+ reason="no API key configured (neither tenant BYOK nor env)",+ )++ blocked, reason = _airgap_blocks(base_url)+ if blocked:+ return LlmConfig(+ allowed=False,+ base_url=final_base,+ model=final_model,+ api_key=api_key, # safe: caller does not log this on failure+ source=source,+ reason=reason,+ )++ return LlmConfig(+ allowed=True,+ base_url=final_base,+ model=final_model,+ api_key=api_key,+ source=source,+ reason="",+ )+++# ---------------------------------------------------------------------------+# Internal helpers+# ---------------------------------------------------------------------------+++def _decrypt_vault_token(vault_token: str, tenant_ref: str) -> str | None:+ """Decrypt a stored ``vault:v1:<base64>`` token. Returns ``None`` on failure.++ The vault may be unconfigured (``AISOC_CREDENTIAL_KEY`` not set) on+ operator boxes that haven't migrated to BYOK yet — that's a+ legitimate configuration, so we log once and fall back rather than+ treating it as an error.+ """+ vault = get_vault()+ if vault is None:+ logger.warning(+ "explain.llm_resolve_vault_disabled",+ tenant=tenant_ref,+ reason=(+ "AISOC_CREDENTIAL_KEY not set; tenant BYOK key cannot be "+ "decrypted, falling back to environment baseline"+ ),+ )+ return None+ try:+ return vault.decrypt(vault_token)+ except CredentialVaultError as exc:+ logger.warning(+ "explain.llm_resolve_decrypt_failed",+ tenant=tenant_ref,+ error=str(exc),+ )+ return None+++def _classify_source(+ *,+ env_base_url: str,+ env_model: str,+ env_key: str | None,+ tenant_contributed_base_url: bool,+ tenant_contributed_model: bool,+ tenant_contributed_key: bool,+) -> str:+ """Label the merge result. Mirrors ``tenant_llm_status`` in services/api.++ The three flags say which fields the *tenant row* contributed. The+ env values say which fields the environment *would have*+ contributed if the tenant didn't. If both contributed something to+ the final config, the merge is "mixed" — exactly what the API-side+ status endpoint reports.+ """+ tenant_contributed = (+ tenant_contributed_base_url+ or tenant_contributed_model+ or tenant_contributed_key+ )+ env_contributed = (+ (not tenant_contributed_base_url and bool(env_base_url))+ or (not tenant_contributed_model and bool(env_model))+ or (not tenant_contributed_key and bool(env_key))+ )++ if tenant_contributed and env_contributed:+ return "mixed"+ if tenant_contributed:+ return "tenant"+ if env_contributed:+ return "environment"+ return "none"diff --git a/services/agents/tests/test_llm_resolver.py b/services/agents/tests/test_llm_resolver.pynew file mode 100644index 000000000..d40174499--- /dev/null+++ b/services/agents/tests/test_llm_resolver.py@@ -0,0 +1,775 @@+"""Unit tests for ``services/agents/app/security/llm_resolver`` (WS-H2).++The resolver is the read path the explain endpoint uses to answer+"for *this* tenant, what LLM config (if any) should we use right+now?". The tests below cover every branch documented in the module+docstring:++* env-only baseline (no tenant ref, or tenant ref ``"default"``)+* tenant override fully replacing env (``source="tenant"``)+* partial tenant override layered over env (``source="mixed"``)+* corrupt / undecryptable tenant ciphertext degrading to env baseline+* vault not configured (``AISOC_CREDENTIAL_KEY`` unset) degrading to env baseline+* air-gap policy blocking ``api.openai.com`` even with a valid key+* air-gap policy *allowing* a private LiteLLM gateway+* ledger import / pool failure degrading to env baseline+* ``_classify_source`` matrix (tenant / environment / mixed / none)+* round-trip vault decrypt of a token produced by the API-side vault++The DB layer is mocked via a fake ``asyncpg.Pool`` so we never need a+running Postgres for these tests; the explain hot path's real DB+contract is covered separately by the API-side credential tests.+"""++from __future__ import annotations++import os+import sys+import uuid+from pathlib import Path+from types import ModuleType, SimpleNamespace+from typing import Any+from unittest.mock import AsyncMock, MagicMock, patch++import pytest+from cryptography.fernet import Fernet++# ----- import path setup ---------------------------------------------------++_AGENTS_ROOT = Path(__file__).resolve().parents[1]+if str(_AGENTS_ROOT) not in sys.path:+ sys.path.insert(0, str(_AGENTS_ROOT))+++# ----- helpers --------------------------------------------------------------+++def _import_resolver():+ """Import the resolver fresh so per-test env tweaks take effect.++ The resolver itself reads env vars on every call (``_env_baseline``,+ ``_airgap_blocks``), so re-import is *not* required for those — but+ the vendored vault caches a singleton, so we always reset that+ cache before each test (see the autouse fixture below).+ """+ from app.security import llm_resolver # noqa: PLC0415++ return llm_resolver+++def _reset_vault() -> None:+ from app.security import credential_vault # noqa: PLC0415++ credential_vault.reset_vault_for_tests()+++def _set_vault_key(monkeypatch: pytest.MonkeyPatch) -> bytes:+ """Configure ``AISOC_CREDENTIAL_KEY`` with a fresh Fernet key.++ Returns the key bytes so a test can also instantiate a vault+ directly and produce a ciphertext to seed into a mocked DB row.+ """+ key = Fernet.generate_key()+ monkeypatch.setenv("AISOC_CREDENTIAL_KEY", key.decode("ascii"))+ monkeypatch.delenv("AISOC_CREDENTIAL_KEY_ROTATION_FROM", raising=False)+ _reset_vault()+ return key+++def _clear_llm_env(monkeypatch: pytest.MonkeyPatch) -> None:+ """Strip every env var the resolver consumes so tests start clean."""+ for var in (+ "OPENAI_API_KEY",+ "OPENAI_BASE_URL",+ "OPENAI_MODEL",+ "LLM_API_KEY",+ "LLM_BASE_URL",+ "LLM_MODEL",+ "AISOC_LLM_MODEL",+ "AISOC_AIRGAPPED",+ ):+ monkeypatch.delenv(var, raising=False)+++# ----- autouse fixture ------------------------------------------------------++[email protected](autouse=True)+def _reset_state(monkeypatch: pytest.MonkeyPatch) -> None:+ """Reset vault singleton and strip LLM env vars before each test.++ Many of the helpers under test consult module-level / process-level+ state (``_vault_singleton``, ``os.environ``). Without this fixture+ one test's vault key would leak into the next test.+ """+ _clear_llm_env(monkeypatch)+ _reset_vault()+ yield+ _reset_vault()+++# ===========================================================================+# _env_baseline+# ===========================================================================+++class TestEnvBaseline:+ """Verify the resolver speaks the same env-var dialect as the API+ service's status endpoint (``services/api/.../llm_status.py``).+ """++ def test_empty_env_returns_blank_triple(self) -> None:+ resolver = _import_resolver()+ base, model, key = resolver._env_baseline()+ assert base == ""+ assert model == ""+ assert key is None++ def test_openai_style_env_is_picked_up(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env-1")+ monkeypatch.setenv("OPENAI_BASE_URL", "https://api.openai.com")+ monkeypatch.setenv("OPENAI_MODEL", "gpt-4o-mini")++ resolver = _import_resolver()+ base, model, key = resolver._env_baseline()+ assert base == "https://api.openai.com"+ assert model == "gpt-4o-mini"+ assert key == "sk-env-1"++ def test_llm_style_env_is_picked_up_when_openai_unset(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("LLM_API_KEY", "private-llm-key")+ monkeypatch.setenv("LLM_BASE_URL", "http://llm.internal:4000/v1")+ monkeypatch.setenv("LLM_MODEL", "llama3.1-8b")++ resolver = _import_resolver()+ base, model, key = resolver._env_baseline()+ assert base == "http://llm.internal:4000/v1"+ assert model == "llama3.1-8b"+ assert key == "private-llm-key"++ def test_openai_takes_precedence_over_llm_aliases(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("OPENAI_API_KEY", "sk-from-openai")+ monkeypatch.setenv("LLM_API_KEY", "should-be-ignored")+ monkeypatch.setenv("OPENAI_BASE_URL", "https://api.openai.com")+ monkeypatch.setenv("LLM_BASE_URL", "http://nope")+ monkeypatch.setenv("OPENAI_MODEL", "gpt-4o-mini")+ monkeypatch.setenv("LLM_MODEL", "should-be-ignored")++ resolver = _import_resolver()+ base, model, key = resolver._env_baseline()+ assert base == "https://api.openai.com"+ assert model == "gpt-4o-mini"+ assert key == "sk-from-openai"++ def test_aisoc_llm_model_legacy_env_supported(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("AISOC_LLM_MODEL", "claude-3.5-sonnet")+ resolver = _import_resolver()+ _, model, _ = resolver._env_baseline()+ assert model == "claude-3.5-sonnet"+++# ===========================================================================+# _airgap_blocks+# ===========================================================================+++class TestAirgapBlocks:+ """Air-gap policy must mirror the legacy ``_llm_allowed`` helper."""++ def test_airgap_off_never_blocks(self, monkeypatch: pytest.MonkeyPatch) -> None:+ # AISOC_AIRGAPPED unset → never block, regardless of host.+ resolver = _import_resolver()+ for url in ("", "https://api.openai.com", "http://litellm.internal"):+ blocked, _ = resolver._airgap_blocks(url)+ assert blocked is False, f"airgap=off blocked {url!r}"++ @pytest.mark.parametrize("flag", ["1", "true", "yes"])+ def test_airgap_blocks_empty_base_url(+ self, flag: str, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("AISOC_AIRGAPPED", flag)+ resolver = _import_resolver()+ blocked, reason = resolver._airgap_blocks("")+ assert blocked is True+ assert "would default to api.openai.com" in reason++ def test_airgap_blocks_openai_host(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("AISOC_AIRGAPPED", "true")+ resolver = _import_resolver()+ blocked, reason = resolver._airgap_blocks("https://api.openai.com/v1")+ assert blocked is True+ assert "api.openai.com" in reason++ def test_airgap_allows_private_litellm_host(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("AISOC_AIRGAPPED", "true")+ resolver = _import_resolver()+ blocked, _ = resolver._airgap_blocks("http://litellm.internal:4000/v1")+ assert blocked is False+++# ===========================================================================+# _classify_source+# ===========================================================================+++class TestClassifySource:+ """The source label must match what tenant_llm_status reports."""++ def test_none_when_neither_side_contributes(self) -> None:+ resolver = _import_resolver()+ result = resolver._classify_source(+ env_base_url="",+ env_model="",+ env_key=None,+ tenant_contributed_base_url=False,+ tenant_contributed_model=False,+ tenant_contributed_key=False,+ )+ assert result == "none"++ def test_environment_when_only_env_set(self) -> None:+ resolver = _import_resolver()+ result = resolver._classify_source(+ env_base_url="",+ env_model="",+ env_key="sk-env",+ tenant_contributed_base_url=False,+ tenant_contributed_model=False,+ tenant_contributed_key=False,+ )+ assert result == "environment"++ def test_tenant_when_tenant_replaces_everything(self) -> None:+ # Tenant supplied all three fields and env is blank.+ resolver = _import_resolver()+ result = resolver._classify_source(+ env_base_url="",+ env_model="",+ env_key=None,+ tenant_contributed_base_url=True,+ tenant_contributed_model=True,+ tenant_contributed_key=True,+ )+ assert result == "tenant"++ def test_mixed_when_tenant_supplies_key_but_env_supplies_model(self) -> None:+ # Common operator scenario: env defines model, tenant brings key.+ resolver = _import_resolver()+ result = resolver._classify_source(+ env_base_url="",+ env_model="gpt-4o-mini",+ env_key=None,+ tenant_contributed_base_url=False,+ tenant_contributed_model=False,+ tenant_contributed_key=True,+ )+ assert result == "mixed"+++# ===========================================================================+# _decrypt_vault_token+# ===========================================================================+++class TestDecryptVaultToken:+ """Decryption is best-effort and must never raise."""++ def test_returns_none_when_vault_disabled(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # No AISOC_CREDENTIAL_KEY set → vault.get_vault() returns None.+ monkeypatch.delenv("AISOC_CREDENTIAL_KEY", raising=False)+ _reset_vault()++ resolver = _import_resolver()+ result = resolver._decrypt_vault_token("vault:v1:something", "tenant-a")+ assert result is None++ def test_round_trip_with_configured_vault(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ _set_vault_key(monkeypatch)+ from app.security.credential_vault import get_vault # noqa: PLC0415++ vault = get_vault()+ assert vault is not None+ ciphertext = vault.encrypt("sk-tenant-secret")+ assert ciphertext.startswith("vault:v1:")++ resolver = _import_resolver()+ result = resolver._decrypt_vault_token(ciphertext, "tenant-a")+ assert result == "sk-tenant-secret"++ def test_returns_none_on_corrupt_ciphertext(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # Configure a vault with one key, then hand it a token encrypted+ # under a completely different key — must degrade gracefully.+ _set_vault_key(monkeypatch)++ wrong_key_vault_key = Fernet.generate_key()+ wrong_fernet = Fernet(wrong_key_vault_key)+ bogus_token = (+ "vault:v1:" + wrong_fernet.encrypt(b"sk-other").decode("ascii")+ )++ resolver = _import_resolver()+ result = resolver._decrypt_vault_token(bogus_token, "tenant-a")+ assert result is None+++# ===========================================================================+# resolve_llm_config — no DB lookup paths+# ===========================================================================+++class TestResolveNoTenant:+ """Sanity-check the early-exit branches that skip the DB entirely."""++ @pytest.mark.asyncio+ async def test_none_tenant_ref_uses_env_baseline_only(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env")+ monkeypatch.setenv("OPENAI_BASE_URL", "http://litellm.internal:4000")+ monkeypatch.setenv("OPENAI_MODEL", "llama3.1-8b")++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config(None)++ assert cfg.allowed is True+ assert cfg.api_key == "sk-env"+ assert cfg.base_url == "http://litellm.internal:4000"+ assert cfg.model == "llama3.1-8b"+ assert cfg.source == "environment"+ assert cfg.reason == ""++ @pytest.mark.asyncio+ async def test_default_tenant_ref_skips_db(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env")+ monkeypatch.setenv("OPENAI_BASE_URL", "http://litellm.internal:4000")++ # If we reached the DB lookup, importing app.investigator.ledger+ # would explode in this slim test env. The fact that we don't+ # patch it and the test still passes proves the early-exit.+ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("default")+ assert cfg.allowed is True+ assert cfg.source == "environment"++ @pytest.mark.asyncio+ async def test_no_key_anywhere_returns_disallowed_with_reason(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # No env key, no tenant ref → allowed=False with deterministic+ # fallback base_url / model so callers can still log.+ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config(None)++ assert cfg.allowed is False+ assert cfg.api_key is None+ assert cfg.base_url == "https://api.openai.com"+ assert cfg.model == "gpt-4o-mini"+ assert "no api key" in cfg.reason.lower()++ @pytest.mark.asyncio+ async def test_airgap_blocks_default_openai_endpoint(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env")+ monkeypatch.setenv("OPENAI_BASE_URL", "https://api.openai.com")+ monkeypatch.setenv("AISOC_AIRGAPPED", "true")++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config(None)+ assert cfg.allowed is False+ assert "api.openai.com" in cfg.reason++ @pytest.mark.asyncio+ async def test_airgap_allows_private_endpoint(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env")+ monkeypatch.setenv("OPENAI_BASE_URL", "http://litellm.internal:4000/v1")+ monkeypatch.setenv("AISOC_AIRGAPPED", "true")++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config(None)+ assert cfg.allowed is True+ assert cfg.source == "environment"+++# ===========================================================================+# resolve_llm_config — DB lookup paths+# ===========================================================================+++def _make_pool_with_row(row: dict[str, Any] | None) -> MagicMock:+ """Build a fake ``asyncpg.Pool`` whose connection returns ``row``.++ The resolver calls (in order):+ async with pool.acquire() as conn:+ tenant_id = await _resolve_tenant_uuid(conn, tenant_ref)+ await _set_rls_context(conn, tenant_id)+ row = await conn.fetchrow(...)++ Where ``_resolve_tenant_uuid`` itself either parses the ref as a+ UUID (no DB hop) or calls ``conn.fetchrow`` once to look up the+ slug. We arrange for the first ``fetchrow`` call to satisfy the+ UUID-from-slug lookup and the second to return the credential row.+ """+ conn = MagicMock()+ # First fetchrow → tenant id lookup (only used for non-UUID refs).+ # Second fetchrow → tenant_llm_credentials row.+ tenant_id_row = {"id": uuid.uuid4()}+ conn.fetchrow = AsyncMock(side_effect=[tenant_id_row, row])+ conn.execute = AsyncMock()++ acquire_cm = MagicMock()+ acquire_cm.__aenter__ = AsyncMock(return_value=conn)+ acquire_cm.__aexit__ = AsyncMock(return_value=None)++ pool = MagicMock()+ pool.acquire = MagicMock(return_value=acquire_cm)+ return pool+++def _stub_ledger(monkeypatch: pytest.MonkeyPatch, pool: MagicMock | None) -> None:+ """Inject a fake ``app.investigator.ledger`` into ``sys.modules``.++ The real package eagerly imports LangGraph in ``__init__``, so the+ test harness substitutes a hollow package + a stub ``ledger``+ submodule whose ``get_pool`` returns the supplied fake pool.+ """+ pkg_name = "app.investigator"+ ledger_name = "app.investigator.ledger"++ if pkg_name not in sys.modules:+ pkg = ModuleType(pkg_name)+ pkg.__path__ = [str(_AGENTS_ROOT / "app" / "investigator")]+ monkeypatch.setitem(sys.modules, pkg_name, pkg)++ ledger = ModuleType(ledger_name)+ if pool is None:+ async def _raise():+ raise RuntimeError("DATABASE_URL not set")++ ledger.get_pool = _raise # type: ignore[attr-defined]+ else:+ ledger.get_pool = AsyncMock(return_value=pool) # type: ignore[attr-defined]+ monkeypatch.setitem(sys.modules, ledger_name, ledger)+++class TestResolveWithTenant:+ @pytest.mark.asyncio+ async def test_tenant_row_with_full_override_yields_source_tenant(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ _set_vault_key(monkeypatch)+ from app.security.credential_vault import get_vault # noqa: PLC0415++ vault = get_vault()+ assert vault is not None+ ciphertext = vault.encrypt("sk-tenant")++ row = {+ "provider": "openai_compatible",+ "base_url": "http://tenant-llm.example/v1",+ "model": "tenant-model-7b",+ "api_key_vault": ciphertext,+ "settings": {},+ "enabled": True,+ }+ pool = _make_pool_with_row(row)+ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")++ assert cfg.allowed is True+ assert cfg.api_key == "sk-tenant"+ assert cfg.base_url == "http://tenant-llm.example/v1"+ assert cfg.model == "tenant-model-7b"+ assert cfg.source == "tenant"+ assert cfg.reason == ""++ @pytest.mark.asyncio+ async def test_tenant_supplies_only_key_layered_over_env_is_mixed(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # Operator scenario: env sets base_url + model, tenant BYOKs+ # only the key. Source should be "mixed".+ monkeypatch.setenv("OPENAI_BASE_URL", "http://litellm.internal:4000")+ monkeypatch.setenv("OPENAI_MODEL", "llama3.1-8b")+ # Note: no OPENAI_API_KEY.++ _set_vault_key(monkeypatch)+ from app.security.credential_vault import get_vault # noqa: PLC0415++ vault = get_vault()+ assert vault is not None+ ciphertext = vault.encrypt("sk-tenant-byok")++ row = {+ "provider": "openai_compatible",+ "base_url": None,+ "model": None,+ "api_key_vault": ciphertext,+ "settings": {},+ "enabled": True,+ }+ pool = _make_pool_with_row(row)+ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")++ assert cfg.allowed is True+ assert cfg.api_key == "sk-tenant-byok"+ # Tenant did not contribute base_url/model — env values must win.+ assert cfg.base_url == "http://litellm.internal:4000"+ assert cfg.model == "llama3.1-8b"+ assert cfg.source == "mixed"++ @pytest.mark.asyncio+ async def test_disabled_tenant_row_is_treated_as_no_override(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # ``enabled=false`` lets operators pause BYOK without deletion.+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env-fallback")+ monkeypatch.setenv("OPENAI_BASE_URL", "http://litellm.internal:4000")++ _set_vault_key(monkeypatch)+ from app.security.credential_vault import get_vault # noqa: PLC0415++ vault = get_vault()+ assert vault is not None+ row = {+ "provider": "openai_compatible",+ "base_url": "http://disabled.example",+ "model": "disabled-model",+ "api_key_vault": vault.encrypt("sk-disabled"),+ "settings": {},+ "enabled": False,+ }+ pool = _make_pool_with_row(row)+ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")++ # Tenant override ignored; env wins.+ assert cfg.allowed is True+ assert cfg.api_key == "sk-env-fallback"+ assert cfg.base_url == "http://litellm.internal:4000"+ assert cfg.source == "environment"++ @pytest.mark.asyncio+ async def test_corrupt_ciphertext_falls_back_to_env_key(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # If the tenant's stored ciphertext can't be decrypted with the+ # current vault key (mid-rotation, mistyped secret, etc.), the+ # resolver must fall back to the env key — *not* fail open and+ # *not* error out the explain request.+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env-fallback")+ _set_vault_key(monkeypatch)++ # Build a ciphertext under a *different* key, simulating drift.+ wrong_key = Fernet.generate_key()+ wrong_fernet = Fernet(wrong_key)+ bogus_token = (+ "vault:v1:" + wrong_fernet.encrypt(b"sk-tenant").decode("ascii")+ )++ row = {+ "provider": "openai_compatible",+ "base_url": "http://tenant-llm.example/v1",+ "model": "tenant-model",+ "api_key_vault": bogus_token,+ "settings": {},+ "enabled": True,+ }+ pool = _make_pool_with_row(row)+ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")++ # Tenant base_url + model still apply (they decrypt fine), but+ # the api_key fell back to the env value, so source is "mixed".+ assert cfg.allowed is True+ assert cfg.api_key == "sk-env-fallback"+ assert cfg.base_url == "http://tenant-llm.example/v1"+ assert cfg.model == "tenant-model"+ assert cfg.source == "mixed"++ @pytest.mark.asyncio+ async def test_vault_disabled_falls_back_to_env_key(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # AISOC_CREDENTIAL_KEY missing → vault.get_vault() returns None.+ # Resolver must still surface tenant base_url/model and use env key.+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env-fallback")+ monkeypatch.delenv("AISOC_CREDENTIAL_KEY", raising=False)+ _reset_vault()++ row = {+ "provider": "openai_compatible",+ "base_url": "http://tenant-llm.example/v1",+ "model": "tenant-model",+ # Some token — does not need to be valid because the vault+ # can't even attempt decryption without a key.+ "api_key_vault": "vault:v1:opaque",+ "settings": {},+ "enabled": True,+ }+ pool = _make_pool_with_row(row)+ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")++ assert cfg.allowed is True+ assert cfg.api_key == "sk-env-fallback"+ assert cfg.base_url == "http://tenant-llm.example/v1"+ assert cfg.source == "mixed"++ @pytest.mark.asyncio+ async def test_no_tenant_row_uses_env_baseline(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env")+ monkeypatch.setenv("OPENAI_BASE_URL", "http://litellm.internal:4000")++ pool = _make_pool_with_row(None)+ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")+ assert cfg.allowed is True+ assert cfg.api_key == "sk-env"+ assert cfg.source == "environment"++ @pytest.mark.asyncio+ async def test_ledger_unavailable_falls_back_to_env_baseline(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # Simulate the "agents box can't reach the platform DB right now"+ # path. Resolver must never raise on this; it just degrades.+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env")+ _stub_ledger(monkeypatch, pool=None)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")+ assert cfg.allowed is True+ assert cfg.api_key == "sk-env"+ assert cfg.source == "environment"++ @pytest.mark.asyncio+ async def test_db_query_failure_falls_back_to_env_baseline(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # ``conn.fetchrow`` blows up mid-query (e.g. transient outage,+ # connection cap exceeded). Must not propagate.+ monkeypatch.setenv("OPENAI_API_KEY", "sk-env")++ conn = MagicMock()+ conn.fetchrow = AsyncMock(side_effect=RuntimeError("connection lost"))+ conn.execute = AsyncMock()++ acquire_cm = MagicMock()+ acquire_cm.__aenter__ = AsyncMock(return_value=conn)+ acquire_cm.__aexit__ = AsyncMock(return_value=None)+ pool = MagicMock()+ pool.acquire = MagicMock(return_value=acquire_cm)++ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")+ assert cfg.allowed is True+ assert cfg.source == "environment"+++# ===========================================================================+# Integration: airgap interaction with tenant override+# ===========================================================================+++class TestAirgapWithTenantOverride:+ @pytest.mark.asyncio+ async def test_airgap_allows_tenant_private_gateway(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # The whole point of BYOK in air-gapped deployments: tenant+ # points at their internal LLM, AISOC_AIRGAPPED=true does *not*+ # block them.+ monkeypatch.setenv("AISOC_AIRGAPPED", "true")+ _set_vault_key(monkeypatch)+ from app.security.credential_vault import get_vault # noqa: PLC0415++ vault = get_vault()+ assert vault is not None+ ciphertext = vault.encrypt("sk-tenant")++ row = {+ "provider": "openai_compatible",+ "base_url": "http://tenant-llm.internal:4000/v1",+ "model": "internal-model",+ "api_key_vault": ciphertext,+ "settings": {},+ "enabled": True,+ }+ pool = _make_pool_with_row(row)+ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")+ assert cfg.allowed is True+ assert cfg.api_key == "sk-tenant"+ assert cfg.base_url == "http://tenant-llm.internal:4000/v1"++ @pytest.mark.asyncio+ async def test_airgap_blocks_tenant_byok_to_openai(+ self, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ # Even with a valid tenant BYOK row, if it points at OpenAI and+ # AISOC_AIRGAPPED=true, we must refuse the live call.+ monkeypatch.setenv("AISOC_AIRGAPPED", "true")+ _set_vault_key(monkeypatch)+ from app.security.credential_vault import get_vault # noqa: PLC0415++ vault = get_vault()+ assert vault is not None+ ciphertext = vault.encrypt("sk-tenant")++ row = {+ "provider": "openai",+ "base_url": "https://api.openai.com",+ "model": "gpt-4o-mini",+ "api_key_vault": ciphertext,+ "settings": {},+ "enabled": True,+ }+ pool = _make_pool_with_row(row)+ _stub_ledger(monkeypatch, pool)++ resolver = _import_resolver()+ cfg = await resolver.resolve_llm_config("tenant-a")+ assert cfg.allowed is False+ assert "api.openai.com" in cfg.reasondiff --git a/services/api/app/api/v1/endpoints/llm_credentials.py b/services/api/app/api/v1/endpoints/llm_credentials.pynew file mode 100644index 000000000..31a1b5c2b--- /dev/null+++ b/services/api/app/api/v1/endpoints/llm_credentials.py@@ -0,0 +1,430 @@+"""Per-tenant BYOK LLM credentials — WS-H2 (buyer-value plan).++Endpoint contract+-----------------++* ``GET /api/v1/llm/credentials``+ Read-only projection of the tenant's stored credential. Never returns+ the API key; only ``has_api_key: bool`` so the UI can render a+ "credential present, last rotated N days ago" badge.++* ``PUT /api/v1/llm/credentials``+ Upsert (provider + base_url + model + api_key + settings + enabled).+ The plaintext key is encrypted via :class:`CredentialVault` and only+ the ``vault:v1:<base64>`` ciphertext touches the database. Pass+ ``api_key=null`` to keep the existing ciphertext (rotation-only PUT+ that updates provider/base_url/model without re-typing the key).++* ``DELETE /api/v1/llm/credentials``+ Hard-delete the row. The platform falls back to env-var configuration+ on the next request (the existing v1 behaviour).++Why this lives in services/api+------------------------------++The agents service handles the actual LLM call but doesn't own+Postgres write paths — the explain endpoint reads the resolved+credential at request time via a vendored read-path vault (same+pattern as ``services/connectors/app/security/credential_vault.py``).+Keeping the write path here means we have one place that holds the+encrypt key authority, matches connector + OAuth credential management,+and audit rows for credential changes land in the same ``audit_log``+table as the rest of the platform.++Security rules+--------------++* ``settings:read`` to view, ``settings:write`` to mutate. Tenant admin+ only by default (see ``app.core.security.ROLE_PERMISSIONS``).+* The plaintext API key is never logged (we only log fact-of-rotation ++ provider + actor) and never round-tripped to the UI.+* Every PUT/DELETE emits an immutable ``audit_log`` row via+ :func:`emit_audit` so a buyer's compliance team has a paper trail of+ who put which provider where, and when keys rotated.+* The provider whitelist mirrors the DB ``CHECK`` constraint and+ :mod:`app.api.v1.endpoints.llm_status`. Adding a provider is a+ three-way change (migration + Pydantic enum + classifier).+"""++from __future__ import annotations++import logging+from datetime import UTC, datetime+from typing import Annotated, Literal+from urllib.parse import urlparse++from fastapi import APIRouter, Depends, HTTPException, Request, status+from pydantic import BaseModel, Field, field_validator+from sqlalchemy import delete, select++from app.api.v1.deps import AuthUser, DBSession, require_permission+from app.models.llm_credential import TenantLlmCredential+from app.security.credential_vault import CredentialVaultError, get_vault+from app.services.audit import emit_audit++logger = logging.getLogger(__name__)++router = APIRouter(prefix="/llm/credentials", tags=["llm"])+++# --------------------------------------------------------------- constants++# Mirror of the ``CHECK`` constraint in+# ``services/api/migrations/038_tenant_llm_credentials.sql`` and the+# classifier in :mod:`app.api.v1.endpoints.llm_status`. Keep in sync.+LlmProvider = Literal[+ "openai",+ "anthropic",+ "azure-openai",+ "local-ollama",+ "local-vllm",+ "local-litellm",+ "custom",+]++# Providers that *require* a non-NULL ``base_url``. Hosted SaaS providers+# default to a canonical endpoint and reject base-URL spoofing; local ++# custom providers must tell us where to point.+_PROVIDERS_REQUIRING_BASE_URL: frozenset[str] = frozenset(+ {"local-ollama", "local-vllm", "local-litellm", "custom"}+)++# Providers that require a non-NULL API key. Local providers without+# auth (Ollama default install, vLLM behind an internal LB) legitimately+# have no key. ``custom`` is permissive — operators run all sorts of+# proxies behind it (LiteLLM with org auth, internal API gateways, etc.).+_PROVIDERS_REQUIRING_KEY: frozenset[str] = frozenset({"openai", "anthropic", "azure-openai"})+++# --------------------------------------------------------------- request / response models+++class LlmCredentialUpsert(BaseModel):+ """Payload for ``PUT /api/v1/llm/credentials``.++ ``api_key`` is intentionally optional. Sending ``null`` (or omitting+ it) on an *existing* row keeps the stored ciphertext untouched — that+ lets operators flip ``enabled``, swap ``model``, or change the+ ``base_url`` without re-typing a 100-character secret. On a *new*+ row, ``api_key`` is required when the chosen provider mandates one+ (we surface a 422 in that case rather than silently storing NULL).+ """++ provider: LlmProvider+ base_url: str | None = Field(default=None, max_length=512)+ model: str | None = Field(default=None, max_length=120)+ api_key: str | None = Field(default=None, max_length=4096)+ settings: dict | None = Field(default=None)+ enabled: bool = Field(default=True)++ @field_validator("base_url")+ @classmethod+ def _validate_base_url(cls, v: str | None) -> str | None:+ if v is None:+ return v+ v = v.strip()+ if not v:+ return None+ # Light shape check — we don't want to over-validate (operators+ # legitimately run weird proxies on weird ports inside private+ # networks) but we should reject obvious garbage so the UI gets+ # a useful 422 instead of a 500 at request time.+ parsed = urlparse(v)+ if parsed.scheme not in {"http", "https"}:+ raise ValueError("base_url must be an http(s) URL")+ if not parsed.hostname:+ raise ValueError("base_url must include a hostname")+ return v++ @field_validator("api_key")+ @classmethod+ def _validate_api_key(cls, v: str | None) -> str | None:+ if v is None:+ return v+ v = v.strip()+ return v or None+++class LlmCredentialView(BaseModel):+ """Read-only projection of :class:`TenantLlmCredential`.++ The plaintext key is intentionally absent. ``has_api_key`` is the+ only signal we expose — same convention the OAuth credential view+ uses (``has_secret``).+ """++ provider: LlmProvider+ base_url: str | None+ model: str | None+ has_api_key: bool+ settings: dict+ enabled: bool+ created_at: datetime+ updated_at: datetime+ last_rotated_at: datetime | None+++# --------------------------------------------------------------- helpers+++def _project(row: TenantLlmCredential) -> LlmCredentialView:+ """Build the read-only view from an ORM row."""+ return LlmCredentialView(+ provider=row.provider, # type: ignore[arg-type]+ base_url=row.base_url,+ model=row.model,+ has_api_key=bool(row.api_key_vault),+ settings=dict(row.settings or {}),+ enabled=row.enabled,+ created_at=row.created_at,+ updated_at=row.updated_at,+ last_rotated_at=row.last_rotated_at,+ )+++def _enforce_provider_invariants(+ payload: LlmCredentialUpsert,+ *,+ existing: TenantLlmCredential | None,+) -> None:+ """Raise 422 if the payload is internally inconsistent.++ Cross-field rules can't be expressed in a single ``field_validator``+ so they live here. Mirrors the contract documented at the top of the+ module: hosted SaaS providers need a key, local + custom providers+ need a base_url. On rotation-only PUTs we check the *resolved* state+ (``payload`` ⊕ ``existing``), so a PUT with ``api_key=null`` on an+ existing row passes as long as the row already has a key.+ """+ if payload.provider in _PROVIDERS_REQUIRING_BASE_URL and not payload.base_url:+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail=f"provider '{payload.provider}' requires base_url",+ )++ requires_key = payload.provider in _PROVIDERS_REQUIRING_KEY+ if requires_key:+ # Either the payload supplies a fresh key, or the existing row+ # already has one we're keeping. Otherwise refuse.+ has_existing_key = bool(existing and existing.api_key_vault)+ if not payload.api_key and not has_existing_key:+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail=f"provider '{payload.provider}' requires api_key",+ )+++# --------------------------------------------------------------- endpoints+++ "",+ response_model=LlmCredentialView | None,+ summary="Read the tenant's BYOK LLM credential",+)+async def get_llm_credential(+ current_user: Annotated[AuthUser, Depends(require_permission("settings:read"))],+ db: DBSession,+) -> LlmCredentialView | None:+ """Return the per-tenant LLM credential, or ``null`` if none stored.++ Returning ``null`` (rather than 404) on the empty case lets the UI+ render a single "Configure your LLM" CTA without dispatching on+ error codes. The Settings panel renders a "no credential set —+ falling back to platform defaults" hint when this is null.+ """+ res = await db.execute(+ select(TenantLlmCredential).where(+ TenantLlmCredential.tenant_id == current_user.tenant_id,+ )+ )+ row = res.scalar_one_or_none()+ if row is None:+ return None+ return _project(row)+++ "",+ response_model=LlmCredentialView,+ summary="Upsert the tenant's BYOK LLM credential",+)+async def upsert_llm_credential(+ payload: LlmCredentialUpsert,+ request: Request,+ current_user: Annotated[AuthUser, Depends(require_permission("settings:write"))],+ db: DBSession,+) -> LlmCredentialView:+ """Create or update the tenant's BYOK credential.++ Behaviour notes:++ * ``api_key`` is required for hosted SaaS providers on first write.+ A subsequent PUT with ``api_key=null`` keeps the stored+ ciphertext (useful when toggling ``enabled`` or changing+ ``model`` without re-typing the secret).+ * Whenever ``api_key`` is provided we bump ``last_rotated_at`` so+ the UI can render "rotated 3 days ago".+ * The audit row's ``changes`` payload includes the *fact* of a+ rotation but never the key itself — we only record whether the+ key was changed, the previous + new providers, and whether the+ row is enabled.+ """+ res = await db.execute(+ select(TenantLlmCredential).where(+ TenantLlmCredential.tenant_id == current_user.tenant_id,+ )+ )+ existing: TenantLlmCredential | None = res.scalar_one_or_none()++ _enforce_provider_invariants(payload, existing=existing)++ # Encrypt the new key if one was supplied. We do this *before* the+ # upsert so a vault failure short-circuits without partial writes.+ new_ciphertext: str | None+ if payload.api_key is not None:+ try:+ new_ciphertext = get_vault().encrypt(payload.api_key)+ except CredentialVaultError as exc:+ raise HTTPException(+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,+ detail="Failed to encrypt LLM API key",+ ) from exc+ else:+ new_ciphertext = None # marker: keep existing++ now = datetime.now(UTC)+ rotated = False+ previous_provider: str | None = None+ if existing is None:+ row = TenantLlmCredential(+ tenant_id=current_user.tenant_id,+ provider=payload.provider,+ base_url=payload.base_url,+ model=payload.model,+ api_key_vault=new_ciphertext,+ settings=payload.settings or {},+ enabled=payload.enabled,+ created_at=now,+ updated_at=now,+ last_rotated_at=now if new_ciphertext else None,+ )+ db.add(row)+ rotated = bool(new_ciphertext)+ else:+ previous_provider = existing.provider+ existing.provider = payload.provider+ existing.base_url = payload.base_url+ existing.model = payload.model+ if new_ciphertext is not None:+ existing.api_key_vault = new_ciphertext+ existing.last_rotated_at = now+ rotated = True+ if payload.settings is not None:+ existing.settings = payload.settings+ existing.enabled = payload.enabled+ existing.updated_at = now+ row = existing++ await db.commit()+ await db.refresh(row)++ # Audit row. We deliberately keep this tight — provider transitions,+ # rotation flag, enabled state. No URL / model leakage either, since+ # the next call to GET /llm/credentials returns those fields if the+ # auditor needs them.+ try:+ await emit_audit(+ db=db,+ tenant_id=current_user.tenant_id,+ actor_id=current_user.user_id,+ actor_email=current_user.email,+ action="settings.llm.upsert",+ resource="llm_credential",+ resource_id=str(current_user.tenant_id),+ changes={+ "provider": payload.provider,+ "previous_provider": previous_provider,+ "rotated": rotated,+ "enabled": payload.enabled,+ "created": existing is None,+ },+ request=request,+ )+ except Exception as exc: # noqa: BLE001 — audit must not fail the call+ logger.error(+ "settings.llm.audit.failed tenant_id=%s error=%s",+ current_user.tenant_id,+ exc,+ )++ logger.info(+ "settings.llm.upsert",+ extra={+ "tenant": str(current_user.tenant_id),+ "provider": payload.provider,+ "rotated": rotated,+ "created": existing is None,+ },+ )+ return _project(row)+++ "",+ status_code=status.HTTP_204_NO_CONTENT,+ response_model=None,+ summary="Remove the tenant's BYOK LLM credential",+)+async def delete_llm_credential(+ request: Request,+ current_user: Annotated[AuthUser, Depends(require_permission("settings:write"))],+ db: DBSession,+) -> None:+ """Hard-delete the credential. Platform falls back to env-var config."""+ res = await db.execute(+ select(TenantLlmCredential).where(+ TenantLlmCredential.tenant_id == current_user.tenant_id,+ )+ )+ existing: TenantLlmCredential | None = res.scalar_one_or_none()++ await db.execute(+ delete(TenantLlmCredential).where(+ TenantLlmCredential.tenant_id == current_user.tenant_id,+ )+ )+ await db.commit()++ if existing is None:+ # Idempotent — DELETE on an empty row is a no-op, no audit needed.+ return++ try:+ await emit_audit(+ db=db,+ tenant_id=current_user.tenant_id,+ actor_id=current_user.user_id,+ actor_email=current_user.email,+ action="settings.llm.delete",+ resource="llm_credential",+ resource_id=str(current_user.tenant_id),+ changes={+ "provider": existing.provider,+ },+ request=request,+ )+ except Exception as exc: # noqa: BLE001+ logger.error(+ "settings.llm.audit.failed tenant_id=%s error=%s",+ current_user.tenant_id,+ exc,+ )++ logger.info(+ "settings.llm.delete",+ extra={+ "tenant": str(current_user.tenant_id),+ "provider": existing.provider,+ },+ )diff --git a/services/api/app/api/v1/endpoints/llm_status.py b/services/api/app/api/v1/endpoints/llm_status.pyindex 1fc909fac..fc91b69ca 100644--- a/services/api/app/api/v1/endpoints/llm_status.py+++ b/services/api/app/api/v1/endpoints/llm_status.py@@ -4,8 +4,8 @@ Exposes a redacted snapshot of the active LLM configuration so operatorsand auditors can verify the runtime LLM provider, model, and air-gapcompliance without shelling into a pod or grepping ``.env``. Theendpoint mirrors the shape of ``/api/v1/airgap/status`` and is paired-with the new "Deployment & AI" Settings panel in the web app-(WS-H2 visibility slice / WS-H4 air-gap operator UX).+with the "Deployment & AI" Settings panel in the web app+(WS-H2 BYOK / WS-H4 air-gap operator UX).Contract--------@@ -13,33 +13,58 @@ Contract* **Read-only.** No knobs to flip, no secrets returned. The API keyitself is never serialized — only ``key_set: bool``.* **Best-effort.** When the operator hasn't set ``OPENAI_API_KEY`` /- ``OPENAI_BASE_URL``, the endpoint reports ``provider="none"`` and- ``effective_path="fallback"`` so the UI can clearly say+ ``OPENAI_BASE_URL`` (and the tenant hasn't BYOK'd a row in+ ``tenant_llm_credentials``), the endpoint reports ``provider="none"``+ and ``effective_path="fallback"`` so the UI can clearly say"running on deterministic fallback".* **Air-gap aware.** Reuses ``app.core.airgap.is_host_allowed_for_airgap``so the answer to "would my LLM call actually leave the pod under thecurrent air-gap policy?" comes from the *same* code path that gatesegress at request time. No drift between the indicator and reality.+Two views: env-only vs tenant-resolved+--------------------------------------++* :func:`llm_status` returns the *environment-only* baseline — what the+ pod would do if no tenant context were involved. This is what the+ unauthenticated ``GET /api/v1/llm/status`` endpoint serves and what+ legacy callers like ``cost_dashboard`` consume.+* :func:`tenant_llm_status` layers per-tenant BYOK overrides from+ ``tenant_llm_credentials`` on top of the env baseline. Used by callers+ that already have an authenticated tenant context. The ``source``+ field on the returned snapshot says where each value came from+ (``"environment"`` / ``"tenant"`` / ``"mixed"``) so the UI can surface+ the layering honestly.++The agents service performs the equivalent resolution over its raw+``asyncpg`` connection (see ``services/agents/app/api/explain.py``)+using the vendored credential vault, so the indicator and the+runtime path stay in lockstep.+Why we don't reuse ``deployment.get_airgap_status``---------------------------------------------------``app.api.v1.endpoints.deployment.get_airgap_status`` is an in-memory-mock for a UI that hasn't shipped yet. This endpoint deliberately reads-the *real* env-var-driven config so the indicator the operator sees in-Settings matches the policy the gateway enforces. They will eventually-converge once per-tenant overrides exist (v1.1 BYOK).+mock for a legacy UI. This module deliberately reads the *real*+env-var-driven config (and ``tenant_llm_credentials``) so the+indicator the operator sees in Settings matches the policy the+gateway enforces."""from __future__ import annotationsimport os+import uuidfrom urllib.parse import urlparsefrom fastapi import APIRouter+from sqlalchemy import select+from sqlalchemy.ext.asyncio import AsyncSessionfrom app.core.airgap import is_host_allowed_for_airgapfrom app.core.config import settings+from app.models.llm_credential import TenantLlmCredential+from app.security.credential_vault import CredentialVaultError, get_vaultrouter = APIRouter(prefix="/llm", tags=["llm"])@@ -109,31 +134,13 @@ def _is_loopback_or_private_host(host: str) -> bool:return False-def llm_status() -> dict[str, object]:- """Return the live LLM provider snapshot.+def _env_baseline() -> tuple[str, str, bool]:+ """Resolve the ``(base_url, model, key_set)`` triple from env vars.- Shape::-- {- "provider": "local-ollama",- "model": "llama3.1:8b",- "base_url": "http://ollama:11434/v1",- "host": "ollama",- "key_set": false,- "airgap_enabled": true,- "airgap_compliant": true,- "is_local": true,- "effective_path": "live",- "policy_note": "...",- }-- Notes:- * ``base_url`` is returned verbatim (no path-stripping) because- operators sometimes encode the model behind the path on- single-tenant LiteLLM gateways and stripping it confuses- troubleshooting.- * ``key_set`` is the *only* signal we expose for the API key.- We never return the key itself, even partially redacted.+ Pulled out of :func:`llm_status` so both the env-only path and the+ tenant-resolved path start from exactly the same baseline. Returns+ a tuple rather than a dict so callers can immediately layer+ overrides on top by name."""base_url = os.getenv("LLM_BASE_URL") or os.getenv("OPENAI_BASE_URL") or ""model = (@@ -143,6 +150,24 @@ def llm_status() -> dict[str, object]:or "")key_set = bool(os.getenv("OPENAI_API_KEY") or os.getenv("LLM_API_KEY"))+ return base_url, model, key_set+++def _compute_status(+ *,+ base_url: str,+ model: str,+ key_set: bool,+ source: str,+) -> dict[str, object]:+ """Compute the redacted status snapshot for a resolved config triple.++ All classification (``provider``, ``airgap_compliant``, ``is_local``,+ ``effective_path``, ``policy_note``) happens here so env-only and+ tenant-resolved paths produce identical-shaped payloads. The+ ``source`` argument is threaded through to the response so callers+ can tell where each value came from without re-doing the merge.+ """airgap_enabled = bool(settings.AISOC_AIRGAPPED)# If neither base_url nor key are set, the operator is running on@@ -230,17 +255,164 @@ def llm_status() -> dict[str, object]:"is_local": is_local,"effective_path": effective_path,"policy_note": policy_note,+ "source": source,}+def llm_status() -> dict[str, object]:+ """Return the env-only LLM provider snapshot.++ Shape::++ {+ "provider": "local-ollama",+ "model": "llama3.1:8b",+ "base_url": "http://ollama:11434/v1",+ "host": "ollama",+ "key_set": false,+ "airgap_enabled": true,+ "airgap_compliant": true,+ "is_local": true,+ "effective_path": "live",+ "policy_note": "...",+ "source": "environment",+ }++ Notes:+ * ``base_url`` is returned verbatim (no path-stripping) because+ operators sometimes encode the model behind the path on+ single-tenant LiteLLM gateways and stripping it confuses+ troubleshooting.+ * ``key_set`` is the *only* signal we expose for the API key.+ We never return the key itself, even partially redacted.+ * For tenant-resolved snapshots that layer+ ``tenant_llm_credentials`` overrides on top of the env+ baseline, see :func:`tenant_llm_status`.+ """+ base_url, model, key_set = _env_baseline()+ return _compute_status(+ base_url=base_url,+ model=model,+ key_set=key_set,+ source="environment",+ )+++async def tenant_llm_status(+ db: AsyncSession,+ tenant_id: uuid.UUID,+) -> dict[str, object]:+ """Return the LLM snapshot with per-tenant BYOK overrides applied.++ Layers values from ``tenant_llm_credentials`` on top of the env+ baseline. Resolution rules per field:++ * ``base_url`` → tenant value if set, else env baseline.+ * ``model`` → tenant value if set, else env baseline.+ * ``key_set`` → ``True`` when the tenant row carries a vault token+ *or* the env baseline has one. The vault token is decrypted only+ to validate it, never returned.++ The returned ``source`` field reports the layering result:++ * ``"environment"`` — no tenant row (or row disabled) and at least+ one env-level value resolved.+ * ``"tenant"`` — every populated field came from the tenant row.+ * ``"mixed"`` — the tenant row contributed at least one value+ but at least one other value still came from the env baseline.++ A disabled tenant row (``enabled = false``) is treated as if it did+ not exist, matching the behaviour of the agents service.+ """+ env_base_url, env_model, env_key_set = _env_baseline()++ row = await db.execute(+ select(TenantLlmCredential).where(+ TenantLlmCredential.tenant_id == tenant_id+ )+ )+ cred = row.scalar_one_or_none()++ if cred is None or not cred.enabled:+ return _compute_status(+ base_url=env_base_url,+ model=env_model,+ key_set=env_key_set,+ source="environment",+ )++ # Layer tenant overrides on top of env baseline. Track which fields+ # the tenant actually contributed so the ``source`` field can+ # reflect the merge honestly (full tenant vs partial overlay).+ tenant_contributed = False+ env_contributed = False++ if cred.base_url:+ base_url = cred.base_url+ tenant_contributed = True+ else:+ base_url = env_base_url+ if env_base_url:+ env_contributed = True++ if cred.model:+ model = cred.model+ tenant_contributed = True+ else:+ model = env_model+ if env_model:+ env_contributed = True++ tenant_key_set = False+ if cred.api_key_vault:+ # Validate the vault token decrypts cleanly so we don't claim+ # ``key_set=True`` for a row whose ciphertext is corrupt or+ # written under a key we no longer hold. Plaintext is dropped+ # immediately — never returned, never logged.+ try:+ get_vault().decrypt(cred.api_key_vault)+ tenant_key_set = True+ tenant_contributed = True+ except CredentialVaultError:+ tenant_key_set = False++ if tenant_key_set:+ key_set = True+ else:+ key_set = env_key_set+ if env_key_set:+ env_contributed = True++ if tenant_contributed and env_contributed:+ source = "mixed"+ elif tenant_contributed:+ source = "tenant"+ else:+ source = "environment"++ return _compute_status(+ base_url=base_url,+ model=model,+ key_set=key_set,+ source=source,+ )++@router.get("/status", summary="Current LLM provider configuration")async def get_llm_status() -> dict[str, object]:- """Return the live LLM provider snapshot for this pod.+ """Return the env-only LLM provider snapshot for this pod.++ Intentionally **unauthenticated** and **env-only**, mirroring+ ``GET /api/v1/airgap/status``: the response carries no secrets+ (only ``key_set: bool``) and is safe for operator dashboards,+ auditors, and k8s liveness probes.++ Tenant-aware callers (the web UI's "Deployment & AI" Settings+ panel, the agents service's explain path) layer per-tenant BYOK+ overrides on top of this baseline by either:- The response is safe to surface in operator UIs: secrets are never- included (we only return ``key_set: bool``) and the body is- deterministic for a given environment so the same payload renders- identically across pods. Pair with ``/api/v1/airgap/status`` to- show operators a full picture of "where will my AI calls go".+ * calling :func:`tenant_llm_status` with their own DB session, or+ * pairing this response with ``GET /api/v1/llm/credentials`` to+ perform the merge client-side."""return llm_status()diff --git a/services/api/app/api/v1/router.py b/services/api/app/api/v1/router.pyindex fec781d15..cf695ec82 100644--- a/services/api/app/api/v1/router.py+++ b/services/api/app/api/v1/router.py@@ -32,6 +32,7 @@ from app.api.v1.endpoints import (inbox_itsm,knowledge_base,lake,+ llm_credentials,llm_status,nl_detection,nl_query,@@ -162,6 +163,14 @@ api_router.include_router(airgap.router)# in the UI cannot drift from runtime behaviour (WS-H2/H4 visibility).api_router.include_router(llm_status.router)+# BYOK per-tenant LLM credentials — WS-H2 (buyer-value plan).+# Settings-write CRUD over tenant_llm_credentials. The plaintext API key+# is encrypted via the credential vault before it touches Postgres; the+# read path returns has_api_key/last_rotated_at but never the key itself.+# Resolution order at request time is per-tenant row > env-var fallback+# (see llm_status.py + agents/explain.py).+api_router.include_router(llm_credentials.router)+# STIX/TAXII threat intelligence publishing (Tier 4)api_router.include_router(stix_taxii.router)diff --git a/services/api/app/models/__init__.py b/services/api/app/models/__init__.pyindex 1e03d8849..bc6750b18 100644--- a/services/api/app/models/__init__.py+++ b/services/api/app/models/__init__.py@@ -16,6 +16,7 @@ from app.models.investigation import (InvestigationEvent,InvestigationRun,)+from app.models.llm_credential import TenantLlmCredentialfrom app.models.oauth import OAuthAppCredential, OAuthStatefrom app.models.mssp import (MSSPDelegation,@@ -75,6 +76,7 @@ __all__ = ["OAuthAppCredential","OAuthState","TenantInboxToken",+ "TenantLlmCredential","PasskeyCredential","PasskeyChallenge","OnCallStatus",diff --git a/services/api/app/models/llm_credential.py b/services/api/app/models/llm_credential.pynew file mode 100644index 000000000..2434e9ba9--- /dev/null+++ b/services/api/app/models/llm_credential.py@@ -0,0 +1,108 @@+"""Per-tenant BYOK LLM credential ORM model (WS-H2).++Backs the ``/api/v1/llm/credentials`` endpoints and is consumed at request+time by ``services/agents/app/api/explain.py`` to resolve which API key,+model, and base URL to use when generating an alert explanation.++Why one row per tenant+----------------------+Every buyer we have spoken to wants exactly one active LLM credential.+"Multiple stored credentials with one marked active" is a v1.1 polish; if+we need it we can drop the PK and add an ``is_active`` partial unique+index without a destructive migration.++Encryption+----------+``api_key_vault`` is opaque ciphertext (``vault:v1:<base64>``) produced by+:class:`app.security.credential_vault.CredentialVault`. The plaintext key+never round-trips to the UI — the read endpoint exposes ``has_api_key:+bool`` only and the audit log records the *fact* of rotation, not the+key material.++Provider whitelist+------------------+The ``provider`` enum is enforced at the DB layer (CHECK constraint in+``038_tenant_llm_credentials.sql``) and mirrored in the application's+classification logic at+:func:`app.api.v1.endpoints.llm_status._classify_provider`. Any change to+the enum is a coordinated migration + code edit.+"""++from __future__ import annotations++import uuid+from datetime import UTC, datetime++from sqlalchemy import Boolean, DateTime, ForeignKey, String, Text+from sqlalchemy.dialects.postgresql import JSONB, UUID+from sqlalchemy.orm import Mapped, mapped_column++from app.db.database import Base+++class TenantLlmCredential(Base):+ """Per-tenant LLM provider configuration for BYOK + air-gap routing.++ The row resolves three questions for the request path:++ * Which provider to call (``provider``).+ * Where to send the request (``base_url``; ``NULL`` falls back to the+ provider's canonical endpoint, encoded in+ :data:`app.api.v1.endpoints.llm_status._PROVIDER_DEFAULTS`).+ * Which API key to authenticate with (``api_key_vault``; ``NULL`` is+ legitimate for ``local-*`` providers without auth).++ ``model`` and ``settings`` are optional overrides; the application+ layer is expected to fall back to platform defaults when they are+ ``NULL``.+ """++ __tablename__ = "tenant_llm_credentials"++ tenant_id: Mapped[uuid.UUID] = mapped_column(+ UUID(as_uuid=True),+ ForeignKey("tenants.id", ondelete="CASCADE"),+ primary_key=True,+ )+ # One of: openai | anthropic | azure-openai | local-ollama |+ # local-vllm | local-litellm | custom. The DB enforces this via a+ # CHECK constraint (see 038_tenant_llm_credentials.sql); the+ # application layer mirrors the list in llm_status._classify_provider.+ provider: Mapped[str] = mapped_column(String(40), nullable=False)+ # Optional override of the provider's default base URL. Required for+ # ``custom`` and the three ``local-*`` providers (the API endpoint+ # validates this on PUT). NULL for hosted providers means "use the+ # canonical endpoint".+ base_url: Mapped[str | None] = mapped_column(Text, nullable=True)+ # Optional model override. NULL means "use the platform default+ # (gpt-4o-mini for openai, claude-3-5-sonnet for anthropic, etc.)".+ model: Mapped[str | None] = mapped_column(String(120), nullable=True)+ # Vault-encrypted API key. Format: vault:v1:<base64>. Nullable because+ # local-* providers without auth legitimately have no key.+ api_key_vault: Mapped[str | None] = mapped_column(Text, nullable=True)+ # Provider-specific settings escape hatch (e.g. Azure deployment name,+ # OpenAI organization id, max_tokens override). Empty in v1.0; v1.1+ # features can populate this without a migration.+ settings: Mapped[dict] = mapped_column(+ JSONB, default=dict, nullable=False+ )+ # Soft-disable without deleting. Lets an operator pause LLM for a+ # tenant during a key rotation without losing the saved configuration.+ enabled: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)+ created_at: Mapped[datetime] = mapped_column(+ DateTime(timezone=True),+ default=lambda: datetime.now(UTC),+ nullable=False,+ )+ updated_at: Mapped[datetime] = mapped_column(+ DateTime(timezone=True),+ default=lambda: datetime.now(UTC),+ onupdate=lambda: datetime.now(UTC),+ nullable=False,+ )+ # Bumped only when api_key_vault changes (the API endpoint manages+ # this). Drives the "rotated 3 days ago" hint in the Settings UI so+ # operators have a key-hygiene signal at a glance.+ last_rotated_at: Mapped[datetime | None] = mapped_column(+ DateTime(timezone=True), nullable=True+ )diff --git a/services/api/migrations/038_tenant_llm_credentials.sql b/services/api/migrations/038_tenant_llm_credentials.sqlnew file mode 100644index 000000000..8aa404052--- /dev/null+++ b/services/api/migrations/038_tenant_llm_credentials.sql@@ -0,0 +1,123 @@+-- Migration 038: Per-tenant BYOK LLM credentials (WS-H2).+--+-- Why this table exists+-- ---------------------+-- Up to v1.0 the platform resolved LLM configuration from process-level+-- environment variables (``OPENAI_API_KEY`` / ``OPENAI_BASE_URL`` /+-- ``OPENAI_MODEL``). That is fine for a single-tenant dev box but it+-- doesn't survive contact with real buyers, who want one of three+-- things:+--+-- 1. "We have an enterprise OpenAI / Anthropic agreement — please+-- bill our API key, not the platform's." (Bring-Your-Own-Key.)+-- 2. "Route this tenant's traffic through our local Ollama / vLLM /+-- LiteLLM proxy on a private host." (Air-gapped overlay.)+-- 3. "Please don't call any LLM at all for this tenant; turn off the+-- Explain feature." (Disable BYOK + leave the env-var fallback+-- empty → ``_llm_allowed()`` returns false.)+--+-- All three collapse to: store one *active* LLM credential per tenant,+-- vault-encrypt the API key, and have the request path resolve+-- (tenant → row) before falling back to the process env vars.+--+-- Schema decisions+-- ----------------+-- * One row per tenant (``tenant_id`` is the PRIMARY KEY). We could+-- have keyed ``(tenant_id, provider)`` to allow multiple stored+-- credentials per tenant with one marked active, but every tenant+-- we have talked to wants exactly one. We can split rows in v1.1+-- without a destructive migration (drop PK, add a ``is_active``+-- partial unique index).+-- * ``provider`` is a ``VARCHAR`` with a CHECK constraint matching the+-- classification logic in ``services/api/app/api/v1/endpoints/llm_status.py``.+-- This keeps the storage layer honest about what the application+-- layer can actually resolve. Adding a new provider is a one-line+-- migration.+-- * ``api_key_vault`` is opaque ciphertext (``vault:v1:<base64>``)+-- produced by ``services/api/app/security/credential_vault.py``.+-- It is *never* round-tripped to the UI in plaintext; the read+-- endpoint exposes ``has_api_key: bool`` only.+-- * ``base_url`` and ``model`` are nullable overrides. NULL means+-- "use the provider's default" (e.g. ``https://api.openai.com/v1``+-- for OpenAI, ``http://ollama:11434/v1`` for local-ollama). We+-- intentionally do not enforce URL shape here — operators running+-- funky reverse proxies inside an air-gapped network need the+-- flexibility. The application layer validates on PUT.+-- * ``settings`` is a JSONB escape hatch for provider-specific knobs+-- (max_tokens override, organization id, deployment name for+-- Azure OpenAI). v1.0 ships with the column empty; v1.1 features+-- can populate it without a migration.+--+-- Audit + rotation+-- ----------------+-- Every PUT/DELETE on this table emits an audit row via+-- :func:`services.api.app.services.audit.emit_audit` with action+-- ``settings.llm.upsert`` / ``settings.llm.delete``. Rotation is "PUT+-- a new key, the old ciphertext is overwritten." We do not maintain+-- a history table — the audit log is the durable record of who+-- rotated and when.+--+-- RLS+-- ---+-- Row-level security mirrors the pattern in 037_saved_views.sql:+-- a tenant can only see its own row. The ``current_tenant_id()``+-- helper is set in 002_rls.sql.++BEGIN;++CREATE TABLE IF NOT EXISTS tenant_llm_credentials (+ tenant_id UUID PRIMARY KEY REFERENCES tenants(id) ON DELETE CASCADE,+ -- One of: openai | anthropic | azure-openai | local-ollama |+ -- local-vllm | local-litellm | custom. Mirrors the enum in+ -- services/api/app/api/v1/endpoints/llm_status.py::_classify_provider.+ provider VARCHAR(40) NOT NULL,+ -- Optional override of the provider's default API base URL.+ -- Required for ``custom`` and the three ``local-*`` providers+ -- (the application layer enforces this on PUT). NULL for the+ -- hosted SaaS providers means "use the canonical endpoint".+ base_url TEXT,+ -- Optional model override. NULL means "use the platform default+ -- (gpt-4o-mini for openai, claude-3-5-sonnet for anthropic, etc.)".+ model VARCHAR(120),+ -- Vault-encrypted API key. Format: ``vault:v1:<base64>``.+ -- Nullable because local-* providers (Ollama / vLLM with no auth)+ -- legitimately have no key. ``custom`` may or may not require one+ -- depending on the proxy.+ api_key_vault TEXT,+ -- Provider-specific settings escape hatch. Empty in v1.0.+ settings JSONB NOT NULL DEFAULT '{}'::jsonb,+ -- Soft-disable without deleting. Lets an operator say "pause LLM+ -- for this tenant while we rotate" without losing the saved+ -- configuration.+ enabled BOOLEAN NOT NULL DEFAULT TRUE,+ -- Audit metadata. ``last_rotated_at`` is bumped whenever+ -- ``api_key_vault`` changes (the API endpoint does this); the UI+ -- renders "rotated 3 days ago" so operators have a key-hygiene+ -- signal at a glance.+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),+ last_rotated_at TIMESTAMPTZ,+ CONSTRAINT tenant_llm_credentials_provider_check CHECK (+ provider IN (+ 'openai',+ 'anthropic',+ 'azure-openai',+ 'local-ollama',+ 'local-vllm',+ 'local-litellm',+ 'custom'+ )+ )+);++-- ============================================================+-- RLS: tenant isolation. Mirrors saved_views.+-- ============================================================+ALTER TABLE tenant_llm_credentials ENABLE ROW LEVEL SECURITY;+ALTER TABLE tenant_llm_credentials FORCE ROW LEVEL SECURITY;++DROP POLICY IF EXISTS tenant_llm_credentials_isolation ON tenant_llm_credentials;+CREATE POLICY tenant_llm_credentials_isolation ON tenant_llm_credentials+ USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL);++COMMIT;diff --git a/services/api/tests/test_llm_credentials_endpoint.py b/services/api/tests/test_llm_credentials_endpoint.pynew file mode 100644index 000000000..1cbc0d94c--- /dev/null+++ b/services/api/tests/test_llm_credentials_endpoint.py@@ -0,0 +1,907 @@+"""Unit tests for the per-tenant BYOK LLM credentials endpoint (WS-H2).++The ``/api/v1/llm/credentials`` GET/PUT/DELETE handlers wire together+four collaborators:++* the SQLAlchemy session (read existing row, upsert new one, hard-delete+ on remove),+* :class:`app.security.credential_vault.CredentialVault` (encrypt the+ plaintext API key into ``vault:v1:<base64>`` ciphertext),+* the audit log (an immutable row appended on every write so a buyer's+ compliance team has a paper trail of who rotated which provider when),+* and the cross-field validator that enforces "hosted SaaS providers+ require an api_key, local/custom providers require a base_url".++Spinning up the full FastAPI app + Postgres + Fernet wiring for these+units would be slow and would double-cover the lower layers. We follow+the same pattern as the sibling endpoint tests in this repo+(``test_saved_views_endpoint.py``, ``test_lake_endpoint.py``): mock the+DB session, build a ``SimpleNamespace`` user, and assert the contract+the endpoint should keep.++What we exercise here:++* ``GET`` returns ``None`` cleanly when no row exists, returns a+ projection (with ``has_api_key`` but never the key itself) when one+ does.+* ``PUT`` happy path: create + encrypt + audit on first write, rotation-+ only PUT (api_key=None) leaves the existing ciphertext alone, fresh+ api_key bumps ``last_rotated_at``.+* ``PUT`` validation: missing ``base_url`` for local providers,+ missing ``api_key`` for hosted SaaS, base_url with non-http scheme,+ base_url without hostname.+* ``PUT`` failure: a vault encrypt error must surface as a 500 *before*+ any DB mutation lands (so the caller does not end up with a half-+ written row).+* ``DELETE`` is idempotent (204 with no audit when nothing to delete)+ and emits an audit row when there was a row to delete.+* The vault round-trip is sound — a string encrypted by the vault+ decrypts back to the same string, and the projection surfaces+ ``has_api_key=True`` when the ciphertext column is non-empty.+"""++from __future__ import annotations++import uuid+from datetime import UTC, datetime+from types import SimpleNamespace+from typing import Any+from unittest.mock import AsyncMock, MagicMock, patch++import pytest+from app.api.v1.endpoints.llm_credentials import (+ LlmCredentialUpsert,+ _enforce_provider_invariants,+ _project,+ delete_llm_credential,+ get_llm_credential,+ upsert_llm_credential,+)+from app.security.credential_vault import (+ _CIPHER_PREFIX,+ CredentialVault,+ CredentialVaultError,+)+from cryptography.fernet import Fernet+from fastapi import HTTPException+from pydantic import ValidationError+++# ---------------------------------------------------------------------------+# Test fixtures / helpers+# ---------------------------------------------------------------------------+++def _build_user(+ *,+ tenant_id: uuid.UUID | None = None,+ user_id: uuid.UUID | None = None,+ role: str = "tenant_admin",+) -> SimpleNamespace:+ """Minimal CurrentUser stand-in.++ The endpoint reads ``tenant_id``, ``user_id``, ``role``, and ``email``+ only — a SimpleNamespace keeps the tests free of the auth import+ graph. Note that the dependency-injected ``require_permission``+ check is bypassed when we call the endpoint function directly; the+ permission contract is owned by the FastAPI dep wiring (asserted+ indirectly via the integration smoke tests in CI).+ """+ return SimpleNamespace(+ tenant_id=tenant_id or uuid.uuid4(),+ user_id=user_id or uuid.uuid4(),+ role=role,+ email="[email protected]",+ )+++def _build_request(+ *,+ headers: dict[str, str] | None = None,+ client_host: str = "127.0.0.1",+) -> Any:+ """Build a minimal FastAPI Request stand-in for the audit emission.++ The audit helper only reads ``request.headers`` and+ ``request.client``; no need for a full ASGI scope. Same shape as+ :func:`tests.test_lake_endpoint._build_request`.+ """+ req = MagicMock()+ req.headers = headers or {}+ req.client = MagicMock()+ req.client.host = client_host+ return req+++def _llm_row(+ *,+ tenant_id: uuid.UUID | None = None,+ provider: str = "openai",+ base_url: str | None = None,+ model: str | None = None,+ api_key_vault: str | None = "vault:v1:fakeciphertext",+ settings: dict[str, Any] | None = None,+ enabled: bool = True,+ created_at: datetime | None = None,+ updated_at: datetime | None = None,+ last_rotated_at: datetime | None = None,+) -> SimpleNamespace:+ """Build a TenantLlmCredential row stand-in.++ SimpleNamespace rather than the real ORM class so tests don't+ accidentally touch SQLAlchemy's instrumentation — assigning to+ ``row.provider`` on a real model would dirty the session, which is+ test-irrelevant noise here.+ """+ now = datetime.now(UTC)+ return SimpleNamespace(+ tenant_id=tenant_id or uuid.uuid4(),+ provider=provider,+ base_url=base_url,+ model=model,+ api_key_vault=api_key_vault,+ settings=settings if settings is not None else {},+ enabled=enabled,+ created_at=created_at or now,+ updated_at=updated_at or now,+ last_rotated_at=last_rotated_at,+ )+++def _mock_db_with_existing(+ existing: Any | None,+ *,+ extra_executes: int = 0,+) -> MagicMock:+ """Build a DB whose first ``execute`` returns ``existing`` via ``scalar_one_or_none``.++ ``extra_executes`` lets callers (e.g. DELETE, which runs a second+ ``execute`` for the actual DELETE statement) reserve more results+ in the side_effect chain. Subsequent results return a generic+ "no row" mock so any later ``scalar_one_or_none`` call is benign.+ """+ select_result = MagicMock()+ select_result.scalar_one_or_none = MagicMock(return_value=existing)++ extras = []+ for _ in range(extra_executes):+ result = MagicMock()+ result.scalar_one_or_none = MagicMock(return_value=None)+ extras.append(result)++ db = MagicMock()+ db.execute = AsyncMock(side_effect=[select_result, *extras])+ db.add = MagicMock()+ db.commit = AsyncMock()+ db.refresh = AsyncMock()+ db.rollback = AsyncMock()+ return db+++# ---------------------------------------------------------------------------+# _project — the read-only ORM → wire-model projection+# ---------------------------------------------------------------------------+++def test_project_strips_ciphertext_and_surfaces_has_api_key() -> None:+ """The plaintext key never leaves the server — only ``has_api_key``.++ This is the core privacy guarantee of the GET endpoint. Even an+ operator with DB read access who pokes the API gets back a boolean,+ not a key — they have to re-rotate to "see" a new one. This test+ pins that contract so a future refactor that accidentally adds+ ``api_key`` to the response model fails CI loudly.+ """+ row = _llm_row(+ provider="openai",+ base_url="https://api.openai.com",+ model="gpt-4o-mini",+ api_key_vault="vault:v1:abcdef",+ settings={"organization": "org-123"},+ )+ view = _project(row)++ assert view.provider == "openai"+ assert view.base_url == "https://api.openai.com"+ assert view.model == "gpt-4o-mini"+ assert view.has_api_key is True+ assert view.settings == {"organization": "org-123"}+ # Confirm there is no ``api_key`` attribute leaking through; the+ # response model schema doesn't include one, and runtime introspection+ # should agree.+ assert not hasattr(view, "api_key")+ # Pydantic dump must also be free of any plaintext-key flavoured key.+ dumped = view.model_dump()+ assert "api_key" not in dumped+ assert dumped["has_api_key"] is True+++def test_project_marks_has_api_key_false_when_ciphertext_empty() -> None:+ """A NULL ciphertext column → ``has_api_key=False``.++ Important so the UI can render the "Configure your LLM" CTA for a+ tenant that has stored a base_url + provider for a local-* deployment+ that does not need a key.+ """+ row = _llm_row(api_key_vault=None, provider="local-ollama")+ view = _project(row)+ assert view.has_api_key is False+++def test_project_normalises_settings_none_to_empty_dict() -> None:+ """A ``None`` settings column round-trips as ``{}``.++ The DB column is NOT NULL with default ``{}`` in the migration, but+ a fresh ORM instance pre-flush can be ``None``. Defensive normalisation+ here means the response model never blows up on a transient race.+ """+ row = _llm_row(settings=None)+ view = _project(row)+ assert view.settings == {}+++# ---------------------------------------------------------------------------+# LlmCredentialUpsert — Pydantic field validators+# ---------------------------------------------------------------------------+++ "value,expected",+ [+ ("https://api.openai.com", "https://api.openai.com"),+ ("http://localhost:11434", "http://localhost:11434"),+ # Whitespace gets trimmed; empty strings collapse to None.+ (" https://example.com ", "https://example.com"),+ (" ", None),+ ("", None),+ (None, None),+ ],+)+def test_base_url_validator_accepts_valid_inputs(+ value: str | None, expected: str | None+) -> None:+ """The validator strips whitespace and collapses empties to ``None``."""+ payload = LlmCredentialUpsert(provider="custom", base_url=value, api_key="k")+ assert payload.base_url == expected++[email protected]("bad", ["ftp://x.example", "not-a-url", "ws://x.example"])+def test_base_url_validator_rejects_non_http_schemes(bad: str) -> None:+ """Non-http(s) schemes 422 — the explain path uses an http client.++ A wss:// URL would silently fail at request time; the 422 here gives+ the operator a precise error in the UI before the row is written.+ """+ with pytest.raises(ValidationError) as exc:+ LlmCredentialUpsert(provider="custom", base_url=bad, api_key="k")+ assert "base_url must be an http(s) URL" in str(exc.value)+++def test_base_url_validator_rejects_missing_hostname() -> None:+ """A scheme-only URL has no host to point the LLM client at."""+ with pytest.raises(ValidationError) as exc:+ LlmCredentialUpsert(provider="custom", base_url="https://", api_key="k")+ assert "base_url must include a hostname" in str(exc.value)+++def test_api_key_validator_strips_and_collapses_whitespace_only() -> None:+ """Whitespace-only api_key → ``None`` (treated as "keep existing").++ Operators sometimes paste a key with a trailing newline, or paste+ nothing into the input. Stripping prevents the latter from looking+ like a key in the audit row.+ """+ payload = LlmCredentialUpsert(provider="openai", api_key=" ")+ assert payload.api_key is None+ payload = LlmCredentialUpsert(provider="openai", api_key=" abc ")+ assert payload.api_key == "abc"+++def test_provider_enum_rejects_unknown_value() -> None:+ """Unknown provider → 422 *before* it reaches the DB CHECK constraint.++ The enum lives in three places (DB CHECK, Pydantic Literal, and the+ classifier in ``llm_status``). Catching here keeps the failure mode+ "422 with a useful detail" rather than "500 from a constraint+ violation pretending to be a server error".+ """+ with pytest.raises(ValidationError):+ LlmCredentialUpsert(provider="bedrock", api_key="k") # type: ignore[arg-type]+++# ---------------------------------------------------------------------------+# _enforce_provider_invariants — cross-field rules+# ---------------------------------------------------------------------------+++ "provider", ["local-ollama", "local-vllm", "local-litellm", "custom"]+)+def test_enforce_invariants_requires_base_url_for_local_and_custom(+ provider: str,+) -> None:+ """Local + custom providers must point at a base_url.++ A NULL base_url for these providers is meaningless (we have no+ canonical endpoint to fall back to). Catching here means the API+ refuses to create the row rather than letting the agents service+ log a baffling "no base_url configured" warning at request time.+ """+ payload = LlmCredentialUpsert(provider=provider, base_url=None, api_key="k")+ with pytest.raises(HTTPException) as exc:+ _enforce_provider_invariants(payload, existing=None)+ assert exc.value.status_code == 422+ assert "requires base_url" in exc.value.detail++[email protected]("provider", ["openai", "anthropic", "azure-openai"])+def test_enforce_invariants_requires_api_key_for_hosted_when_no_existing(+ provider: str,+) -> None:+ """Hosted SaaS providers require a key on the *first* write.++ A subsequent rotation-only PUT (``api_key=None`` on an existing+ row that already has a ciphertext) is fine — see the next test.+ """+ payload = LlmCredentialUpsert(provider=provider, api_key=None)+ with pytest.raises(HTTPException) as exc:+ _enforce_provider_invariants(payload, existing=None)+ assert exc.value.status_code == 422+ assert "requires api_key" in exc.value.detail+++def test_enforce_invariants_allows_rotation_only_put_for_hosted() -> None:+ """``api_key=None`` is fine when the existing row already has one.++ This is the critical UX for "edit base_url without re-typing a+ 100-char secret". If we 422'd here, every PUT would force a key+ re-entry, which is exactly the friction operators complain about.+ """+ existing = _llm_row(provider="openai", api_key_vault="vault:v1:abc")+ payload = LlmCredentialUpsert(provider="openai", api_key=None)+ # Should not raise.+ _enforce_provider_invariants(payload, existing=existing)+++def test_enforce_invariants_local_with_no_key_is_allowed() -> None:+ """Local providers without auth legitimately have no key."""+ payload = LlmCredentialUpsert(+ provider="local-ollama",+ base_url="http://localhost:11434",+ api_key=None,+ )+ # Should not raise.+ _enforce_provider_invariants(payload, existing=None)+++# ---------------------------------------------------------------------------+# GET /api/v1/llm/credentials+# ---------------------------------------------------------------------------+++async def test_get_returns_none_when_no_row() -> None:+ """Empty case returns ``None`` (not 404) so the UI can render its CTA.++ Returning a 404 would force the frontend to dispatch on error codes+ when the "no credential set" case is the *expected* state for a+ fresh tenant. ``None`` keeps the success/failure axes orthogonal.+ """+ db = _mock_db_with_existing(None)+ user = _build_user()++ result = await get_llm_credential(current_user=user, db=db)++ assert result is None+ db.execute.assert_awaited_once()+++async def test_get_returns_projection_when_row_exists() -> None:+ """Existing row → ``LlmCredentialView`` with ``has_api_key=True``.++ The plaintext key remains in the vault — the response only signals+ presence so the UI can render "credential present" without ever+ receiving the secret.+ """+ user = _build_user()+ row = _llm_row(+ tenant_id=user.tenant_id,+ provider="anthropic",+ model="claude-3-5-sonnet-20241022",+ api_key_vault="vault:v1:abc",+ )+ db = _mock_db_with_existing(row)++ result = await get_llm_credential(current_user=user, db=db)++ assert result is not None+ assert result.provider == "anthropic"+ assert result.model == "claude-3-5-sonnet-20241022"+ assert result.has_api_key is True+++async def test_get_scopes_query_to_current_tenant() -> None:+ """The SELECT must filter by the caller's tenant_id.++ Defence-in-depth — the DB also has RLS on this table — but a+ forgotten predicate here would leak credentials across tenants if+ RLS were ever disabled in a future migration. Asserting the+ compiled SQL shape catches the regression at unit-test speed.+ """+ user = _build_user()+ db = _mock_db_with_existing(None)+ await get_llm_credential(current_user=user, db=db)++ stmt = db.execute.await_args.args[0]+ compiled = str(stmt.compile(compile_kwargs={"literal_binds": True}))+ assert "tenant_llm_credentials" in compiled+ assert (str(user.tenant_id) in compiled) or (user.tenant_id.hex in compiled)+++# ---------------------------------------------------------------------------+# PUT /api/v1/llm/credentials — happy paths+# ---------------------------------------------------------------------------+++async def test_put_creates_new_row_encrypts_and_emits_audit() -> None:+ """First-time PUT for a tenant: row added, key encrypted, audit emitted.++ This is the "I just got an OpenAI key, let me wire it in" flow. We+ assert the four invariants that matter:++ * ``db.add`` is called exactly once with a row object whose+ ``api_key_vault`` is a ``vault:v1:`` ciphertext (not the plaintext).+ * ``last_rotated_at`` is populated (powers the "rotated N days ago"+ UI hint).+ * ``emit_audit`` is called with ``action="settings.llm.upsert"``,+ ``rotated=True``, ``created=True``, and the new provider — the+ compliance-evidence shape the buyer's auditor reads.+ * The audit changes never include the plaintext key.+ """+ user = _build_user()+ db = _mock_db_with_existing(None)+ request = _build_request()++ payload = LlmCredentialUpsert(+ provider="openai",+ base_url="https://api.openai.com",+ model="gpt-4o-mini",+ api_key="sk-real-secret-DO-NOT-LOG",+ )++ with patch(+ "app.api.v1.endpoints.llm_credentials.emit_audit",+ new=AsyncMock(return_value=None),+ ) as mock_emit:+ result = await upsert_llm_credential(+ payload=payload, request=request, current_user=user, db=db+ )++ db.add.assert_called_once()+ added_row = db.add.call_args.args[0]+ assert added_row.provider == "openai"+ assert added_row.api_key_vault is not None+ assert added_row.api_key_vault.startswith(_CIPHER_PREFIX)+ assert "sk-real-secret-DO-NOT-LOG" not in added_row.api_key_vault+ assert added_row.last_rotated_at is not None+ db.commit.assert_awaited_once()++ mock_emit.assert_awaited_once()+ audit_kwargs = mock_emit.call_args.kwargs+ assert audit_kwargs["action"] == "settings.llm.upsert"+ assert audit_kwargs["resource"] == "llm_credential"+ assert audit_kwargs["actor_id"] == user.user_id+ assert audit_kwargs["actor_email"] == user.email+ assert audit_kwargs["request"] is request++ changes = audit_kwargs["changes"]+ assert changes["provider"] == "openai"+ assert changes["previous_provider"] is None+ assert changes["rotated"] is True+ assert changes["created"] is True+ assert changes["enabled"] is True+ # Plaintext leakage check — the audit row must never carry the key.+ assert "api_key" not in changes+ assert "sk-real-secret-DO-NOT-LOG" not in str(changes)++ assert result.provider == "openai"+ assert result.has_api_key is True+++async def test_put_rotation_only_keeps_existing_ciphertext_and_does_not_bump_rotated_at() -> None:+ """``api_key=None`` on an existing row: keep ciphertext, no rotation bump.++ Use case: operator changes ``model`` from gpt-4o to gpt-4o-mini+ without re-typing the secret. We assert (1) the stored ciphertext+ is unchanged, (2) ``last_rotated_at`` is not bumped (it's only the+ *key rotation* timestamp, not the *row update* timestamp), and (3)+ the audit row reports ``rotated=False`` so the compliance log+ distinguishes config tweaks from key rotations.+ """+ user = _build_user()+ original_ciphertext = "vault:v1:original_token"+ original_rotated_at = datetime(2024, 1, 1, tzinfo=UTC)+ existing = _llm_row(+ tenant_id=user.tenant_id,+ provider="openai",+ model="gpt-4o",+ api_key_vault=original_ciphertext,+ last_rotated_at=original_rotated_at,+ )+ db = _mock_db_with_existing(existing)+ request = _build_request()++ payload = LlmCredentialUpsert(+ provider="openai",+ model="gpt-4o-mini", # changed+ api_key=None, # keep existing+ )++ with patch(+ "app.api.v1.endpoints.llm_credentials.emit_audit",+ new=AsyncMock(return_value=None),+ ) as mock_emit:+ result = await upsert_llm_credential(+ payload=payload, request=request, current_user=user, db=db+ )++ # The mutated row is the same object; assert the in-place mutations.+ assert existing.api_key_vault == original_ciphertext+ assert existing.last_rotated_at == original_rotated_at+ assert existing.model == "gpt-4o-mini"+ db.add.assert_not_called() # update path, not insert.+ db.commit.assert_awaited_once()++ mock_emit.assert_awaited_once()+ changes = mock_emit.call_args.kwargs["changes"]+ assert changes["rotated"] is False+ assert changes["created"] is False+ assert result.has_api_key is True+++async def test_put_with_new_key_bumps_last_rotated_at_on_existing() -> None:+ """Supplying a new ``api_key`` on an existing row → fresh ciphertext + rotation bump.++ The audit row's ``rotated=True`` flag is the compliance signal that+ a key rotation happened. Operators can join this against the+ rotation policy ("rotate every 90 days") to prove hygiene.+ """+ user = _build_user()+ original_ciphertext = "vault:v1:old_token"+ original_rotated_at = datetime(2024, 1, 1, tzinfo=UTC)+ existing = _llm_row(+ tenant_id=user.tenant_id,+ provider="openai",+ api_key_vault=original_ciphertext,+ last_rotated_at=original_rotated_at,+ )+ db = _mock_db_with_existing(existing)+ request = _build_request()++ payload = LlmCredentialUpsert(provider="openai", api_key="sk-new-secret")++ with patch(+ "app.api.v1.endpoints.llm_credentials.emit_audit",+ new=AsyncMock(return_value=None),+ ) as mock_emit:+ await upsert_llm_credential(+ payload=payload, request=request, current_user=user, db=db+ )++ # Ciphertext changed; the new value is a real vault token (not the+ # plaintext) and is not the original.+ assert existing.api_key_vault != original_ciphertext+ assert existing.api_key_vault.startswith(_CIPHER_PREFIX)+ assert "sk-new-secret" not in existing.api_key_vault+ # last_rotated_at advanced.+ assert existing.last_rotated_at > original_rotated_at++ changes = mock_emit.call_args.kwargs["changes"]+ assert changes["rotated"] is True+ assert changes["created"] is False+++async def test_put_records_provider_transition_in_audit_changes() -> None:+ """Switching providers writes both the new and previous provider.++ The buyer's auditor wants to know "this tenant moved from openai+ to azure-openai on day X" — the ``previous_provider`` key in the+ audit row's changes payload is that signal.+ """+ user = _build_user()+ existing = _llm_row(+ tenant_id=user.tenant_id,+ provider="openai",+ api_key_vault="vault:v1:original",+ )+ db = _mock_db_with_existing(existing)+ request = _build_request()++ payload = LlmCredentialUpsert(+ provider="azure-openai", api_key="sk-azure"+ )++ with patch(+ "app.api.v1.endpoints.llm_credentials.emit_audit",+ new=AsyncMock(return_value=None),+ ) as mock_emit:+ await upsert_llm_credential(+ payload=payload, request=request, current_user=user, db=db+ )++ changes = mock_emit.call_args.kwargs["changes"]+ assert changes["provider"] == "azure-openai"+ assert changes["previous_provider"] == "openai"+++# ---------------------------------------------------------------------------+# PUT /api/v1/llm/credentials — failure paths+# ---------------------------------------------------------------------------+++async def test_put_vault_failure_surfaces_500_and_does_not_commit() -> None:+ """A vault failure must not leave a half-written row.++ The endpoint encrypts *before* mutating the session. We assert:++ * the call surfaces a 500 (not a 422 — the operator did nothing+ wrong; the platform's vault is misconfigured),+ * ``db.commit()`` was never awaited,+ * no row was added to the session.+ """+ user = _build_user()+ db = _mock_db_with_existing(None)+ request = _build_request()+ payload = LlmCredentialUpsert(provider="openai", api_key="sk-secret")++ bad_vault = MagicMock()+ bad_vault.encrypt.side_effect = CredentialVaultError("vault offline")++ with patch(+ "app.api.v1.endpoints.llm_credentials.get_vault",+ return_value=bad_vault,+ ):+ with pytest.raises(HTTPException) as exc:+ await upsert_llm_credential(+ payload=payload, request=request, current_user=user, db=db+ )++ assert exc.value.status_code == 500+ assert "Failed to encrypt" in exc.value.detail+ db.add.assert_not_called()+ db.commit.assert_not_awaited()+++async def test_put_validation_blocks_local_provider_without_base_url() -> None:+ """Local providers need a base_url — the endpoint must 422 before any DB write.++ Catching here (rather than letting the migration's CHECK constraint+ fire at commit time) gives the UI a precise error and avoids a+ rollback that would look like a 500 to the operator.+ """+ user = _build_user()+ db = _mock_db_with_existing(None)+ request = _build_request()+ payload = LlmCredentialUpsert(provider="local-ollama", base_url=None)++ with pytest.raises(HTTPException) as exc:+ await upsert_llm_credential(+ payload=payload, request=request, current_user=user, db=db+ )++ assert exc.value.status_code == 422+ assert "requires base_url" in exc.value.detail+ db.add.assert_not_called()+ db.commit.assert_not_awaited()+++async def test_put_validation_blocks_first_openai_write_without_api_key() -> None:+ """Openai (and other hosted SaaS) require an api_key on first write."""+ user = _build_user()+ db = _mock_db_with_existing(None)+ request = _build_request()+ payload = LlmCredentialUpsert(provider="openai", api_key=None)++ with pytest.raises(HTTPException) as exc:+ await upsert_llm_credential(+ payload=payload, request=request, current_user=user, db=db+ )++ assert exc.value.status_code == 422+ assert "requires api_key" in exc.value.detail+ db.add.assert_not_called()+++async def test_put_audit_failure_does_not_500_the_call() -> None:+ """The audit log must never tank a successful write.++ If the audit DB is briefly unavailable, the buyer's tenant should+ still be able to rotate keys — losing one audit row is preferable+ to losing the rotation (which a panicked admin would re-attempt,+ likely with the wrong key).+ """+ user = _build_user()+ db = _mock_db_with_existing(None)+ request = _build_request()+ payload = LlmCredentialUpsert(provider="openai", api_key="sk-secret")++ with patch(+ "app.api.v1.endpoints.llm_credentials.emit_audit",+ new=AsyncMock(side_effect=RuntimeError("audit table is gone")),+ ):+ # Should NOT raise — the endpoint catches and logs.+ result = await upsert_llm_credential(+ payload=payload, request=request, current_user=user, db=db+ )++ assert result.provider == "openai"+ assert result.has_api_key is True+ db.commit.assert_awaited_once()+++# ---------------------------------------------------------------------------+# DELETE /api/v1/llm/credentials+# ---------------------------------------------------------------------------+++async def test_delete_existing_row_emits_audit_with_provider() -> None:+ """DELETE on an existing row → 204 + audit row capturing the provider.++ The audit row is the only post-delete record of *what* was there.+ We assert it carries the provider so the buyer's compliance team+ can answer "what credential was active when X happened?".+ """+ user = _build_user()+ existing = _llm_row(+ tenant_id=user.tenant_id,+ provider="openai",+ api_key_vault="vault:v1:abc",+ )+ db = _mock_db_with_existing(existing, extra_executes=1) # SELECT then DELETE+ request = _build_request()++ with patch(+ "app.api.v1.endpoints.llm_credentials.emit_audit",+ new=AsyncMock(return_value=None),+ ) as mock_emit:+ result = await delete_llm_credential(+ request=request, current_user=user, db=db+ )++ assert result is None # 204 response+ # Two execute calls: SELECT existing, then DELETE.+ assert db.execute.await_count == 2+ db.commit.assert_awaited_once()++ mock_emit.assert_awaited_once()+ kwargs = mock_emit.call_args.kwargs+ assert kwargs["action"] == "settings.llm.delete"+ assert kwargs["resource"] == "llm_credential"+ assert kwargs["changes"] == {"provider": "openai"}+++async def test_delete_idempotent_when_no_row_exists() -> None:+ """DELETE on an empty row is a no-op: 204, no audit, no error.++ Idempotency matters because the UI may issue a DELETE on an+ already-cleared tenant during a "Clear configuration" flow. A 404+ here would force the UI to special-case the empty state, which we+ avoid by mirroring the GET endpoint's "None is fine" semantics.+ """+ user = _build_user()+ db = _mock_db_with_existing(None, extra_executes=1) # SELECT (None) then DELETE+ request = _build_request()++ with patch(+ "app.api.v1.endpoints.llm_credentials.emit_audit",+ new=AsyncMock(return_value=None),+ ) as mock_emit:+ result = await delete_llm_credential(+ request=request, current_user=user, db=db+ )++ assert result is None+ db.commit.assert_awaited_once()+ # No row to audit, so no audit row.+ mock_emit.assert_not_awaited()+++async def test_delete_audit_failure_does_not_500_the_call() -> None:+ """Audit failures on DELETE are also non-fatal — same rationale as PUT."""+ user = _build_user()+ existing = _llm_row(tenant_id=user.tenant_id, provider="openai")+ db = _mock_db_with_existing(existing, extra_executes=1)+ request = _build_request()++ with patch(+ "app.api.v1.endpoints.llm_credentials.emit_audit",+ new=AsyncMock(side_effect=RuntimeError("audit table is gone")),+ ):+ result = await delete_llm_credential(+ request=request, current_user=user, db=db+ )++ assert result is None+ db.commit.assert_awaited_once()+++# ---------------------------------------------------------------------------+# Vault round-trip — covers the encrypt/decrypt boundary that the+# endpoint relies on. These are pure-vault tests but live here because+# they encode the contract the endpoint depends on (a fresh PUT must+# round-trip to the agents service's resolver and back).+# ---------------------------------------------------------------------------+++def test_vault_roundtrip_preserves_plaintext() -> None:+ """encrypt → decrypt yields the same plaintext.++ The agents service uses a vendored copy of this vault to decrypt+ at request time; if these two ever diverge on the wire format, the+ explain endpoint will silently fall back to the env baseline. This+ is the "the wire format hasn't drifted" smoke test.+ """+ vault = CredentialVault(Fernet.generate_key())+ plaintext = "sk-very-real-secret"+ ciphertext = vault.encrypt(plaintext)+ assert ciphertext.startswith(_CIPHER_PREFIX)+ assert plaintext not in ciphertext+ assert vault.decrypt(ciphertext) == plaintext+++def test_vault_round_trip_through_endpoint_path_and_back() -> None:+ """End-to-end: PUT-time encrypt + GET-time projection + agents-side decrypt.++ Simulates the full WS-H2 hot path:++ 1. The endpoint encrypts the plaintext key using the canonical vault.+ 2. The row is projected to the wire model — ``has_api_key=True``,+ no plaintext.+ 3. The agents service (using the same key material) decrypts the+ stored ciphertext and recovers the plaintext.++ We use the canonical vault (services/api) to encrypt and the+ vendored vault (services/agents) to decrypt, exactly mirroring the+ cross-service runtime contract.+ """+ # ruff: noqa: PLC0415 — local imports keep agents-side bootstrapping+ # out of the module-load path; this test is the only one that needs them.+ import sys+ from pathlib import Path++ agents_root = Path(__file__).resolve().parents[2] / "agents"+ if str(agents_root) not in sys.path:+ sys.path.insert(0, str(agents_root))++ from app.security.credential_vault import CredentialVault as AgentsVault++ key = Fernet.generate_key()+ api_vault = CredentialVault(key)+ agents_vault = AgentsVault(key)++ plaintext = "sk-cross-service-secret"+ ciphertext = api_vault.encrypt(plaintext)++ # Endpoint stores ``ciphertext`` in ``api_key_vault``; projection+ # surfaces only a presence flag.+ row = _llm_row(api_key_vault=ciphertext)+ view = _project(row)+ assert view.has_api_key is True+ assert "api_key" not in view.model_dump()++ # Agents-side resolver decrypts the same ciphertext and recovers+ # the original plaintext.+ assert agents_vault.decrypt(ciphertext) == plaintext