Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(llm): offer a custom endpoint's chat models in the BYOK selector (#1543)
Vito14ae676bf11a
diff --git a/apps/docs/docs/api/rest.md b/apps/docs/docs/api/rest.mdindex 73c37b6ba..ed8688895 100644--- a/apps/docs/docs/api/rest.md+++ b/apps/docs/docs/api/rest.md@@ -263,6 +263,7 @@ Where the model runs, and whether the credential pointed at it works.| `GET` | `/llm/runtime` | Where the local model is **actually** running || `GET/PUT/DELETE` | `/llm/credentials` | The tenant's own provider credential (BYOK) || `POST` | `/llm/credentials/test` | Place one real call with it and report what happened |+| `POST` | `/llm/credentials/models` | List the chat models a `custom` endpoint serves |`/llm/status` reports *configuration*; `/llm/runtime` reports *outcome*. Thetwo differ more often than you would expect: a GPU reservation is a request,@@ -330,6 +331,55 @@ Requires `settings:write` rather than `settings:read`: it spends the tenant'smoney and touches an external service, which is not a read. The request is onetoken against a fixed prompt.+#### Listing a custom endpoint's models++The `custom` provider points at any OpenAI-compatible endpoint. Rather than+typing a model id, the console can ask the endpoint's `GET {base_url}/models`+and offer the chat models it lists:++```bash+curl -sX POST "$AISOC_API/api/v1/llm/credentials/models" \+ -H "Authorization: Bearer $AISOC_TOKEN" \+ -H "Content-Type: application/json" \+ -d '{"base_url": "https://llm.internal.example/v1", "api_key": "..."}'+```++```json+{+ "outcome": "ok",+ "detail": "2 chat model(s) of 5 listed.",+ "models": [+ { "id": "vendor/model-a", "name": null },+ { "id": "vendor/model-b", "name": null }+ ],+ "listed": 5,+ "excluded": 3,+ "truncated": false+}+```++Both body fields are optional. `base_url` falls back to the saved `custom`+credential's. `api_key` is used for this request only and never stored; when+it is omitted the saved key is used, **but only if the target is the saved base+URL** — a form that changed the base URL and left the key blank sends no key+rather than the saved one.++`/models` rows say little in the OpenAI shape, so the filter reads, strongest+first: a declared `capabilities` flag, declared input/output modalities+(`architecture.modality`, `input_modalities`/`output_modalities`), and a `type`+or `mode` label (`chat`, or an image, video, audio, embedding or rerank family).+When any row's label says chat, a row labelled otherwise is left out, because+that endpoint names its routes in the label. A row that declares nothing is+kept unless its id names a non-text family (`embed`, `tts`, `whisper`, `image`,+…). It is a filter for a dropdown, not a capability guarantee: any id can still+be typed, and `/test` is what proves it answers.++Bounded on purpose: 10 s for the whole exchange, 2 MB of body, 5,000 rows+examined and 500 models returned (`truncated` says when that cap was reached).+Redirects are not followed. The same air-gap check and SSRF guard as `/test`+apply, and `outcome` uses the same four values. It requires `settings:write`,+like `/test`, because it makes the server dial an address the caller chose.+See [Where the model runs](../operations/where-the-model-runs) for the fourdeployment options these routes report on.diff --git a/apps/docs/docs/operations/where-the-model-runs.md b/apps/docs/docs/operations/where-the-model-runs.mdindex af7b69f69..df30be95b 100644--- a/apps/docs/docs/operations/where-the-model-runs.md+++ b/apps/docs/docs/operations/where-the-model-runs.md@@ -136,6 +136,15 @@ and is safe to run from a health check. See the[REST reference](../api/rest#testing-a-credential) for the full responseshape.+**Choosing a model on a custom endpoint.** With the `custom` provider, *List+models from endpoint* asks the endpoint's `/models` and offers the chat models+it lists — image, audio, embedding and similar entries are filtered out. It+works before you save, and the model field still takes any typed id: an+endpoint without a model list, or one that lists something the filter misses,+is never a dead end. See the+[REST reference](../api/rest#listing-a-custom-endpoints-models) for how the+filter decides.+## Checking what is actually runningA GPU reservation is a *request*. A model can still land on the CPU for want ofdiff --git a/apps/web/src/components/settings/SettingsView.byok.test.tsx b/apps/web/src/components/settings/SettingsView.byok.test.tsxindex ac975fea7..14c25a0f9 100644--- a/apps/web/src/components/settings/SettingsView.byok.test.tsx+++ b/apps/web/src/components/settings/SettingsView.byok.test.tsx@@ -44,6 +44,7 @@ vi.mock('swr', () => ({const mockGetLlmCredential = vi.hoisted(() => vi.fn());const mockUpsertLlmCredential = vi.hoisted(() => vi.fn());const mockDeleteLlmCredential = vi.hoisted(() => vi.fn());+const mockDiscoverLlmModels = vi.hoisted(() => vi.fn());const mockGetAirgapStatus = vi.hoisted(() => vi.fn());const mockGetLlmStatus = vi.hoisted(() => vi.fn());const mockGetConnectors = vi.hoisted(() => vi.fn());@@ -57,6 +58,7 @@ vi.mock('@/lib/api', () => ({getLlmCredential: mockGetLlmCredential,upsertLlmCredential: mockUpsertLlmCredential,deleteLlmCredential: mockDeleteLlmCredential,+ discoverLlmModels: mockDiscoverLlmModels,},connectorsApi: {list: mockGetConnectors,@@ -469,3 +471,160 @@ describe('WS-H2 — BYOKCard: loading & error states', () => {).toBeInTheDocument();});});++// ──────────────────────────────────────────────────────────────────────────────+describe('BYOKCard: model discovery for the custom provider', () => {+ beforeEach(() => {+ clearSwrMaps();+ vi.clearAllMocks();+ mockUpsertLlmCredential.mockResolvedValue(undefined);+ });++ async function openCustomForm() {+ await renderDeploymentTab({ credential: null });+ await userEvent.click(+ await screen.findByRole('button', { name: /configure byok/i }),+ );+ await userEvent.selectOptions(+ await screen.findByRole('combobox', { name: /provider/i }),+ 'custom',+ );+ await userEvent.type(+ screen.getByPlaceholderText('https://internal-llm.example.com/v1'),+ 'https://gw.example.com/v1',+ );+ await userEvent.type(screen.getByLabelText(/api key/i), 'typed-key');+ }++ it('is offered for the custom provider only', async () => {+ await renderDeploymentTab({ credential: null });+ await userEvent.click(+ await screen.findByRole('button', { name: /configure byok/i }),+ );++ expect(+ screen.queryByRole('button', { name: /list models from endpoint/i }),+ ).not.toBeInTheDocument();++ await userEvent.selectOptions(+ screen.getByRole('combobox', { name: /provider/i }),+ 'custom',+ );++ expect(+ screen.getByRole('button', { name: /list models from endpoint/i }),+ ).toBeInTheDocument();+ });++ it('lists what the server returned and saves the chosen id', async () => {+ mockDiscoverLlmModels.mockResolvedValue({+ outcome: 'ok',+ detail: '2 chat model(s) of 5 listed.',+ models: [{ id: 'acme/chat-large' }, { id: 'acme/chat-small', name: 'Chat Small' }],+ listed: 5,+ excluded: 3,+ truncated: false,+ });+ await openCustomForm();++ await userEvent.click(+ screen.getByRole('button', { name: /list models from endpoint/i }),+ );++ expect(mockDiscoverLlmModels).toHaveBeenCalledWith({+ base_url: 'https://gw.example.com/v1',+ api_key: 'typed-key',+ });+ const picker = await screen.findByRole('combobox', {+ name: /discovered chat models/i,+ });+ const offered = Array.from((picker as HTMLSelectElement).options)+ .map(o => o.value)+ .filter(Boolean);+ expect(offered).toEqual(['acme/chat-large', 'acme/chat-small']);+ expect(screen.getByRole('status')).toHaveTextContent(+ /2 chat models of 5 listed .*3 .*non-chat entries hidden/i,+ );++ await userEvent.selectOptions(picker, 'acme/chat-small');+ expect(screen.getByPlaceholderText('provider default')).toHaveValue(+ 'acme/chat-small',+ );++ await userEvent.click(+ screen.getByRole('button', { name: /save credential/i }),+ );+ await waitFor(() =>+ expect(mockUpsertLlmCredential).toHaveBeenCalledWith(+ expect.objectContaining({+ provider: 'custom',+ base_url: 'https://gw.example.com/v1',+ model: 'acme/chat-small',+ }),+ ),+ );+ });++ it('keeps the typed id working when discovery fails', async () => {+ mockDiscoverLlmModels.mockResolvedValue({+ outcome: 'refused',+ detail: 'The endpoint does not publish a model list at /models (HTTP 404).',+ models: [],+ listed: 0,+ excluded: 0,+ truncated: false,+ });+ await openCustomForm();++ await userEvent.click(+ screen.getByRole('button', { name: /list models from endpoint/i }),+ );++ expect(await screen.findByRole('status')).toHaveTextContent(+ /HTTP 404.*still accepts a typed id/i,+ );+ expect(+ screen.queryByRole('combobox', { name: /discovered chat models/i }),+ ).not.toBeInTheDocument();++ await userEvent.type(+ screen.getByPlaceholderText('provider default'),+ 'hand-typed-model',+ );+ await userEvent.click(+ screen.getByRole('button', { name: /save credential/i }),+ );+ await waitFor(() =>+ expect(mockUpsertLlmCredential).toHaveBeenCalledWith(+ expect.objectContaining({ model: 'hand-typed-model' }),+ ),+ );+ });++ it('drops a list fetched for one base URL when the base URL changes', async () => {+ mockDiscoverLlmModels.mockResolvedValue({+ outcome: 'ok',+ detail: '1 chat model(s) of 1 listed.',+ models: [{ id: 'acme/chat-large' }],+ listed: 1,+ excluded: 0,+ truncated: false,+ });+ await openCustomForm();+ await userEvent.click(+ screen.getByRole('button', { name: /list models from endpoint/i }),+ );+ expect(+ await screen.findByRole('combobox', { name: /discovered chat models/i }),+ ).toBeInTheDocument();++ await userEvent.type(+ screen.getByPlaceholderText('https://internal-llm.example.com/v1'),+ '2',+ );++ expect(+ screen.queryByRole('combobox', { name: /discovered chat models/i }),+ ).not.toBeInTheDocument();+ });+});diff --git a/apps/web/src/components/settings/SettingsView.tsx b/apps/web/src/components/settings/SettingsView.tsxindex 158b80f39..83625db7a 100644--- a/apps/web/src/components/settings/SettingsView.tsx+++ b/apps/web/src/components/settings/SettingsView.tsx@@ -37,6 +37,7 @@ import {type FullTenant,type LlmCredentialUpsert,type LlmCredentialView,+ type LlmModelDiscoveryResult,type LlmProvider,type LlmStatus,type LlmWritableProvider,@@ -1918,6 +1919,14 @@ function BYOKCard({const [apiKey, setApiKey] = useState('');const [enabled, setEnabled] = useState(true);const [formError, setFormError] = useState<string | null>(null);+ const [discovering, setDiscovering] = useState(false);+ // Remembers which base URL the list came from: a list fetched for one+ // endpoint says nothing about another, so it is shown only while that base+ // URL is still the one in the form.+ const [discovery, setDiscovery] = useState<{+ baseUrl: string;+ result: LlmModelDiscoveryResult;+ } | null>(null);// Reset form whenever we enter edit mode (or the underlying credential// moves while editing). Never preload the API key — it lives only in@@ -1994,6 +2003,42 @@ function BYOKCard({setEditing(false);};+ // Ask the custom endpoint which chat models it serves. Uses what is in the+ // form, so it works before the credential is saved; a blank key means "use+ // the stored one", which the server sends only to the stored base URL. On+ // any failure the list stays empty and the typed model id is what is saved.+ const onDiscoverClick = async () => {+ const target = baseUrl.trim();+ setDiscovering(true);+ try {+ const result = await deploymentApi.discoverLlmModels({+ base_url: target ? target : null,+ api_key: apiKey.trim() ? apiKey.trim() : null,+ });+ setDiscovery({ baseUrl: target, result });+ } catch (err) {+ setDiscovery({+ baseUrl: target,+ result: {+ outcome: 'refused',+ detail: extractApiDetail(err, 'Could not list models.'),+ models: [],+ listed: 0,+ excluded: 0,+ truncated: false,+ },+ });+ } finally {+ setDiscovering(false);+ }+ };++ const shownDiscovery =+ provider === 'custom' && discovery && discovery.baseUrl === baseUrl.trim()+ ? discovery.result+ : null;+ const discoveredModels = shownDiscovery?.models ?? [];+// One real call with the saved credential. Reports four outcomes, and// `unverified` is deliberately not an error: an air-gapped deployment// refusing the egress is the posture working, and a red toast there would@@ -2302,6 +2347,67 @@ function BYOKCard({/></Field></div>+ {provider === 'custom' ? (+ <div className="mt-4 flex flex-col gap-2">+ <div className="flex flex-col gap-2 sm:flex-row sm:items-center">+ <button+ type="button"+ onClick={onDiscoverClick}+ disabled={discovering || submitting || !baseUrl.trim()}+ className="shrink-0 rounded-md border border-gray-700 bg-gray-900 px-3 py-1.5 text-xs font-medium text-gray-200 hover:bg-gray-800 disabled:opacity-50"+ >+ {discovering ? 'Listing models…' : 'List models from endpoint'}+ </button>+ {discoveredModels.length > 0 ? (+ <select+ aria-label="Discovered chat models"+ value={discoveredModels.some(m => m.id === model) ? model : ''}+ onChange={e => {+ if (e.target.value) setModel(e.target.value);+ }}+ disabled={submitting}+ className={inputClass()}+ >+ <option value="">Choose a discovered model…</option>+ {discoveredModels.map(m => (+ <option key={m.id} value={m.id}>+ {m.name && m.name !== m.id ? `${m.id} — ${m.name}` : m.id}+ </option>+ ))}+ </select>+ ) : null}+ </div>+ {shownDiscovery ? (+ <p+ role="status"+ className={clsx(+ 'text-xs',+ shownDiscovery.outcome === 'ok' && discoveredModels.length > 0+ ? 'text-gray-400'+ : 'text-amber-300/80',+ )}+ >+ {shownDiscovery.outcome === 'ok' && discoveredModels.length > 0+ ? `${discoveredModels.length} chat model${+ discoveredModels.length === 1 ? '' : 's'+ } of ${shownDiscovery.listed} listed by the endpoint${+ shownDiscovery.excluded > 0+ ? ` (${shownDiscovery.excluded} image, audio, embedding or other non-chat entries hidden)`+ : ''+ }.${+ shownDiscovery.truncated ? ' Only the first part of a longer list is shown.' : ''+ } Pick one, or type any id in the Model field.`+ : `${shownDiscovery.detail} The Model field still accepts a typed id.`}+ </p>+ ) : (+ <p className="text-xs text-gray-500">+ Asks the endpoint's <code className="font-mono">/models</code>{' '}+ for the chat models it serves. Optional — a typed model id+ always works.+ </p>+ )}+ </div>+ ) : null}<div className="mt-4"><Togglechecked={enabled}diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.tsindex 41217da47..bacface38 100644--- a/apps/web/src/lib/api.ts+++ b/apps/web/src/lib/api.ts@@ -6438,6 +6438,23 @@ export interface LlmCredentialTestResult {latency_ms?: number;}+/**+ * The chat models a custom OpenAI-compatible endpoint listed at ``/models``.+ *+ * ``listed`` is every distinct id the endpoint returned and ``excluded`` the+ * ones judged not chat-capable (image, audio, embedding and similar), so the+ * selector can say how much it filtered. Any outcome other than ``ok`` comes+ * with an empty ``models``: the hand-typed id is the fallback, never a guess.+ */+export interface LlmModelDiscoveryResult {+ outcome: "ok" | "refused" | "unreachable" | "unverified";+ detail: string;+ models: { id: string; name?: string | null }[];+ listed: number;+ excluded: number;+ truncated: boolean;+}+export const deploymentApi = {/** Live air-gap policy snapshot for this pod. Safe to poll. */getAirgapStatus: () => request<AirgapStatus>("/api/v1/airgap/status"),@@ -6473,6 +6490,19 @@ export const deploymentApi = {request<LlmCredentialTestResult>("/api/v1/llm/credentials/test", {method: "POST",}),++ /**+ * List the chat models a ``custom`` endpoint serves.+ *+ * Works before the form is saved: ``base_url`` and ``api_key`` are used for+ * this one request and never stored. Omit ``api_key`` to use the stored key,+ * which the server sends only to the stored base URL.+ */+ discoverLlmModels: (payload: { base_url?: string | null; api_key?: string | null }) =>+ request<LlmModelDiscoveryResult>("/api/v1/llm/credentials/models", {+ method: "POST",+ body: JSON.stringify(payload),+ }),};// ─── Reports / Executive digest (WS-G2) ─────────────────────────────────────diff --git a/docs/openapi.yaml b/docs/openapi.yamlindex 2de81fae7..382b89a42 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -19882,6 +19882,53 @@ paths:$ref: '#/components/schemas/LlmCredentialTestResult'security:- HTTPBearer: []+ /api/v1/llm/credentials/models:+ post:+ tags:+ - llm+ summary: List the chat models a custom OpenAI-compatible endpoint serves+ description: 'Ask ``GET {base_url}/models`` and return the chat-capable entries.+++ For the ``custom`` provider, whose model was otherwise a hand-typed id.+++ `settings:write`, like `/test`: this dials an operator-supplied URL with++ private addresses permitted, which is not something a read-only role++ should be able to point at the network.+++ The stored key goes only to the stored base URL. A form that has changed++ the base URL and left the key blank gets a keyless request, not the saved++ key sent somewhere it was never configured to go; an endpoint that needs a++ key then answers 401 and the detail says so.'+ operationId: discover_llm_models_api_v1_llm_credentials_models_post+ requestBody:+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/LlmModelDiscoveryRequest'+ required: true+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/LlmModelDiscoveryResult'+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ security:+ - HTTPBearer: []/api/v1/threatintel/stix/indicators:get:tags:@@ -28138,6 +28185,20 @@ components:required:- discoveredtitle: DiscoverResponse+ DiscoveredModelView:+ properties:+ id:+ type: string+ title: Id+ name:+ anyOf:+ - type: string+ - type: 'null'+ title: Name+ type: object+ required:+ - id+ title: DiscoveredModelViewDismissTuningRequest:properties:reason:@@ -32475,6 +32536,76 @@ components:only signal we expose — same convention the OAuth credential viewuses (``has_secret``).'+ LlmModelDiscoveryRequest:+ properties:+ base_url:+ anyOf:+ - type: string+ maxLength: 512+ - type: 'null'+ title: Base Url+ api_key:+ anyOf:+ - type: string+ maxLength: 4096+ - type: 'null'+ title: Api Key+ type: object+ title: LlmModelDiscoveryRequest+ description: 'Where to list models from, for a form that may not be saved yet.+++ Both fields are optional. ``base_url`` falls back to the stored ``custom``++ credential''s. ``api_key`` is used for this one request and never stored;++ when it is omitted the stored key is used, but only if the target is the++ stored base URL -- see :func:`discover_llm_models`.'+ LlmModelDiscoveryResult:+ properties:+ outcome:+ type: string+ enum:+ - ok+ - refused+ - unreachable+ - unverified+ title: Outcome+ detail:+ type: string+ title: Detail+ models:+ items:+ $ref: '#/components/schemas/DiscoveredModelView'+ type: array+ title: Models+ listed:+ type: integer+ title: Listed+ excluded:+ type: integer+ title: Excluded+ truncated:+ type: boolean+ title: Truncated+ type: object+ required:+ - outcome+ - detail+ - models+ - listed+ - excluded+ - truncated+ title: LlmModelDiscoveryResult+ description: 'The chat models an endpoint listed, or why there are none.+++ ``outcome`` uses the Test button''s vocabulary. Anything other than ``ok``++ leaves ``models`` empty and the console falls back to the hand-typed id,++ which is the path that always worked; nothing is invented to fill the gap.'LoginRequest:properties:email:diff --git a/services/api/app/api/v1/endpoints/llm_credentials.py b/services/api/app/api/v1/endpoints/llm_credentials.pyindex 36f6143c7..ede1cf326 100644--- a/services/api/app/api/v1/endpoints/llm_credentials.py+++ b/services/api/app/api/v1/endpoints/llm_credentials.py@@ -19,6 +19,14 @@ Endpoint contractHard-delete the row. The platform falls back to env-var configurationon the next request (the existing v1 behaviour).+* ``POST /api/v1/llm/credentials/test``+ Place one minimal completion with the stored credential.++* ``POST /api/v1/llm/credentials/models``+ List the chat models a ``custom`` endpoint serves, so the console can+ offer them instead of requiring a hand-typed id. See+ :mod:`app.services.llm_model_discovery`.+Why this lives in services/api------------------------------@@ -62,6 +70,7 @@ from app.models.llm_credential import TenantLlmCredentialfrom app.security.credential_vault import CredentialVaultError, get_vaultfrom app.services.audit import emit_auditfrom app.services.llm_credential_probe import ProbeResult, probe_credential+from app.services.llm_model_discovery import Discovery, discover_chat_modelslogger = logging.getLogger(__name__)@@ -98,6 +107,31 @@ _PROVIDERS_REQUIRING_KEY: frozenset[str] = frozenset({"openai", "anthropic", "az# --------------------------------------------------------------- request / response models+def _check_base_url(v: str | None) -> str | None:+ if v is None:+ return v+ v = v.strip()+ if not v:+ return None+ # Light shape check — we don't want to over-validate (operators+ # legitimately run weird proxies on weird ports inside private+ # networks) but we should reject obvious garbage so the UI gets+ # a useful 422 instead of a 500 at request time.+ parsed = urlparse(v)+ if parsed.scheme not in {"http", "https"}:+ raise ValueError("base_url must be an http(s) URL")+ if not parsed.hostname:+ raise ValueError("base_url must include a hostname")+ return v+++def _blank_to_none(v: str | None) -> str | None:+ if v is None:+ return v+ v = v.strip()+ return v or None++class LlmCredentialUpsert(BaseModel):"""Payload for ``PUT /api/v1/llm/credentials``.@@ -119,29 +153,12 @@ class LlmCredentialUpsert(BaseModel):@field_validator("base_url")@classmethoddef _validate_base_url(cls, v: str | None) -> str | None:- if v is None:- return v- v = v.strip()- if not v:- return None- # Light shape check — we don't want to over-validate (operators- # legitimately run weird proxies on weird ports inside private- # networks) but we should reject obvious garbage so the UI gets- # a useful 422 instead of a 500 at request time.- parsed = urlparse(v)- if parsed.scheme not in {"http", "https"}:- raise ValueError("base_url must be an http(s) URL")- if not parsed.hostname:- raise ValueError("base_url must include a hostname")- return v+ return _check_base_url(v)@field_validator("api_key")@classmethoddef _validate_api_key(cls, v: str | None) -> str | None:- if v is None:- return v- v = v.strip()- return v or None+ return _blank_to_none(v)class LlmCredentialView(BaseModel):@@ -521,3 +538,125 @@ async def test_llm_credential(model=result.model,latency_ms=result.latency_ms,)+++# --------------------------------------------------------- model discovery+++class LlmModelDiscoveryRequest(BaseModel):+ """Where to list models from, for a form that may not be saved yet.++ Both fields are optional. ``base_url`` falls back to the stored ``custom``+ credential's. ``api_key`` is used for this one request and never stored;+ when it is omitted the stored key is used, but only if the target is the+ stored base URL -- see :func:`discover_llm_models`.+ """++ base_url: str | None = Field(default=None, max_length=512)+ api_key: str | None = Field(default=None, max_length=4096)++ @field_validator("base_url")+ @classmethod+ def _validate_base_url(cls, v: str | None) -> str | None:+ return _check_base_url(v)++ @field_validator("api_key")+ @classmethod+ def _validate_api_key(cls, v: str | None) -> str | None:+ return _blank_to_none(v)+++class DiscoveredModelView(BaseModel):+ id: str+ name: str | None = None+++class LlmModelDiscoveryResult(BaseModel):+ """The chat models an endpoint listed, or why there are none.++ ``outcome`` uses the Test button's vocabulary. Anything other than ``ok``+ leaves ``models`` empty and the console falls back to the hand-typed id,+ which is the path that always worked; nothing is invented to fill the gap.+ """++ outcome: Literal["ok", "refused", "unreachable", "unverified"]+ detail: str+ models: list[DiscoveredModelView]+ listed: int+ excluded: int+ truncated: bool+++def _same_endpoint(a: str | None, b: str | None) -> bool:+ return bool(a and b) and (a or "").strip().rstrip("/") == (b or "").strip().rstrip("/")+++ "/models",+ response_model=LlmModelDiscoveryResult,+ summary="List the chat models a custom OpenAI-compatible endpoint serves",+)+async def discover_llm_models(+ payload: LlmModelDiscoveryRequest,+ current_user: Annotated[AuthUser, Depends(require_permission("settings:write"))],+ db: DBSession,+) -> LlmModelDiscoveryResult:+ """Ask ``GET {base_url}/models`` and return the chat-capable entries.++ For the ``custom`` provider, whose model was otherwise a hand-typed id.++ `settings:write`, like `/test`: this dials an operator-supplied URL with+ private addresses permitted, which is not something a read-only role+ should be able to point at the network.++ The stored key goes only to the stored base URL. A form that has changed+ the base URL and left the key blank gets a keyless request, not the saved+ key sent somewhere it was never configured to go; an endpoint that needs a+ key then answers 401 and the detail says so.+ """+ res = await db.execute(select(TenantLlmCredential).where(TenantLlmCredential.tenant_id == current_user.tenant_id))+ existing: TenantLlmCredential | None = res.scalar_one_or_none()+ stored = existing if existing is not None and existing.provider == "custom" else None++ base_url = payload.base_url or (stored.base_url if stored else None)+ if not base_url:+ raise HTTPException(+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,+ detail="Model discovery needs a base URL: enter one, or save a custom credential first.",+ )++ api_key = payload.api_key+ if api_key is None and stored is not None and stored.api_key_vault and _same_endpoint(stored.base_url, base_url):+ try:+ api_key = get_vault().decrypt(stored.api_key_vault)+ except CredentialVaultError:+ return LlmModelDiscoveryResult(+ outcome="refused",+ detail=(+ "The stored key could not be decrypted. It was encrypted with a vault key "+ "this deployment no longer has; re-enter it to fix."+ ),+ models=[],+ listed=0,+ excluded=0,+ truncated=False,+ )++ result: Discovery = await discover_chat_models(base_url=base_url, api_key=api_key)+ logger.info(+ "settings.llm.discover",+ extra={+ "tenant": str(current_user.tenant_id),+ "outcome": result.outcome,+ "listed": result.listed,+ "offered": len(result.models),+ },+ )+ return LlmModelDiscoveryResult(+ outcome=result.outcome,+ detail=result.detail,+ models=[DiscoveredModelView(id=m.id, name=m.name) for m in result.models],+ listed=result.listed,+ excluded=result.excluded,+ truncated=result.truncated,+ )diff --git a/services/api/app/services/llm_model_discovery.py b/services/api/app/services/llm_model_discovery.pynew file mode 100644index 000000000..510f8d148--- /dev/null+++ b/services/api/app/services/llm_model_discovery.py@@ -0,0 +1,455 @@+"""List the chat models an OpenAI-compatible endpoint serves, for the BYOK selector.++Why this exists+---------------+The ``custom`` provider points a tenant at any OpenAI-compatible endpoint, and+the model was a hand-typed id. Nothing checked it, so a typo or an id the+endpoint does not serve surfaced as triage quietly falling back to the+deterministic path. Most such endpoints publish ``GET {base}/models``; this asks+it, with the tenant's key, and offers the chat-capable answers as choices. The+hand-typed id stays: discovery is a convenience over it, never a gate on it.++Bounds+------+The URL and the response are both tenant-controlled, so neither decides what+this process spends:++* :data:`DISCOVERY_TIMEOUT_SECONDS` is a deadline on the *whole* exchange, not+ httpx's per-read timeout, so an endpoint that drips one byte every nine+ seconds cannot hold a settings panel open;+* :data:`DISCOVERY_MAX_BYTES` caps the body as it streams in, after content+ decoding, so a compressed response cannot expand past it either;+* :data:`DISCOVERY_MAX_ROWS` caps how many rows are examined and+ :data:`DISCOVERY_MAX_MODELS` how many chat models are returned. They are+ separate on purpose: an aggregating endpoint can list hundreds of image,+ audio and embedding rows ahead of its chat models, and capping the raw rows+ at the selector size would drop chat models for the sake of rows that were+ going to be filtered out anyway.++Redirects are not followed. A ``3xx`` is reported, not chased to a host the+guard below never saw.++What counts as chat-capable+---------------------------+AiSOC calls ``POST {base}/chat/completions`` and nothing else, so the question+is whether an id can serve that. The OpenAI ``/models`` shape does not say: a+row is ``{"id", "object", "created", "owned_by"}``. Endpoints that aggregate+many model families often add a field that does, under one of a few spellings.+Each row is judged on what it declares, strongest evidence first:++1. **Declared metadata.** Three optional fields are read when present:++ * ``capabilities`` -- an object with a boolean ``chat``,+ ``completion_chat``, ``chat_completion`` or ``chat_completions`` key, or+ a list naming one of those;+ * modalities -- ``architecture.modality`` as ``"text+image->text"``,+ ``architecture.input_modalities`` / ``output_modalities``, the same two+ at the top level, or ``modalities`` as ``{"input": [...],+ "output": [...]}``. Output without ``text``, or input without ``text``,+ is not chat. Text in and text out is;+ * a label -- ``type`` or ``mode``, split into words. A word naming a+ non-text family (image, video, audio, speech, embedding, rerank,+ moderation, ...) is not chat; ``chat``, ``llm``, ``vlm``, ``language``+ or ``conversational`` is.++ A declared negative wins over a declared positive: a row that is wrongly+ hidden can still be typed by hand, and a row that is wrongly offered fails+ at triage time, quietly.++2. **A label the catalogue uses as a classifier.** Some endpoints list one row+ per (model, endpoint) pair and name the endpoint in ``type`` -- a model that+ only serves an asynchronous or non-chat route appears under a label that is+ neither ``chat`` nor a known non-text family. When *any* row's label says+ chat, the catalogue has shown how it spells chat, and a row whose label says+ something else is not offered. When no row's label says chat, the field is+ not a capability classifier in that catalogue (it may mean "base" vs+ "fine-tuned") and it is ignored.++3. **The id, when nothing was declared.** A plain OpenAI-compatible server+ (vLLM, a local runtime, most proxies) declares nothing, and its rows are+ chat models unless the id names a non-text family -- ``embed``,+ ``embedding``, ``tts``, ``whisper``, ``transcribe``, ``image``, ``video``,+ ``rerank``, ``moderation`` and the like, matched as whole words of the id.+ Everything else is offered.++An id appearing on several rows is offered once, if any of its rows is chat.+The heuristic is a selector filter, not an authorisation: what an endpoint+actually serves is decided by the endpoint, and the Test button remains the+way to find out.++SSRF and air-gap+----------------+The same two guards :mod:`app.services.llm_credential_probe` applies to this+very base URL, in the same order: air-gap first, before anything is dialled,+then ``validate_outbound_url(..., allow_private=True)`` so a private model host+is reachable while loopback and link-local -- the cloud-metadata address --+stay refused. See that module for why the webhook guard is the wrong shape.+"""++from __future__ import annotations++import asyncio+import json+import re+from dataclasses import dataclass, field+from typing import Any, Literal++import httpx+import structlog++from app._vendor.ssrf_guard import SSRFError, validate_outbound_url+from app.core.airgap import AirgapViolation, enforce_airgap_for_url+from app.services.model_aliases import completions_url_for_base++logger = structlog.get_logger(__name__)++Outcome = Literal["ok", "refused", "unreachable", "unverified"]+Verdict = Literal["chat", "not_chat", "unknown"]++#: A human is waiting on this in a settings panel.+DISCOVERY_TIMEOUT_SECONDS = 10.0+#: A model list is a few hundred kilobytes even for an endpoint aggregating+#: hundreds of models; two megabytes is generous and still bounded.+DISCOVERY_MAX_BYTES = 2_000_000+#: Rows examined. Bounds the work on a body that fits under the byte cap by+#: being nothing but tiny rows.+DISCOVERY_MAX_ROWS = 5_000+#: Chat models returned. More than this is not a dropdown anyone can use, and+#: the response says when it was reached.+DISCOVERY_MAX_MODELS = 500++_CHAT_COMPLETIONS_SUFFIX = "/chat/completions"++_WORD = re.compile(r"[a-z0-9]+")++#: Words in a ``type``/``mode`` label that mean the row serves chat.+_CHAT_LABEL_WORDS = frozenset({"chat", "llm", "vlm", "language", "conversational"})++#: Words that name a family which cannot answer a text chat completion. Used+#: for labels and, when a row declares nothing, for the id.+_NON_CHAT_WORDS = frozenset(+ {+ "audio",+ "diffusion",+ "embed",+ "embedding",+ "embeddings",+ "image",+ "images",+ "moderation",+ "moderations",+ "music",+ "ocr",+ "realtime",+ "rerank",+ "reranker",+ "reranking",+ "speech",+ "transcribe",+ "transcription",+ "transcriptions",+ "tts",+ "video",+ "videos",+ "whisper",+ }+)++_CHAT_CAPABILITY_KEYS = ("chat", "completion_chat", "chat_completion", "chat_completions")+_LABEL_FIELDS = ("type", "mode")+++@dataclass(frozen=True)+class DiscoveredModel:+ """What a selector needs. No pricing, no owner blob, nothing echoed raw."""++ id: str+ name: str | None = None+++@dataclass(frozen=True)+class Discovery:+ """Outcome of one discovery attempt.++ ``listed`` counts the distinct ids the endpoint returned and ``excluded``+ the ones judged not chat-capable, so the console can say "2 of 5"+ rather than present a filtered list as the whole catalogue.+ """++ outcome: Outcome+ detail: str+ models: list[DiscoveredModel] = field(default_factory=list)+ listed: int = 0+ excluded: int = 0+ truncated: bool = False+++class _DiscoveryError(Exception):+ def __init__(self, outcome: Outcome, detail: str) -> None:+ super().__init__(detail)+ self.outcome: Outcome = outcome+ self.detail = detail+++# --------------------------------------------------------------- the URL+++def models_url_for_base(base_url: str) -> str:+ """``/models`` beside the chat-completions URL the runtime will call.++ Derived from :func:`completions_url_for_base` rather than by appending to+ the base, so a base given as ``https://host`` or as+ ``https://host/v1/chat/completions`` lists models from the same place triage+ sends completions to.+ """+ completions = completions_url_for_base(base_url)+ return completions[: -len(_CHAT_COMPLETIONS_SUFFIX)] + "/models"+++# --------------------------------------------------------------- the filter+++def _words(value: str) -> set[str]:+ return set(_WORD.findall(value.lower()))+++def _label(row: dict[str, Any]) -> str | None:+ for key in _LABEL_FIELDS:+ value = row.get(key)+ if isinstance(value, str) and value.strip():+ return value+ return None+++def label_verdict(label: str) -> Verdict:+ """Judge a ``type``/``mode`` label. A non-text word wins over a chat word."""+ words = _words(label)+ if words & _NON_CHAT_WORDS:+ return "not_chat"+ if words & _CHAT_LABEL_WORDS:+ return "chat"+ return "unknown"+++def _capabilities_verdict(row: dict[str, Any]) -> Verdict | None:+ caps = row.get("capabilities")+ if isinstance(caps, dict):+ for key in _CHAT_CAPABILITY_KEYS:+ if isinstance(caps.get(key), bool):+ return "chat" if caps[key] else "not_chat"+ return None+ if isinstance(caps, list):+ named = {item.strip().lower() for item in caps if isinstance(item, str)}+ if named & set(_CHAT_CAPABILITY_KEYS):+ return "chat"+ return None+++def _modality_list(value: Any) -> list[str] | None:+ if isinstance(value, list):+ return [item.strip().lower() for item in value if isinstance(item, str)]+ return None+++def _modalities_verdict(row: dict[str, Any]) -> Verdict | None:+ raw_arch = row.get("architecture")+ arch: dict[str, Any] = raw_arch if isinstance(raw_arch, dict) else {}+ declared = row.get("modalities")++ inputs = _modality_list(arch.get("input_modalities"))+ if inputs is None:+ inputs = _modality_list(row.get("input_modalities"))+ outputs = _modality_list(arch.get("output_modalities"))+ if outputs is None:+ outputs = _modality_list(row.get("output_modalities"))+ if isinstance(declared, dict):+ if inputs is None:+ inputs = _modality_list(declared.get("input"))+ if outputs is None:+ outputs = _modality_list(declared.get("output"))+ elif inputs is None:+ inputs = _modality_list(declared)++ arrow = arch.get("modality") if isinstance(arch.get("modality"), str) else row.get("modality")+ if isinstance(arrow, str) and "->" in arrow:+ left, right = arrow.lower().split("->", 1)+ if inputs is None:+ inputs = [part.strip() for part in left.split("+") if part.strip()]+ if outputs is None:+ outputs = [part.strip() for part in right.split("+") if part.strip()]++ if outputs is not None and "text" not in outputs:+ return "not_chat"+ if inputs is not None and "text" not in inputs:+ return "not_chat"+ if outputs is not None:+ return "chat"+ # Text input alone says nothing: an embedding model takes text too.+ return None+++def row_verdict(row: dict[str, Any], *, labels_classify: bool) -> Verdict:+ """Judge one row. See the module docstring for the order and why.++ ``labels_classify`` is the catalogue-level fact from step 2: whether any+ row's label said chat.+ """+ label = _label(row)+ declared = [+ _capabilities_verdict(row),+ _modalities_verdict(row),+ label_verdict(label) if label is not None else None,+ ]+ if "not_chat" in declared:+ return "not_chat"+ if "chat" in declared:+ return "chat"+ if label is not None and labels_classify:+ return "not_chat"+ model_id = row.get("id")+ if isinstance(model_id, str) and _words(model_id) & _NON_CHAT_WORDS:+ return "not_chat"+ return "chat"+++def filter_chat_models(rows: list[Any]) -> Discovery:+ """Apply the filter to a ``data`` list. Pure, so it is testable on fixtures."""+ usable: list[dict[str, Any]] = []+ for row in rows[:DISCOVERY_MAX_ROWS]:+ if not isinstance(row, dict):+ continue+ model_id = row.get("id")+ if isinstance(model_id, str) and model_id.strip() and len(model_id) <= 256:+ usable.append(row)++ labels_classify = any(label_verdict(label) == "chat" for label in (_label(row) for row in usable) if label is not None)++ seen: dict[str, DiscoveredModel] = {}+ all_ids: set[str] = set()+ for row in usable:+ model_id = row["id"].strip()+ all_ids.add(model_id)+ if model_id in seen or row_verdict(row, labels_classify=labels_classify) != "chat":+ continue+ name = row.get("name")+ seen[model_id] = DiscoveredModel(id=model_id, name=name.strip()[:200] if isinstance(name, str) and name.strip() else None)++ chat = sorted(seen.values(), key=lambda m: m.id)+ truncated = len(chat) > DISCOVERY_MAX_MODELS or len(rows) > DISCOVERY_MAX_ROWS+ chat = chat[:DISCOVERY_MAX_MODELS]+ excluded = len(all_ids) - len(seen)++ if not chat:+ detail = f"The endpoint listed {len(all_ids)} model(s) and none looked chat-capable. Type the model id instead."+ else:+ detail = f"{len(chat)} chat model(s) of {len(all_ids)} listed."+ if truncated:+ detail += " The list was longer than this panel shows; type an id that is not offered."+ return Discovery(outcome="ok", detail=detail, models=chat, listed=len(all_ids), excluded=excluded, truncated=truncated)+++# --------------------------------------------------------------- the call+++async def _read_bounded(client: httpx.AsyncClient, url: str, headers: dict[str, str]) -> bytes:+ async with client.stream("GET", url, headers=headers) as response:+ code = response.status_code+ if code in (401, 403):+ raise _DiscoveryError("refused", f"The endpoint rejected the key when listing models (HTTP {code}).")+ if code == 404:+ raise _DiscoveryError(+ "refused",+ "The endpoint does not publish a model list at /models (HTTP 404). Type the model id instead.",+ )+ if code != 200:+ raise _DiscoveryError("refused", f"The endpoint answered HTTP {code} when listing models.")+ chunks: list[bytes] = []+ total = 0+ async for chunk in response.aiter_bytes():+ total += len(chunk)+ if total > DISCOVERY_MAX_BYTES:+ raise _DiscoveryError(+ "refused",+ f"The model list is larger than {DISCOVERY_MAX_BYTES // 1_000_000} MB, so it was not read. Type the model id instead.",+ )+ chunks.append(chunk)+ return b"".join(chunks)+++async def discover_chat_models(+ *,+ base_url: str,+ api_key: str | None,+ transport: httpx.AsyncBaseTransport | None = None,+) -> Discovery:+ """List the endpoint's chat models. Never raises; every failure is a state.++ ``transport`` exists for tests: they keep the real client, the real+ streaming read and the real guards, and swap only what is on the wire.+ """+ base = (base_url or "").strip().rstrip("/")+ if not base:+ return Discovery(outcome="refused", detail="No base URL is configured, so there is nothing to list.")+ url = models_url_for_base(base)++ try:+ enforce_airgap_for_url(url)+ except AirgapViolation as exc:+ return Discovery(+ outcome="unverified",+ detail=f"This deployment is air-gapped, so the model list was not requested: {exc}",+ )++ try:+ validate_outbound_url(url, allow_private=True)+ except SSRFError as exc:+ return Discovery(outcome="refused", detail=f"The base URL was refused: {exc}")++ headers = {"Accept": "application/json"}+ if api_key:+ headers["Authorization"] = f"Bearer {api_key}"++ try:+ async with asyncio.timeout(DISCOVERY_TIMEOUT_SECONDS):+ async with httpx.AsyncClient(+ timeout=DISCOVERY_TIMEOUT_SECONDS,+ follow_redirects=False,+ transport=transport,+ ) as client:+ raw = await _read_bounded(client, url, headers)+ except _DiscoveryError as exc:+ return Discovery(outcome=exc.outcome, detail=exc.detail)+ except (TimeoutError, httpx.TimeoutException):+ return Discovery(+ outcome="unreachable",+ detail=f"{base} did not return its model list within {DISCOVERY_TIMEOUT_SECONDS:.0f}s.",+ )+ except Exception as exc: # noqa: BLE001 - every failure is a state+ logger.info("llm_discovery.unreachable", error=type(exc).__name__)+ return Discovery(outcome="unreachable", detail=f"Could not reach {base} ({type(exc).__name__}).")++ try:+ body = json.loads(raw)+ except ValueError:+ return Discovery(outcome="refused", detail="The endpoint's model list was not JSON.")++ rows = body.get("data") if isinstance(body, dict) else body+ if not isinstance(rows, list):+ return Discovery(outcome="refused", detail="The endpoint's answer had no `data` list of models.")++ return filter_chat_models(rows)+++__all__ = [+ "DISCOVERY_MAX_BYTES",+ "DISCOVERY_MAX_MODELS",+ "DISCOVERY_MAX_ROWS",+ "DISCOVERY_TIMEOUT_SECONDS",+ "DiscoveredModel",+ "Discovery",+ "discover_chat_models",+ "filter_chat_models",+ "label_verdict",+ "models_url_for_base",+ "row_verdict",+]diff --git a/services/api/tests/test_llm_model_discovery.py b/services/api/tests/test_llm_model_discovery.pynew file mode 100644index 000000000..0e5b9481a--- /dev/null+++ b/services/api/tests/test_llm_model_discovery.py@@ -0,0 +1,498 @@+"""A custom endpoint's chat models can populate the selector, and nothing else can.++What this closes+----------------+The ``custom`` BYOK provider points a tenant at any OpenAI-compatible endpoint,+and the model was a hand-typed id. ``POST /api/v1/llm/credentials/models`` asks+``GET {base}/models`` and returns the chat-capable entries.++Three things are pinned here, and each has a way to look right while being+wrong:++* **The filter.** An aggregating endpoint lists image, video, audio and+ embedding models beside its chat models, and its ``/models`` rows say so only+ through optional fields. A filter that passed everything would still produce+ a populated dropdown; the fixtures below assert the exact surviving set.+* **The bounds.** The URL and the body are both tenant-controlled. Each bound is+ driven past its limit through the real ``httpx`` client and streaming read --+ only the bytes on the wire are swapped, via ``httpx.MockTransport``.+* **The key.** The stored key goes only to the stored base URL. A form that+ changed the base URL and left the key blank must not send the saved key to+ the new host.+"""++from __future__ import annotations++import asyncio+import functools+import json+import uuid+from datetime import UTC, datetime+from types import SimpleNamespace+from typing import Any++import httpx+import pytest+from app.api.v1.deps import CurrentUser+from app.api.v1.endpoints import llm_credentials as llm_credentials_module+from app.api.v1.endpoints.auth import get_current_user+from app.db.database import get_db+from app.security.credential_vault import get_vault+from app.services import llm_model_discovery as discovery_mod+from app.services.llm_model_discovery import (+ DISCOVERY_MAX_BYTES,+ DISCOVERY_MAX_MODELS,+ discover_chat_models,+ filter_chat_models,+ models_url_for_base,+)+from fastapi import FastAPI+from fastapi.testclient import TestClient++#: A private literal: permitted by ``allow_private=True`` and needs no DNS.+PRIVATE_BASE = "http://10.20.30.40:8000/v1"++[email protected](autouse=True)+def _airgap_off(monkeypatch: pytest.MonkeyPatch) -> None:+ # Patched where it is read, at call time -- see test_llm_credential_probe.+ from app.core import airgap as airgap_mod++ monkeypatch.setattr(airgap_mod.settings, "AISOC_AIRGAPPED", False, raising=False)+++def _ids(rows: list[Any]) -> set[str]:+ return {m.id for m in filter_chat_models(rows).models}+++# --------------------------------------------------------------- fixtures++#: Shaped like an aggregating gateway: one row per (model, endpoint) pair, the+#: endpoint named in ``type``, and several model families in one list. Every+#: name is invented.+MIXED_GATEWAY_ROWS: list[dict[str, Any]] = [+ {"id": "acme/chat-large", "type": "vendor-a/chat-completions"},+ {"id": "acme/chat-large", "type": "vendor-a/responses"},+ {"id": "acme/chat-small", "type": "vendor-a/chat-completions"},+ # Served over a second chat protocol too; offered once.+ {"id": "globex/reasoner-2", "type": "vendor-b/messages"},+ {"id": "globex/reasoner-2", "type": "vendor-a/chat-completions"},+ # A chat row whose id contains a non-text word: the declared type wins.+ {"id": "acme/voice-audio-preview", "type": "vendor-a/chat-completions"},+ # Only reachable through an asynchronous, non-chat route.+ {"id": "acme/deep-thinker-pro", "type": "vendor-a/responses/submit"},+ {"id": "initech/decider-1", "type": "internal/decisions"},+ {"id": "acme/pixel-3", "type": "vendor-a/image-generations"},+ {"id": "acme/pixel-3-edit", "type": "vendor-a/image-editing"},+ {"id": "umbrella/clipmaker", "type": "internal/video-generations/submit"},+ {"id": "acme/vector-small", "type": "vendor-a/embeddings"},+ {"id": "hooli/narrator", "type": "internal/text-to-speech"},+ {"id": "hooli/scribe", "type": "internal/speech-to-text/submit"},+ {"id": "hooli/composer", "type": "internal/audio-generations/submit"},+ {"id": "initech/page-reader", "type": "internal/optical-character-recognition"},+ # A multi-purpose router listed under every route it accepts.+ {"id": "globex/router", "type": "vendor-a/image-generations"},+ {"id": "globex/router", "type": "vendor-a/chat-completions"},+]++MIXED_GATEWAY_CHAT = {"acme/chat-large", "acme/chat-small", "globex/reasoner-2", "acme/voice-audio-preview", "globex/router"}+++class TestTheFilter:+ def test_a_mixed_gateway_list_keeps_exactly_its_chat_models(self) -> None:+ result = filter_chat_models(MIXED_GATEWAY_ROWS)++ assert {m.id for m in result.models} == MIXED_GATEWAY_CHAT+ assert result.listed == 15+ assert result.excluded == 10+ assert [m.id for m in result.models] == sorted(MIXED_GATEWAY_CHAT), "offered once each, in a stable order"++ def test_a_plain_openai_shaped_list_is_judged_by_the_id(self) -> None:+ """A self-hosted server declares nothing. Its rows are chat models+ unless the id names a non-text family, as a whole word."""+ rows = [+ {"id": i, "object": "model", "owned_by": "local"}+ for i in (+ "example-chat-7b",+ "example-instruct-70b:q4",+ "example-embed-large",+ "text-embedding-small",+ "example-tts-1",+ "example-whisper-small",+ "example-image-xl",+ "example-rerank-base",+ "imagery-analyst-13b",+ )+ ]++ assert _ids(rows) == {"example-chat-7b", "example-instruct-70b:q4", "imagery-analyst-13b"}++ def test_declared_modalities_are_honoured(self) -> None:+ rows = [+ {"id": "a/text", "architecture": {"modality": "text->text"}},+ {"id": "a/vision", "architecture": {"modality": "text+image->text"}},+ {"id": "a/painter", "architecture": {"modality": "text->image"}},+ {"id": "a/listener", "architecture": {"input_modalities": ["audio"], "output_modalities": ["text"]}},+ {"id": "a/both", "architecture": {"input_modalities": ["text"], "output_modalities": ["text", "image"]}},+ {"id": "a/split", "modalities": {"input": ["text"], "output": ["audio"]}},+ {"id": "a/top", "input_modalities": ["text"], "output_modalities": ["text"]},+ ]++ assert _ids(rows) == {"a/text", "a/vision", "a/both", "a/top"}++ def test_declared_capabilities_are_honoured(self) -> None:+ rows = [+ {"id": "a/one", "capabilities": {"completion_chat": True, "vision": False}},+ {"id": "a/two", "capabilities": {"completion_chat": False}},+ {"id": "a/three", "capabilities": ["chat", "tools"]},+ {"id": "a/four-embed", "capabilities": {"function_calling": True}},+ ]++ # a/four-embed declares nothing about chat, so its id decides.+ assert _ids(rows) == {"a/one", "a/three"}++ def test_a_declared_negative_beats_a_declared_positive(self) -> None:+ """Wrongly hidden can still be typed; wrongly offered fails at triage."""+ rows = [{"id": "a/x", "type": "chat", "architecture": {"modality": "text->image"}}]++ assert _ids(rows) == set()++ def test_a_label_that_never_says_chat_is_not_a_classifier(self) -> None:+ """Some endpoints use ``type`` for "base" vs "fine-tuned". Reading+ that as "not chat" would empty the list."""+ rows = [+ {"id": "a/base-model", "type": "base"},+ {"id": "a/tuned", "type": "fine-tuned"},+ {"id": "a/embed-v1", "type": "base"},+ ]++ assert _ids(rows) == {"a/base-model", "a/tuned"}++ def test_rows_without_a_usable_id_are_skipped(self) -> None:+ rows: list[Any] = ["a-string", None, {"id": ""}, {"id": 7}, {"name": "x"}, {"id": "ok/model"}]++ result = filter_chat_models(rows)++ assert [m.id for m in result.models] == ["ok/model"]+ assert result.listed == 1++ def test_nothing_chat_capable_says_so_and_points_at_the_typed_id(self) -> None:+ result = filter_chat_models([{"id": "a/embed", "type": "embeddings"}])++ assert result.outcome == "ok"+ assert result.models == []+ assert "type the model id" in result.detail.lower()+++class TestTheUrl:+ @pytest.mark.parametrize(+ ("base", "expected"),+ [+ ("https://gw.example.com/v1", "https://gw.example.com/v1/models"),+ ("https://gw.example.com/v1/", "https://gw.example.com/v1/models"),+ ("https://gw.example.com", "https://gw.example.com/v1/models"),+ ("https://gw.example.com/v1/chat/completions", "https://gw.example.com/v1/models"),+ ],+ )+ def test_models_is_listed_beside_the_completions_url_triage_calls(self, base: str, expected: str) -> None:+ assert models_url_for_base(base) == expected+++# --------------------------------------------------------------- the call+++class _Recorder:+ """A ``MockTransport`` handler that records what reached the wire."""++ def __init__(self, response: httpx.Response | None = None, *, delay: float = 0.0) -> None:+ self.response = response or httpx.Response(200, json={"object": "list", "data": MIXED_GATEWAY_ROWS})+ self.delay = delay+ self.requests: list[httpx.Request] = []++ async def __call__(self, request: httpx.Request) -> httpx.Response:+ self.requests.append(request)+ if self.delay:+ await asyncio.sleep(self.delay)+ return self.response++ @property+ def transport(self) -> httpx.MockTransport:+ return httpx.MockTransport(self)+++class TestTheCall:+ @pytest.mark.asyncio+ async def test_it_asks_models_with_the_key_and_returns_only_chat(self) -> None:+ wire = _Recorder()++ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k-123", transport=wire.transport)++ assert result.outcome == "ok", result.detail+ assert {m.id for m in result.models} == MIXED_GATEWAY_CHAT+ [request] = wire.requests+ assert request.method == "GET"+ assert str(request.url) == f"{PRIVATE_BASE}/models"+ assert request.headers["authorization"] == "Bearer k-123"++ @pytest.mark.asyncio+ async def test_no_key_means_no_authorization_header(self) -> None:+ wire = _Recorder()++ await discover_chat_models(base_url=PRIVATE_BASE, api_key=None, transport=wire.transport)++ assert "authorization" not in wire.requests[0].headers++ @pytest.mark.parametrize(+ ("status_code", "needle"),+ [(401, "rejected the key"), (403, "rejected the key"), (404, "type the model id"), (500, "http 500")],+ )+ @pytest.mark.asyncio+ async def test_each_refusal_is_a_reported_state(self, status_code: int, needle: str) -> None:+ wire = _Recorder(httpx.Response(status_code, text="nope"))++ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)++ assert result.outcome == "refused"+ assert needle in result.detail.lower()+ assert result.models == []++ @pytest.mark.asyncio+ async def test_a_redirect_is_reported_not_followed(self) -> None:+ """Following it would dial a host the SSRF guard never saw."""+ wire = _Recorder(httpx.Response(302, headers={"location": "http://169.254.169.254/latest/meta-data/"}))++ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)++ assert result.outcome == "refused"+ assert len(wire.requests) == 1++ @pytest.mark.parametrize("body", [b"<html>not json</html>", b'{"models": []}', b'"a string"'])+ @pytest.mark.asyncio+ async def test_a_body_that_is_not_a_model_list_is_refused(self, body: bytes) -> None:+ wire = _Recorder(httpx.Response(200, content=body))++ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)++ assert result.outcome == "refused"+ assert result.models == []+++class TestTheBounds:+ @pytest.mark.asyncio+ async def test_a_body_over_the_byte_cap_is_not_read(self) -> None:+ rows = [{"id": f"m/{i:07d}", "type": "chat", "pad": "x" * 200} for i in range(DISCOVERY_MAX_BYTES // 200 + 10)]+ wire = _Recorder(httpx.Response(200, content=json.dumps({"data": rows}).encode()))++ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)++ assert result.outcome == "refused"+ assert "larger than" in result.detail+ assert result.models == []++ @pytest.mark.asyncio+ async def test_the_selector_is_capped_and_says_so(self) -> None:+ rows = [{"id": f"m/{i:04d}", "type": "chat"} for i in range(DISCOVERY_MAX_MODELS + 25)]+ rows += [{"id": f"img/{i:04d}", "type": "image"} for i in range(50)]+ wire = _Recorder(httpx.Response(200, json={"data": rows}))++ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)++ assert result.outcome == "ok"+ assert len(result.models) == DISCOVERY_MAX_MODELS+ assert result.truncated is True+ assert result.listed == DISCOVERY_MAX_MODELS + 75++ @pytest.mark.asyncio+ async def test_the_raw_row_count_does_not_cut_chat_models(self) -> None:+ """Non-chat rows ahead of the chat ones must not use up the selector's+ budget: capping raw rows at the selector size would drop these."""+ rows = [{"id": f"img/{i:04d}", "type": "image"} for i in range(DISCOVERY_MAX_MODELS + 100)]+ rows += [{"id": "acme/chat-late", "type": "chat"}]+ wire = _Recorder(httpx.Response(200, json={"data": rows}))++ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)++ assert [m.id for m in result.models] == ["acme/chat-late"]+ assert result.truncated is False++ @pytest.mark.asyncio+ async def test_the_deadline_covers_the_whole_exchange(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(discovery_mod, "DISCOVERY_TIMEOUT_SECONDS", 0.2)+ wire = _Recorder(delay=2.0)++ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)++ assert result.outcome == "unreachable"+ assert "did not return" in result.detail+++class TestTheGuards:+ @pytest.mark.parametrize(+ "base",+ ["http://169.254.169.254/v1", "http://127.0.0.1:8000/v1", "http://[::1]:8000/v1"],+ )+ @pytest.mark.asyncio+ async def test_metadata_and_loopback_are_refused_before_anything_is_dialled(self, base: str) -> None:+ wire = _Recorder()++ result = await discover_chat_models(base_url=base, api_key="k", transport=wire.transport)++ assert result.outcome == "refused"+ assert "refused" in result.detail.lower()+ assert wire.requests == []++ @pytest.mark.asyncio+ async def test_air_gap_reports_unverified_and_dials_nothing(self, monkeypatch: pytest.MonkeyPatch) -> None:+ from app.core import airgap as airgap_mod++ monkeypatch.setattr(airgap_mod.settings, "AISOC_AIRGAPPED", True, raising=False)+ wire = _Recorder()++ result = await discover_chat_models(base_url="https://gw.example.com/v1", api_key="k", transport=wire.transport)++ assert result.outcome == "unverified"+ assert wire.requests == []+++# --------------------------------------------------------------- the route+++class _Session:+ """Answers the endpoint's one SELECT with a fixed row."""++ def __init__(self, row: Any) -> None:+ self.row = row++ async def execute(self, *_: Any, **__: Any) -> Any:+ return SimpleNamespace(scalar_one_or_none=lambda: self.row)+++def _stored(provider: str = "custom", base_url: str = PRIVATE_BASE, key: str | None = "stored-key") -> SimpleNamespace:+ now = datetime.now(UTC)+ return SimpleNamespace(+ tenant_id=uuid.uuid4(),+ provider=provider,+ base_url=base_url,+ model=None,+ api_key_vault=get_vault().encrypt(key) if key else None,+ settings={},+ enabled=True,+ created_at=now,+ updated_at=now,+ last_rotated_at=now,+ )+++def route(monkeypatch: pytest.MonkeyPatch):+ """The real router, the real permission dependency and the real discovery+ function; only the database row and the bytes on the wire are supplied."""+ wire = _Recorder()+ monkeypatch.setattr(+ llm_credentials_module,+ "discover_chat_models",+ functools.partial(discover_chat_models, transport=wire.transport),+ )+ app = FastAPI()+ app.include_router(llm_credentials_module.router, prefix="/api/v1")+ state: dict[str, Any] = {"row": None, "scopes": None}+ app.dependency_overrides[get_db] = lambda: _Session(state["row"])+ app.dependency_overrides[get_current_user] = lambda: CurrentUser(+ user_id=uuid.uuid4(),+ tenant_id=uuid.uuid4(),+ role="tenant_admin",+ email="[email protected]",+ scopes=state["scopes"],+ )+ return TestClient(app), wire, state+++class TestTheRoute:+ def test_it_is_published_where_the_console_asks_for_it(self) -> None:+ from pathlib import Path++ from app.main import app++ client = (Path(__file__).resolve().parents[3] / "apps" / "web" / "src" / "lib" / "api.ts").read_text(encoding="utf-8")++ assert "/api/v1/llm/credentials/models" in app.openapi()["paths"]+ assert "/api/v1/llm/credentials/models" in client++ def test_an_unsaved_form_lists_with_the_typed_key(self, route: Any) -> None:+ client, wire, _ = route++ response = client.post("/api/v1/llm/credentials/models", json={"base_url": PRIVATE_BASE, "api_key": "typed-key"})++ assert response.status_code == 200, response.text+ body = response.json()+ assert body["outcome"] == "ok"+ assert {m["id"] for m in body["models"]} == MIXED_GATEWAY_CHAT+ assert (body["listed"], body["excluded"]) == (15, 10)+ assert wire.requests[0].headers["authorization"] == "Bearer typed-key"++ def test_a_saved_credential_lists_with_the_stored_key(self, route: Any) -> None:+ client, wire, state = route+ state["row"] = _stored()++ response = client.post("/api/v1/llm/credentials/models", json={})++ assert response.status_code == 200, response.text+ assert str(wire.requests[0].url) == f"{PRIVATE_BASE}/models"+ assert wire.requests[0].headers["authorization"] == "Bearer stored-key"++ def test_the_stored_key_is_not_sent_to_a_different_base_url(self, route: Any) -> None:+ """The form changed the base URL and left the key blank. Sending the+ saved key there would hand it to a host it was never configured for."""+ client, wire, state = route+ state["row"] = _stored()++ response = client.post("/api/v1/llm/credentials/models", json={"base_url": "http://10.99.99.99:9000/v1"})++ assert response.status_code == 200, response.text+ assert str(wire.requests[0].url) == "http://10.99.99.99:9000/v1/models"+ assert "authorization" not in wire.requests[0].headers++ def test_another_providers_key_is_never_used(self, route: Any) -> None:+ client, wire, state = route+ state["row"] = _stored(provider="local-litellm")++ response = client.post("/api/v1/llm/credentials/models", json={"base_url": PRIVATE_BASE})++ assert response.status_code == 200, response.text+ assert "authorization" not in wire.requests[0].headers++ def test_no_base_url_anywhere_is_a_422(self, route: Any) -> None:+ client, wire, _ = route++ response = client.post("/api/v1/llm/credentials/models", json={})++ assert response.status_code == 422+ assert wire.requests == []++ def test_a_non_http_base_url_is_a_422(self, route: Any) -> None:+ client, wire, _ = route++ response = client.post("/api/v1/llm/credentials/models", json={"base_url": "file:///etc/passwd"})++ assert response.status_code == 422+ assert wire.requests == []++ def test_a_read_scoped_key_cannot_make_the_server_dial_out(self, route: Any) -> None:+ """`settings:write`, like `/test`: the route dials an operator-supplied+ URL with private addresses permitted."""+ client, wire, state = route+ state["scopes"] = ["settings:read"]++ response = client.post("/api/v1/llm/credentials/models", json={"base_url": PRIVATE_BASE})++ assert response.status_code == 403+ assert wire.requests == []++ def test_a_write_scoped_key_is_not_refused(self, route: Any) -> None:+ """The negative control for the refusal above."""+ client, _, state = route+ state["scopes"] = ["settings:write"]++ response = client.post("/api/v1/llm/credentials/models", json={"base_url": PRIVATE_BASE})++ assert response.status_code == 200, response.text