Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(llm): offer a custom endpoint's chat models in the BYOK selector (#1543)

Vito14ae676bf11a
diff --git a/apps/docs/docs/api/rest.md b/apps/docs/docs/api/rest.md
index 73c37b6ba..ed8688895 100644
--- a/apps/docs/docs/api/rest.md
+++ b/apps/docs/docs/api/rest.md
@@ -263,6 +263,7 @@ Where the model runs, and whether the credential pointed at it works.
| `GET` | `/llm/runtime` | Where the local model is **actually** running |
| `GET/PUT/DELETE` | `/llm/credentials` | The tenant's own provider credential (BYOK) |
| `POST` | `/llm/credentials/test` | Place one real call with it and report what happened |
+| `POST` | `/llm/credentials/models` | List the chat models a `custom` endpoint serves |
`/llm/status` reports *configuration*; `/llm/runtime` reports *outcome*. The
two differ more often than you would expect: a GPU reservation is a request,
@@ -330,6 +331,55 @@ Requires `settings:write` rather than `settings:read`: it spends the tenant's
money and touches an external service, which is not a read. The request is one
token against a fixed prompt.
+#### Listing a custom endpoint's models
+
+The `custom` provider points at any OpenAI-compatible endpoint. Rather than
+typing a model id, the console can ask the endpoint's `GET {base_url}/models`
+and offer the chat models it lists:
+
+```bash
+curl -sX POST "$AISOC_API/api/v1/llm/credentials/models" \
+ -H "Authorization: Bearer $AISOC_TOKEN" \
+ -H "Content-Type: application/json" \
+ -d '{"base_url": "https://llm.internal.example/v1", "api_key": "..."}'
+```
+
+```json
+{
+ "outcome": "ok",
+ "detail": "2 chat model(s) of 5 listed.",
+ "models": [
+ { "id": "vendor/model-a", "name": null },
+ { "id": "vendor/model-b", "name": null }
+ ],
+ "listed": 5,
+ "excluded": 3,
+ "truncated": false
+}
+```
+
+Both body fields are optional. `base_url` falls back to the saved `custom`
+credential's. `api_key` is used for this request only and never stored; when
+it is omitted the saved key is used, **but only if the target is the saved base
+URL** — a form that changed the base URL and left the key blank sends no key
+rather than the saved one.
+
+`/models` rows say little in the OpenAI shape, so the filter reads, strongest
+first: a declared `capabilities` flag, declared input/output modalities
+(`architecture.modality`, `input_modalities`/`output_modalities`), and a `type`
+or `mode` label (`chat`, or an image, video, audio, embedding or rerank family).
+When any row's label says chat, a row labelled otherwise is left out, because
+that endpoint names its routes in the label. A row that declares nothing is
+kept unless its id names a non-text family (`embed`, `tts`, `whisper`, `image`,
+…). It is a filter for a dropdown, not a capability guarantee: any id can still
+be typed, and `/test` is what proves it answers.
+
+Bounded on purpose: 10 s for the whole exchange, 2 MB of body, 5,000 rows
+examined and 500 models returned (`truncated` says when that cap was reached).
+Redirects are not followed. The same air-gap check and SSRF guard as `/test`
+apply, and `outcome` uses the same four values. It requires `settings:write`,
+like `/test`, because it makes the server dial an address the caller chose.
+
See [Where the model runs](../operations/where-the-model-runs) for the four
deployment options these routes report on.
diff --git a/apps/docs/docs/operations/where-the-model-runs.md b/apps/docs/docs/operations/where-the-model-runs.md
index af7b69f69..df30be95b 100644
--- a/apps/docs/docs/operations/where-the-model-runs.md
+++ b/apps/docs/docs/operations/where-the-model-runs.md
@@ -136,6 +136,15 @@ and is safe to run from a health check. See the
[REST reference](../api/rest#testing-a-credential) for the full response
shape.
+**Choosing a model on a custom endpoint.** With the `custom` provider, *List
+models from endpoint* asks the endpoint's `/models` and offers the chat models
+it lists — image, audio, embedding and similar entries are filtered out. It
+works before you save, and the model field still takes any typed id: an
+endpoint without a model list, or one that lists something the filter misses,
+is never a dead end. See the
+[REST reference](../api/rest#listing-a-custom-endpoints-models) for how the
+filter decides.
+
## Checking what is actually running
A GPU reservation is a *request*. A model can still land on the CPU for want of
diff --git a/apps/web/src/components/settings/SettingsView.byok.test.tsx b/apps/web/src/components/settings/SettingsView.byok.test.tsx
index ac975fea7..14c25a0f9 100644
--- a/apps/web/src/components/settings/SettingsView.byok.test.tsx
+++ b/apps/web/src/components/settings/SettingsView.byok.test.tsx
@@ -44,6 +44,7 @@ vi.mock('swr', () => ({
const mockGetLlmCredential = vi.hoisted(() => vi.fn());
const mockUpsertLlmCredential = vi.hoisted(() => vi.fn());
const mockDeleteLlmCredential = vi.hoisted(() => vi.fn());
+const mockDiscoverLlmModels = vi.hoisted(() => vi.fn());
const mockGetAirgapStatus = vi.hoisted(() => vi.fn());
const mockGetLlmStatus = vi.hoisted(() => vi.fn());
const mockGetConnectors = vi.hoisted(() => vi.fn());
@@ -57,6 +58,7 @@ vi.mock('@/lib/api', () => ({
getLlmCredential: mockGetLlmCredential,
upsertLlmCredential: mockUpsertLlmCredential,
deleteLlmCredential: mockDeleteLlmCredential,
+ discoverLlmModels: mockDiscoverLlmModels,
},
connectorsApi: {
list: mockGetConnectors,
@@ -469,3 +471,160 @@ describe('WS-H2 — BYOKCard: loading & error states', () => {
).toBeInTheDocument();
});
});
+
+// ──────────────────────────────────────────────────────────────────────────────
+describe('BYOKCard: model discovery for the custom provider', () => {
+ beforeEach(() => {
+ clearSwrMaps();
+ vi.clearAllMocks();
+ mockUpsertLlmCredential.mockResolvedValue(undefined);
+ });
+
+ async function openCustomForm() {
+ await renderDeploymentTab({ credential: null });
+ await userEvent.click(
+ await screen.findByRole('button', { name: /configure byok/i }),
+ );
+ await userEvent.selectOptions(
+ await screen.findByRole('combobox', { name: /provider/i }),
+ 'custom',
+ );
+ await userEvent.type(
+ screen.getByPlaceholderText('https://internal-llm.example.com/v1'),
+ 'https://gw.example.com/v1',
+ );
+ await userEvent.type(screen.getByLabelText(/api key/i), 'typed-key');
+ }
+
+ it('is offered for the custom provider only', async () => {
+ await renderDeploymentTab({ credential: null });
+ await userEvent.click(
+ await screen.findByRole('button', { name: /configure byok/i }),
+ );
+
+ expect(
+ screen.queryByRole('button', { name: /list models from endpoint/i }),
+ ).not.toBeInTheDocument();
+
+ await userEvent.selectOptions(
+ screen.getByRole('combobox', { name: /provider/i }),
+ 'custom',
+ );
+
+ expect(
+ screen.getByRole('button', { name: /list models from endpoint/i }),
+ ).toBeInTheDocument();
+ });
+
+ it('lists what the server returned and saves the chosen id', async () => {
+ mockDiscoverLlmModels.mockResolvedValue({
+ outcome: 'ok',
+ detail: '2 chat model(s) of 5 listed.',
+ models: [{ id: 'acme/chat-large' }, { id: 'acme/chat-small', name: 'Chat Small' }],
+ listed: 5,
+ excluded: 3,
+ truncated: false,
+ });
+ await openCustomForm();
+
+ await userEvent.click(
+ screen.getByRole('button', { name: /list models from endpoint/i }),
+ );
+
+ expect(mockDiscoverLlmModels).toHaveBeenCalledWith({
+ base_url: 'https://gw.example.com/v1',
+ api_key: 'typed-key',
+ });
+ const picker = await screen.findByRole('combobox', {
+ name: /discovered chat models/i,
+ });
+ const offered = Array.from((picker as HTMLSelectElement).options)
+ .map(o => o.value)
+ .filter(Boolean);
+ expect(offered).toEqual(['acme/chat-large', 'acme/chat-small']);
+ expect(screen.getByRole('status')).toHaveTextContent(
+ /2 chat models of 5 listed .*3 .*non-chat entries hidden/i,
+ );
+
+ await userEvent.selectOptions(picker, 'acme/chat-small');
+ expect(screen.getByPlaceholderText('provider default')).toHaveValue(
+ 'acme/chat-small',
+ );
+
+ await userEvent.click(
+ screen.getByRole('button', { name: /save credential/i }),
+ );
+ await waitFor(() =>
+ expect(mockUpsertLlmCredential).toHaveBeenCalledWith(
+ expect.objectContaining({
+ provider: 'custom',
+ base_url: 'https://gw.example.com/v1',
+ model: 'acme/chat-small',
+ }),
+ ),
+ );
+ });
+
+ it('keeps the typed id working when discovery fails', async () => {
+ mockDiscoverLlmModels.mockResolvedValue({
+ outcome: 'refused',
+ detail: 'The endpoint does not publish a model list at /models (HTTP 404).',
+ models: [],
+ listed: 0,
+ excluded: 0,
+ truncated: false,
+ });
+ await openCustomForm();
+
+ await userEvent.click(
+ screen.getByRole('button', { name: /list models from endpoint/i }),
+ );
+
+ expect(await screen.findByRole('status')).toHaveTextContent(
+ /HTTP 404.*still accepts a typed id/i,
+ );
+ expect(
+ screen.queryByRole('combobox', { name: /discovered chat models/i }),
+ ).not.toBeInTheDocument();
+
+ await userEvent.type(
+ screen.getByPlaceholderText('provider default'),
+ 'hand-typed-model',
+ );
+ await userEvent.click(
+ screen.getByRole('button', { name: /save credential/i }),
+ );
+ await waitFor(() =>
+ expect(mockUpsertLlmCredential).toHaveBeenCalledWith(
+ expect.objectContaining({ model: 'hand-typed-model' }),
+ ),
+ );
+ });
+
+ it('drops a list fetched for one base URL when the base URL changes', async () => {
+ mockDiscoverLlmModels.mockResolvedValue({
+ outcome: 'ok',
+ detail: '1 chat model(s) of 1 listed.',
+ models: [{ id: 'acme/chat-large' }],
+ listed: 1,
+ excluded: 0,
+ truncated: false,
+ });
+ await openCustomForm();
+ await userEvent.click(
+ screen.getByRole('button', { name: /list models from endpoint/i }),
+ );
+ expect(
+ await screen.findByRole('combobox', { name: /discovered chat models/i }),
+ ).toBeInTheDocument();
+
+ await userEvent.type(
+ screen.getByPlaceholderText('https://internal-llm.example.com/v1'),
+ '2',
+ );
+
+ expect(
+ screen.queryByRole('combobox', { name: /discovered chat models/i }),
+ ).not.toBeInTheDocument();
+ });
+});
diff --git a/apps/web/src/components/settings/SettingsView.tsx b/apps/web/src/components/settings/SettingsView.tsx
index 158b80f39..83625db7a 100644
--- a/apps/web/src/components/settings/SettingsView.tsx
+++ b/apps/web/src/components/settings/SettingsView.tsx
@@ -37,6 +37,7 @@ import {
type FullTenant,
type LlmCredentialUpsert,
type LlmCredentialView,
+ type LlmModelDiscoveryResult,
type LlmProvider,
type LlmStatus,
type LlmWritableProvider,
@@ -1918,6 +1919,14 @@ function BYOKCard({
const [apiKey, setApiKey] = useState('');
const [enabled, setEnabled] = useState(true);
const [formError, setFormError] = useState<string | null>(null);
+ const [discovering, setDiscovering] = useState(false);
+ // Remembers which base URL the list came from: a list fetched for one
+ // endpoint says nothing about another, so it is shown only while that base
+ // URL is still the one in the form.
+ const [discovery, setDiscovery] = useState<{
+ baseUrl: string;
+ result: LlmModelDiscoveryResult;
+ } | null>(null);
// Reset form whenever we enter edit mode (or the underlying credential
// moves while editing). Never preload the API key — it lives only in
@@ -1994,6 +2003,42 @@ function BYOKCard({
setEditing(false);
};
+ // Ask the custom endpoint which chat models it serves. Uses what is in the
+ // form, so it works before the credential is saved; a blank key means "use
+ // the stored one", which the server sends only to the stored base URL. On
+ // any failure the list stays empty and the typed model id is what is saved.
+ const onDiscoverClick = async () => {
+ const target = baseUrl.trim();
+ setDiscovering(true);
+ try {
+ const result = await deploymentApi.discoverLlmModels({
+ base_url: target ? target : null,
+ api_key: apiKey.trim() ? apiKey.trim() : null,
+ });
+ setDiscovery({ baseUrl: target, result });
+ } catch (err) {
+ setDiscovery({
+ baseUrl: target,
+ result: {
+ outcome: 'refused',
+ detail: extractApiDetail(err, 'Could not list models.'),
+ models: [],
+ listed: 0,
+ excluded: 0,
+ truncated: false,
+ },
+ });
+ } finally {
+ setDiscovering(false);
+ }
+ };
+
+ const shownDiscovery =
+ provider === 'custom' && discovery && discovery.baseUrl === baseUrl.trim()
+ ? discovery.result
+ : null;
+ const discoveredModels = shownDiscovery?.models ?? [];
+
// One real call with the saved credential. Reports four outcomes, and
// `unverified` is deliberately not an error: an air-gapped deployment
// refusing the egress is the posture working, and a red toast there would
@@ -2302,6 +2347,67 @@ function BYOKCard({
/>
</Field>
</div>
+ {provider === 'custom' ? (
+ <div className="mt-4 flex flex-col gap-2">
+ <div className="flex flex-col gap-2 sm:flex-row sm:items-center">
+ <button
+ type="button"
+ onClick={onDiscoverClick}
+ disabled={discovering || submitting || !baseUrl.trim()}
+ className="shrink-0 rounded-md border border-gray-700 bg-gray-900 px-3 py-1.5 text-xs font-medium text-gray-200 hover:bg-gray-800 disabled:opacity-50"
+ >
+ {discovering ? 'Listing models…' : 'List models from endpoint'}
+ </button>
+ {discoveredModels.length > 0 ? (
+ <select
+ aria-label="Discovered chat models"
+ value={discoveredModels.some(m => m.id === model) ? model : ''}
+ onChange={e => {
+ if (e.target.value) setModel(e.target.value);
+ }}
+ disabled={submitting}
+ className={inputClass()}
+ >
+ <option value="">Choose a discovered model…</option>
+ {discoveredModels.map(m => (
+ <option key={m.id} value={m.id}>
+ {m.name && m.name !== m.id ? `${m.id} — ${m.name}` : m.id}
+ </option>
+ ))}
+ </select>
+ ) : null}
+ </div>
+ {shownDiscovery ? (
+ <p
+ role="status"
+ className={clsx(
+ 'text-xs',
+ shownDiscovery.outcome === 'ok' && discoveredModels.length > 0
+ ? 'text-gray-400'
+ : 'text-amber-300/80',
+ )}
+ >
+ {shownDiscovery.outcome === 'ok' && discoveredModels.length > 0
+ ? `${discoveredModels.length} chat model${
+ discoveredModels.length === 1 ? '' : 's'
+ } of ${shownDiscovery.listed} listed by the endpoint${
+ shownDiscovery.excluded > 0
+ ? ` (${shownDiscovery.excluded} image, audio, embedding or other non-chat entries hidden)`
+ : ''
+ }.${
+ shownDiscovery.truncated ? ' Only the first part of a longer list is shown.' : ''
+ } Pick one, or type any id in the Model field.`
+ : `${shownDiscovery.detail} The Model field still accepts a typed id.`}
+ </p>
+ ) : (
+ <p className="text-xs text-gray-500">
+ Asks the endpoint&apos;s <code className="font-mono">/models</code>{' '}
+ for the chat models it serves. Optional — a typed model id
+ always works.
+ </p>
+ )}
+ </div>
+ ) : null}
<div className="mt-4">
<Toggle
checked={enabled}
diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts
index 41217da47..bacface38 100644
--- a/apps/web/src/lib/api.ts
+++ b/apps/web/src/lib/api.ts
@@ -6438,6 +6438,23 @@ export interface LlmCredentialTestResult {
latency_ms?: number;
}
+/**
+ * The chat models a custom OpenAI-compatible endpoint listed at ``/models``.
+ *
+ * ``listed`` is every distinct id the endpoint returned and ``excluded`` the
+ * ones judged not chat-capable (image, audio, embedding and similar), so the
+ * selector can say how much it filtered. Any outcome other than ``ok`` comes
+ * with an empty ``models``: the hand-typed id is the fallback, never a guess.
+ */
+export interface LlmModelDiscoveryResult {
+ outcome: "ok" | "refused" | "unreachable" | "unverified";
+ detail: string;
+ models: { id: string; name?: string | null }[];
+ listed: number;
+ excluded: number;
+ truncated: boolean;
+}
+
export const deploymentApi = {
/** Live air-gap policy snapshot for this pod. Safe to poll. */
getAirgapStatus: () => request<AirgapStatus>("/api/v1/airgap/status"),
@@ -6473,6 +6490,19 @@ export const deploymentApi = {
request<LlmCredentialTestResult>("/api/v1/llm/credentials/test", {
method: "POST",
}),
+
+ /**
+ * List the chat models a ``custom`` endpoint serves.
+ *
+ * Works before the form is saved: ``base_url`` and ``api_key`` are used for
+ * this one request and never stored. Omit ``api_key`` to use the stored key,
+ * which the server sends only to the stored base URL.
+ */
+ discoverLlmModels: (payload: { base_url?: string | null; api_key?: string | null }) =>
+ request<LlmModelDiscoveryResult>("/api/v1/llm/credentials/models", {
+ method: "POST",
+ body: JSON.stringify(payload),
+ }),
};
// ─── Reports / Executive digest (WS-G2) ─────────────────────────────────────
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index 2de81fae7..382b89a42 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -19882,6 +19882,53 @@ paths:
$ref: '#/components/schemas/LlmCredentialTestResult'
security:
- HTTPBearer: []
+ /api/v1/llm/credentials/models:
+ post:
+ tags:
+ - llm
+ summary: List the chat models a custom OpenAI-compatible endpoint serves
+ description: 'Ask ``GET {base_url}/models`` and return the chat-capable entries.
+
+
+ For the ``custom`` provider, whose model was otherwise a hand-typed id.
+
+
+ `settings:write`, like `/test`: this dials an operator-supplied URL with
+
+ private addresses permitted, which is not something a read-only role
+
+ should be able to point at the network.
+
+
+ The stored key goes only to the stored base URL. A form that has changed
+
+ the base URL and left the key blank gets a keyless request, not the saved
+
+ key sent somewhere it was never configured to go; an endpoint that needs a
+
+ key then answers 401 and the detail says so.'
+ operationId: discover_llm_models_api_v1_llm_credentials_models_post
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/LlmModelDiscoveryRequest'
+ required: true
+ responses:
+ '200':
+ description: Successful Response
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/LlmModelDiscoveryResult'
+ '422':
+ description: Validation Error
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/HTTPValidationError'
+ security:
+ - HTTPBearer: []
/api/v1/threatintel/stix/indicators:
get:
tags:
@@ -28138,6 +28185,20 @@ components:
required:
- discovered
title: DiscoverResponse
+ DiscoveredModelView:
+ properties:
+ id:
+ type: string
+ title: Id
+ name:
+ anyOf:
+ - type: string
+ - type: 'null'
+ title: Name
+ type: object
+ required:
+ - id
+ title: DiscoveredModelView
DismissTuningRequest:
properties:
reason:
@@ -32475,6 +32536,76 @@ components:
only signal we expose — same convention the OAuth credential view
uses (``has_secret``).'
+ LlmModelDiscoveryRequest:
+ properties:
+ base_url:
+ anyOf:
+ - type: string
+ maxLength: 512
+ - type: 'null'
+ title: Base Url
+ api_key:
+ anyOf:
+ - type: string
+ maxLength: 4096
+ - type: 'null'
+ title: Api Key
+ type: object
+ title: LlmModelDiscoveryRequest
+ description: 'Where to list models from, for a form that may not be saved yet.
+
+
+ Both fields are optional. ``base_url`` falls back to the stored ``custom``
+
+ credential''s. ``api_key`` is used for this one request and never stored;
+
+ when it is omitted the stored key is used, but only if the target is the
+
+ stored base URL -- see :func:`discover_llm_models`.'
+ LlmModelDiscoveryResult:
+ properties:
+ outcome:
+ type: string
+ enum:
+ - ok
+ - refused
+ - unreachable
+ - unverified
+ title: Outcome
+ detail:
+ type: string
+ title: Detail
+ models:
+ items:
+ $ref: '#/components/schemas/DiscoveredModelView'
+ type: array
+ title: Models
+ listed:
+ type: integer
+ title: Listed
+ excluded:
+ type: integer
+ title: Excluded
+ truncated:
+ type: boolean
+ title: Truncated
+ type: object
+ required:
+ - outcome
+ - detail
+ - models
+ - listed
+ - excluded
+ - truncated
+ title: LlmModelDiscoveryResult
+ description: 'The chat models an endpoint listed, or why there are none.
+
+
+ ``outcome`` uses the Test button''s vocabulary. Anything other than ``ok``
+
+ leaves ``models`` empty and the console falls back to the hand-typed id,
+
+ which is the path that always worked; nothing is invented to fill the gap.'
LoginRequest:
properties:
email:
diff --git a/services/api/app/api/v1/endpoints/llm_credentials.py b/services/api/app/api/v1/endpoints/llm_credentials.py
index 36f6143c7..ede1cf326 100644
--- a/services/api/app/api/v1/endpoints/llm_credentials.py
+++ b/services/api/app/api/v1/endpoints/llm_credentials.py
@@ -19,6 +19,14 @@ Endpoint contract
Hard-delete the row. The platform falls back to env-var configuration
on the next request (the existing v1 behaviour).
+* ``POST /api/v1/llm/credentials/test``
+ Place one minimal completion with the stored credential.
+
+* ``POST /api/v1/llm/credentials/models``
+ List the chat models a ``custom`` endpoint serves, so the console can
+ offer them instead of requiring a hand-typed id. See
+ :mod:`app.services.llm_model_discovery`.
+
Why this lives in services/api
------------------------------
@@ -62,6 +70,7 @@ from app.models.llm_credential import TenantLlmCredential
from app.security.credential_vault import CredentialVaultError, get_vault
from app.services.audit import emit_audit
from app.services.llm_credential_probe import ProbeResult, probe_credential
+from app.services.llm_model_discovery import Discovery, discover_chat_models
logger = logging.getLogger(__name__)
@@ -98,6 +107,31 @@ _PROVIDERS_REQUIRING_KEY: frozenset[str] = frozenset({"openai", "anthropic", "az
# --------------------------------------------------------------- request / response models
+def _check_base_url(v: str | None) -> str | None:
+ if v is None:
+ return v
+ v = v.strip()
+ if not v:
+ return None
+ # Light shape check — we don't want to over-validate (operators
+ # legitimately run weird proxies on weird ports inside private
+ # networks) but we should reject obvious garbage so the UI gets
+ # a useful 422 instead of a 500 at request time.
+ parsed = urlparse(v)
+ if parsed.scheme not in {"http", "https"}:
+ raise ValueError("base_url must be an http(s) URL")
+ if not parsed.hostname:
+ raise ValueError("base_url must include a hostname")
+ return v
+
+
+def _blank_to_none(v: str | None) -> str | None:
+ if v is None:
+ return v
+ v = v.strip()
+ return v or None
+
+
class LlmCredentialUpsert(BaseModel):
"""Payload for ``PUT /api/v1/llm/credentials``.
@@ -119,29 +153,12 @@ class LlmCredentialUpsert(BaseModel):
@field_validator("base_url")
@classmethod
def _validate_base_url(cls, v: str | None) -> str | None:
- if v is None:
- return v
- v = v.strip()
- if not v:
- return None
- # Light shape check — we don't want to over-validate (operators
- # legitimately run weird proxies on weird ports inside private
- # networks) but we should reject obvious garbage so the UI gets
- # a useful 422 instead of a 500 at request time.
- parsed = urlparse(v)
- if parsed.scheme not in {"http", "https"}:
- raise ValueError("base_url must be an http(s) URL")
- if not parsed.hostname:
- raise ValueError("base_url must include a hostname")
- return v
+ return _check_base_url(v)
@field_validator("api_key")
@classmethod
def _validate_api_key(cls, v: str | None) -> str | None:
- if v is None:
- return v
- v = v.strip()
- return v or None
+ return _blank_to_none(v)
class LlmCredentialView(BaseModel):
@@ -521,3 +538,125 @@ async def test_llm_credential(
model=result.model,
latency_ms=result.latency_ms,
)
+
+
+# --------------------------------------------------------- model discovery
+
+
+class LlmModelDiscoveryRequest(BaseModel):
+ """Where to list models from, for a form that may not be saved yet.
+
+ Both fields are optional. ``base_url`` falls back to the stored ``custom``
+ credential's. ``api_key`` is used for this one request and never stored;
+ when it is omitted the stored key is used, but only if the target is the
+ stored base URL -- see :func:`discover_llm_models`.
+ """
+
+ base_url: str | None = Field(default=None, max_length=512)
+ api_key: str | None = Field(default=None, max_length=4096)
+
+ @field_validator("base_url")
+ @classmethod
+ def _validate_base_url(cls, v: str | None) -> str | None:
+ return _check_base_url(v)
+
+ @field_validator("api_key")
+ @classmethod
+ def _validate_api_key(cls, v: str | None) -> str | None:
+ return _blank_to_none(v)
+
+
+class DiscoveredModelView(BaseModel):
+ id: str
+ name: str | None = None
+
+
+class LlmModelDiscoveryResult(BaseModel):
+ """The chat models an endpoint listed, or why there are none.
+
+ ``outcome`` uses the Test button's vocabulary. Anything other than ``ok``
+ leaves ``models`` empty and the console falls back to the hand-typed id,
+ which is the path that always worked; nothing is invented to fill the gap.
+ """
+
+ outcome: Literal["ok", "refused", "unreachable", "unverified"]
+ detail: str
+ models: list[DiscoveredModelView]
+ listed: int
+ excluded: int
+ truncated: bool
+
+
+def _same_endpoint(a: str | None, b: str | None) -> bool:
+ return bool(a and b) and (a or "").strip().rstrip("/") == (b or "").strip().rstrip("/")
+
+
+ "/models",
+ response_model=LlmModelDiscoveryResult,
+ summary="List the chat models a custom OpenAI-compatible endpoint serves",
+)
+async def discover_llm_models(
+ payload: LlmModelDiscoveryRequest,
+ current_user: Annotated[AuthUser, Depends(require_permission("settings:write"))],
+ db: DBSession,
+) -> LlmModelDiscoveryResult:
+ """Ask ``GET {base_url}/models`` and return the chat-capable entries.
+
+ For the ``custom`` provider, whose model was otherwise a hand-typed id.
+
+ `settings:write`, like `/test`: this dials an operator-supplied URL with
+ private addresses permitted, which is not something a read-only role
+ should be able to point at the network.
+
+ The stored key goes only to the stored base URL. A form that has changed
+ the base URL and left the key blank gets a keyless request, not the saved
+ key sent somewhere it was never configured to go; an endpoint that needs a
+ key then answers 401 and the detail says so.
+ """
+ res = await db.execute(select(TenantLlmCredential).where(TenantLlmCredential.tenant_id == current_user.tenant_id))
+ existing: TenantLlmCredential | None = res.scalar_one_or_none()
+ stored = existing if existing is not None and existing.provider == "custom" else None
+
+ base_url = payload.base_url or (stored.base_url if stored else None)
+ if not base_url:
+ raise HTTPException(
+ status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+ detail="Model discovery needs a base URL: enter one, or save a custom credential first.",
+ )
+
+ api_key = payload.api_key
+ if api_key is None and stored is not None and stored.api_key_vault and _same_endpoint(stored.base_url, base_url):
+ try:
+ api_key = get_vault().decrypt(stored.api_key_vault)
+ except CredentialVaultError:
+ return LlmModelDiscoveryResult(
+ outcome="refused",
+ detail=(
+ "The stored key could not be decrypted. It was encrypted with a vault key "
+ "this deployment no longer has; re-enter it to fix."
+ ),
+ models=[],
+ listed=0,
+ excluded=0,
+ truncated=False,
+ )
+
+ result: Discovery = await discover_chat_models(base_url=base_url, api_key=api_key)
+ logger.info(
+ "settings.llm.discover",
+ extra={
+ "tenant": str(current_user.tenant_id),
+ "outcome": result.outcome,
+ "listed": result.listed,
+ "offered": len(result.models),
+ },
+ )
+ return LlmModelDiscoveryResult(
+ outcome=result.outcome,
+ detail=result.detail,
+ models=[DiscoveredModelView(id=m.id, name=m.name) for m in result.models],
+ listed=result.listed,
+ excluded=result.excluded,
+ truncated=result.truncated,
+ )
diff --git a/services/api/app/services/llm_model_discovery.py b/services/api/app/services/llm_model_discovery.py
new file mode 100644
index 000000000..510f8d148
--- /dev/null
+++ b/services/api/app/services/llm_model_discovery.py
@@ -0,0 +1,455 @@
+"""List the chat models an OpenAI-compatible endpoint serves, for the BYOK selector.
+
+Why this exists
+---------------
+The ``custom`` provider points a tenant at any OpenAI-compatible endpoint, and
+the model was a hand-typed id. Nothing checked it, so a typo or an id the
+endpoint does not serve surfaced as triage quietly falling back to the
+deterministic path. Most such endpoints publish ``GET {base}/models``; this asks
+it, with the tenant's key, and offers the chat-capable answers as choices. The
+hand-typed id stays: discovery is a convenience over it, never a gate on it.
+
+Bounds
+------
+The URL and the response are both tenant-controlled, so neither decides what
+this process spends:
+
+* :data:`DISCOVERY_TIMEOUT_SECONDS` is a deadline on the *whole* exchange, not
+ httpx's per-read timeout, so an endpoint that drips one byte every nine
+ seconds cannot hold a settings panel open;
+* :data:`DISCOVERY_MAX_BYTES` caps the body as it streams in, after content
+ decoding, so a compressed response cannot expand past it either;
+* :data:`DISCOVERY_MAX_ROWS` caps how many rows are examined and
+ :data:`DISCOVERY_MAX_MODELS` how many chat models are returned. They are
+ separate on purpose: an aggregating endpoint can list hundreds of image,
+ audio and embedding rows ahead of its chat models, and capping the raw rows
+ at the selector size would drop chat models for the sake of rows that were
+ going to be filtered out anyway.
+
+Redirects are not followed. A ``3xx`` is reported, not chased to a host the
+guard below never saw.
+
+What counts as chat-capable
+---------------------------
+AiSOC calls ``POST {base}/chat/completions`` and nothing else, so the question
+is whether an id can serve that. The OpenAI ``/models`` shape does not say: a
+row is ``{"id", "object", "created", "owned_by"}``. Endpoints that aggregate
+many model families often add a field that does, under one of a few spellings.
+Each row is judged on what it declares, strongest evidence first:
+
+1. **Declared metadata.** Three optional fields are read when present:
+
+ * ``capabilities`` -- an object with a boolean ``chat``,
+ ``completion_chat``, ``chat_completion`` or ``chat_completions`` key, or
+ a list naming one of those;
+ * modalities -- ``architecture.modality`` as ``"text+image->text"``,
+ ``architecture.input_modalities`` / ``output_modalities``, the same two
+ at the top level, or ``modalities`` as ``{"input": [...],
+ "output": [...]}``. Output without ``text``, or input without ``text``,
+ is not chat. Text in and text out is;
+ * a label -- ``type`` or ``mode``, split into words. A word naming a
+ non-text family (image, video, audio, speech, embedding, rerank,
+ moderation, ...) is not chat; ``chat``, ``llm``, ``vlm``, ``language``
+ or ``conversational`` is.
+
+ A declared negative wins over a declared positive: a row that is wrongly
+ hidden can still be typed by hand, and a row that is wrongly offered fails
+ at triage time, quietly.
+
+2. **A label the catalogue uses as a classifier.** Some endpoints list one row
+ per (model, endpoint) pair and name the endpoint in ``type`` -- a model that
+ only serves an asynchronous or non-chat route appears under a label that is
+ neither ``chat`` nor a known non-text family. When *any* row's label says
+ chat, the catalogue has shown how it spells chat, and a row whose label says
+ something else is not offered. When no row's label says chat, the field is
+ not a capability classifier in that catalogue (it may mean "base" vs
+ "fine-tuned") and it is ignored.
+
+3. **The id, when nothing was declared.** A plain OpenAI-compatible server
+ (vLLM, a local runtime, most proxies) declares nothing, and its rows are
+ chat models unless the id names a non-text family -- ``embed``,
+ ``embedding``, ``tts``, ``whisper``, ``transcribe``, ``image``, ``video``,
+ ``rerank``, ``moderation`` and the like, matched as whole words of the id.
+ Everything else is offered.
+
+An id appearing on several rows is offered once, if any of its rows is chat.
+The heuristic is a selector filter, not an authorisation: what an endpoint
+actually serves is decided by the endpoint, and the Test button remains the
+way to find out.
+
+SSRF and air-gap
+----------------
+The same two guards :mod:`app.services.llm_credential_probe` applies to this
+very base URL, in the same order: air-gap first, before anything is dialled,
+then ``validate_outbound_url(..., allow_private=True)`` so a private model host
+is reachable while loopback and link-local -- the cloud-metadata address --
+stay refused. See that module for why the webhook guard is the wrong shape.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import re
+from dataclasses import dataclass, field
+from typing import Any, Literal
+
+import httpx
+import structlog
+
+from app._vendor.ssrf_guard import SSRFError, validate_outbound_url
+from app.core.airgap import AirgapViolation, enforce_airgap_for_url
+from app.services.model_aliases import completions_url_for_base
+
+logger = structlog.get_logger(__name__)
+
+Outcome = Literal["ok", "refused", "unreachable", "unverified"]
+Verdict = Literal["chat", "not_chat", "unknown"]
+
+#: A human is waiting on this in a settings panel.
+DISCOVERY_TIMEOUT_SECONDS = 10.0
+#: A model list is a few hundred kilobytes even for an endpoint aggregating
+#: hundreds of models; two megabytes is generous and still bounded.
+DISCOVERY_MAX_BYTES = 2_000_000
+#: Rows examined. Bounds the work on a body that fits under the byte cap by
+#: being nothing but tiny rows.
+DISCOVERY_MAX_ROWS = 5_000
+#: Chat models returned. More than this is not a dropdown anyone can use, and
+#: the response says when it was reached.
+DISCOVERY_MAX_MODELS = 500
+
+_CHAT_COMPLETIONS_SUFFIX = "/chat/completions"
+
+_WORD = re.compile(r"[a-z0-9]+")
+
+#: Words in a ``type``/``mode`` label that mean the row serves chat.
+_CHAT_LABEL_WORDS = frozenset({"chat", "llm", "vlm", "language", "conversational"})
+
+#: Words that name a family which cannot answer a text chat completion. Used
+#: for labels and, when a row declares nothing, for the id.
+_NON_CHAT_WORDS = frozenset(
+ {
+ "audio",
+ "diffusion",
+ "embed",
+ "embedding",
+ "embeddings",
+ "image",
+ "images",
+ "moderation",
+ "moderations",
+ "music",
+ "ocr",
+ "realtime",
+ "rerank",
+ "reranker",
+ "reranking",
+ "speech",
+ "transcribe",
+ "transcription",
+ "transcriptions",
+ "tts",
+ "video",
+ "videos",
+ "whisper",
+ }
+)
+
+_CHAT_CAPABILITY_KEYS = ("chat", "completion_chat", "chat_completion", "chat_completions")
+_LABEL_FIELDS = ("type", "mode")
+
+
+@dataclass(frozen=True)
+class DiscoveredModel:
+ """What a selector needs. No pricing, no owner blob, nothing echoed raw."""
+
+ id: str
+ name: str | None = None
+
+
+@dataclass(frozen=True)
+class Discovery:
+ """Outcome of one discovery attempt.
+
+ ``listed`` counts the distinct ids the endpoint returned and ``excluded``
+ the ones judged not chat-capable, so the console can say "2 of 5"
+ rather than present a filtered list as the whole catalogue.
+ """
+
+ outcome: Outcome
+ detail: str
+ models: list[DiscoveredModel] = field(default_factory=list)
+ listed: int = 0
+ excluded: int = 0
+ truncated: bool = False
+
+
+class _DiscoveryError(Exception):
+ def __init__(self, outcome: Outcome, detail: str) -> None:
+ super().__init__(detail)
+ self.outcome: Outcome = outcome
+ self.detail = detail
+
+
+# --------------------------------------------------------------- the URL
+
+
+def models_url_for_base(base_url: str) -> str:
+ """``/models`` beside the chat-completions URL the runtime will call.
+
+ Derived from :func:`completions_url_for_base` rather than by appending to
+ the base, so a base given as ``https://host`` or as
+ ``https://host/v1/chat/completions`` lists models from the same place triage
+ sends completions to.
+ """
+ completions = completions_url_for_base(base_url)
+ return completions[: -len(_CHAT_COMPLETIONS_SUFFIX)] + "/models"
+
+
+# --------------------------------------------------------------- the filter
+
+
+def _words(value: str) -> set[str]:
+ return set(_WORD.findall(value.lower()))
+
+
+def _label(row: dict[str, Any]) -> str | None:
+ for key in _LABEL_FIELDS:
+ value = row.get(key)
+ if isinstance(value, str) and value.strip():
+ return value
+ return None
+
+
+def label_verdict(label: str) -> Verdict:
+ """Judge a ``type``/``mode`` label. A non-text word wins over a chat word."""
+ words = _words(label)
+ if words & _NON_CHAT_WORDS:
+ return "not_chat"
+ if words & _CHAT_LABEL_WORDS:
+ return "chat"
+ return "unknown"
+
+
+def _capabilities_verdict(row: dict[str, Any]) -> Verdict | None:
+ caps = row.get("capabilities")
+ if isinstance(caps, dict):
+ for key in _CHAT_CAPABILITY_KEYS:
+ if isinstance(caps.get(key), bool):
+ return "chat" if caps[key] else "not_chat"
+ return None
+ if isinstance(caps, list):
+ named = {item.strip().lower() for item in caps if isinstance(item, str)}
+ if named & set(_CHAT_CAPABILITY_KEYS):
+ return "chat"
+ return None
+
+
+def _modality_list(value: Any) -> list[str] | None:
+ if isinstance(value, list):
+ return [item.strip().lower() for item in value if isinstance(item, str)]
+ return None
+
+
+def _modalities_verdict(row: dict[str, Any]) -> Verdict | None:
+ raw_arch = row.get("architecture")
+ arch: dict[str, Any] = raw_arch if isinstance(raw_arch, dict) else {}
+ declared = row.get("modalities")
+
+ inputs = _modality_list(arch.get("input_modalities"))
+ if inputs is None:
+ inputs = _modality_list(row.get("input_modalities"))
+ outputs = _modality_list(arch.get("output_modalities"))
+ if outputs is None:
+ outputs = _modality_list(row.get("output_modalities"))
+ if isinstance(declared, dict):
+ if inputs is None:
+ inputs = _modality_list(declared.get("input"))
+ if outputs is None:
+ outputs = _modality_list(declared.get("output"))
+ elif inputs is None:
+ inputs = _modality_list(declared)
+
+ arrow = arch.get("modality") if isinstance(arch.get("modality"), str) else row.get("modality")
+ if isinstance(arrow, str) and "->" in arrow:
+ left, right = arrow.lower().split("->", 1)
+ if inputs is None:
+ inputs = [part.strip() for part in left.split("+") if part.strip()]
+ if outputs is None:
+ outputs = [part.strip() for part in right.split("+") if part.strip()]
+
+ if outputs is not None and "text" not in outputs:
+ return "not_chat"
+ if inputs is not None and "text" not in inputs:
+ return "not_chat"
+ if outputs is not None:
+ return "chat"
+ # Text input alone says nothing: an embedding model takes text too.
+ return None
+
+
+def row_verdict(row: dict[str, Any], *, labels_classify: bool) -> Verdict:
+ """Judge one row. See the module docstring for the order and why.
+
+ ``labels_classify`` is the catalogue-level fact from step 2: whether any
+ row's label said chat.
+ """
+ label = _label(row)
+ declared = [
+ _capabilities_verdict(row),
+ _modalities_verdict(row),
+ label_verdict(label) if label is not None else None,
+ ]
+ if "not_chat" in declared:
+ return "not_chat"
+ if "chat" in declared:
+ return "chat"
+ if label is not None and labels_classify:
+ return "not_chat"
+ model_id = row.get("id")
+ if isinstance(model_id, str) and _words(model_id) & _NON_CHAT_WORDS:
+ return "not_chat"
+ return "chat"
+
+
+def filter_chat_models(rows: list[Any]) -> Discovery:
+ """Apply the filter to a ``data`` list. Pure, so it is testable on fixtures."""
+ usable: list[dict[str, Any]] = []
+ for row in rows[:DISCOVERY_MAX_ROWS]:
+ if not isinstance(row, dict):
+ continue
+ model_id = row.get("id")
+ if isinstance(model_id, str) and model_id.strip() and len(model_id) <= 256:
+ usable.append(row)
+
+ labels_classify = any(label_verdict(label) == "chat" for label in (_label(row) for row in usable) if label is not None)
+
+ seen: dict[str, DiscoveredModel] = {}
+ all_ids: set[str] = set()
+ for row in usable:
+ model_id = row["id"].strip()
+ all_ids.add(model_id)
+ if model_id in seen or row_verdict(row, labels_classify=labels_classify) != "chat":
+ continue
+ name = row.get("name")
+ seen[model_id] = DiscoveredModel(id=model_id, name=name.strip()[:200] if isinstance(name, str) and name.strip() else None)
+
+ chat = sorted(seen.values(), key=lambda m: m.id)
+ truncated = len(chat) > DISCOVERY_MAX_MODELS or len(rows) > DISCOVERY_MAX_ROWS
+ chat = chat[:DISCOVERY_MAX_MODELS]
+ excluded = len(all_ids) - len(seen)
+
+ if not chat:
+ detail = f"The endpoint listed {len(all_ids)} model(s) and none looked chat-capable. Type the model id instead."
+ else:
+ detail = f"{len(chat)} chat model(s) of {len(all_ids)} listed."
+ if truncated:
+ detail += " The list was longer than this panel shows; type an id that is not offered."
+ return Discovery(outcome="ok", detail=detail, models=chat, listed=len(all_ids), excluded=excluded, truncated=truncated)
+
+
+# --------------------------------------------------------------- the call
+
+
+async def _read_bounded(client: httpx.AsyncClient, url: str, headers: dict[str, str]) -> bytes:
+ async with client.stream("GET", url, headers=headers) as response:
+ code = response.status_code
+ if code in (401, 403):
+ raise _DiscoveryError("refused", f"The endpoint rejected the key when listing models (HTTP {code}).")
+ if code == 404:
+ raise _DiscoveryError(
+ "refused",
+ "The endpoint does not publish a model list at /models (HTTP 404). Type the model id instead.",
+ )
+ if code != 200:
+ raise _DiscoveryError("refused", f"The endpoint answered HTTP {code} when listing models.")
+ chunks: list[bytes] = []
+ total = 0
+ async for chunk in response.aiter_bytes():
+ total += len(chunk)
+ if total > DISCOVERY_MAX_BYTES:
+ raise _DiscoveryError(
+ "refused",
+ f"The model list is larger than {DISCOVERY_MAX_BYTES // 1_000_000} MB, so it was not read. Type the model id instead.",
+ )
+ chunks.append(chunk)
+ return b"".join(chunks)
+
+
+async def discover_chat_models(
+ *,
+ base_url: str,
+ api_key: str | None,
+ transport: httpx.AsyncBaseTransport | None = None,
+) -> Discovery:
+ """List the endpoint's chat models. Never raises; every failure is a state.
+
+ ``transport`` exists for tests: they keep the real client, the real
+ streaming read and the real guards, and swap only what is on the wire.
+ """
+ base = (base_url or "").strip().rstrip("/")
+ if not base:
+ return Discovery(outcome="refused", detail="No base URL is configured, so there is nothing to list.")
+ url = models_url_for_base(base)
+
+ try:
+ enforce_airgap_for_url(url)
+ except AirgapViolation as exc:
+ return Discovery(
+ outcome="unverified",
+ detail=f"This deployment is air-gapped, so the model list was not requested: {exc}",
+ )
+
+ try:
+ validate_outbound_url(url, allow_private=True)
+ except SSRFError as exc:
+ return Discovery(outcome="refused", detail=f"The base URL was refused: {exc}")
+
+ headers = {"Accept": "application/json"}
+ if api_key:
+ headers["Authorization"] = f"Bearer {api_key}"
+
+ try:
+ async with asyncio.timeout(DISCOVERY_TIMEOUT_SECONDS):
+ async with httpx.AsyncClient(
+ timeout=DISCOVERY_TIMEOUT_SECONDS,
+ follow_redirects=False,
+ transport=transport,
+ ) as client:
+ raw = await _read_bounded(client, url, headers)
+ except _DiscoveryError as exc:
+ return Discovery(outcome=exc.outcome, detail=exc.detail)
+ except (TimeoutError, httpx.TimeoutException):
+ return Discovery(
+ outcome="unreachable",
+ detail=f"{base} did not return its model list within {DISCOVERY_TIMEOUT_SECONDS:.0f}s.",
+ )
+ except Exception as exc: # noqa: BLE001 - every failure is a state
+ logger.info("llm_discovery.unreachable", error=type(exc).__name__)
+ return Discovery(outcome="unreachable", detail=f"Could not reach {base} ({type(exc).__name__}).")
+
+ try:
+ body = json.loads(raw)
+ except ValueError:
+ return Discovery(outcome="refused", detail="The endpoint's model list was not JSON.")
+
+ rows = body.get("data") if isinstance(body, dict) else body
+ if not isinstance(rows, list):
+ return Discovery(outcome="refused", detail="The endpoint's answer had no `data` list of models.")
+
+ return filter_chat_models(rows)
+
+
+__all__ = [
+ "DISCOVERY_MAX_BYTES",
+ "DISCOVERY_MAX_MODELS",
+ "DISCOVERY_MAX_ROWS",
+ "DISCOVERY_TIMEOUT_SECONDS",
+ "DiscoveredModel",
+ "Discovery",
+ "discover_chat_models",
+ "filter_chat_models",
+ "label_verdict",
+ "models_url_for_base",
+ "row_verdict",
+]
diff --git a/services/api/tests/test_llm_model_discovery.py b/services/api/tests/test_llm_model_discovery.py
new file mode 100644
index 000000000..0e5b9481a
--- /dev/null
+++ b/services/api/tests/test_llm_model_discovery.py
@@ -0,0 +1,498 @@
+"""A custom endpoint's chat models can populate the selector, and nothing else can.
+
+What this closes
+----------------
+The ``custom`` BYOK provider points a tenant at any OpenAI-compatible endpoint,
+and the model was a hand-typed id. ``POST /api/v1/llm/credentials/models`` asks
+``GET {base}/models`` and returns the chat-capable entries.
+
+Three things are pinned here, and each has a way to look right while being
+wrong:
+
+* **The filter.** An aggregating endpoint lists image, video, audio and
+ embedding models beside its chat models, and its ``/models`` rows say so only
+ through optional fields. A filter that passed everything would still produce
+ a populated dropdown; the fixtures below assert the exact surviving set.
+* **The bounds.** The URL and the body are both tenant-controlled. Each bound is
+ driven past its limit through the real ``httpx`` client and streaming read --
+ only the bytes on the wire are swapped, via ``httpx.MockTransport``.
+* **The key.** The stored key goes only to the stored base URL. A form that
+ changed the base URL and left the key blank must not send the saved key to
+ the new host.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import functools
+import json
+import uuid
+from datetime import UTC, datetime
+from types import SimpleNamespace
+from typing import Any
+
+import httpx
+import pytest
+from app.api.v1.deps import CurrentUser
+from app.api.v1.endpoints import llm_credentials as llm_credentials_module
+from app.api.v1.endpoints.auth import get_current_user
+from app.db.database import get_db
+from app.security.credential_vault import get_vault
+from app.services import llm_model_discovery as discovery_mod
+from app.services.llm_model_discovery import (
+ DISCOVERY_MAX_BYTES,
+ DISCOVERY_MAX_MODELS,
+ discover_chat_models,
+ filter_chat_models,
+ models_url_for_base,
+)
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+#: A private literal: permitted by ``allow_private=True`` and needs no DNS.
+PRIVATE_BASE = "http://10.20.30.40:8000/v1"
+
+
[email protected](autouse=True)
+def _airgap_off(monkeypatch: pytest.MonkeyPatch) -> None:
+ # Patched where it is read, at call time -- see test_llm_credential_probe.
+ from app.core import airgap as airgap_mod
+
+ monkeypatch.setattr(airgap_mod.settings, "AISOC_AIRGAPPED", False, raising=False)
+
+
+def _ids(rows: list[Any]) -> set[str]:
+ return {m.id for m in filter_chat_models(rows).models}
+
+
+# --------------------------------------------------------------- fixtures
+
+#: Shaped like an aggregating gateway: one row per (model, endpoint) pair, the
+#: endpoint named in ``type``, and several model families in one list. Every
+#: name is invented.
+MIXED_GATEWAY_ROWS: list[dict[str, Any]] = [
+ {"id": "acme/chat-large", "type": "vendor-a/chat-completions"},
+ {"id": "acme/chat-large", "type": "vendor-a/responses"},
+ {"id": "acme/chat-small", "type": "vendor-a/chat-completions"},
+ # Served over a second chat protocol too; offered once.
+ {"id": "globex/reasoner-2", "type": "vendor-b/messages"},
+ {"id": "globex/reasoner-2", "type": "vendor-a/chat-completions"},
+ # A chat row whose id contains a non-text word: the declared type wins.
+ {"id": "acme/voice-audio-preview", "type": "vendor-a/chat-completions"},
+ # Only reachable through an asynchronous, non-chat route.
+ {"id": "acme/deep-thinker-pro", "type": "vendor-a/responses/submit"},
+ {"id": "initech/decider-1", "type": "internal/decisions"},
+ {"id": "acme/pixel-3", "type": "vendor-a/image-generations"},
+ {"id": "acme/pixel-3-edit", "type": "vendor-a/image-editing"},
+ {"id": "umbrella/clipmaker", "type": "internal/video-generations/submit"},
+ {"id": "acme/vector-small", "type": "vendor-a/embeddings"},
+ {"id": "hooli/narrator", "type": "internal/text-to-speech"},
+ {"id": "hooli/scribe", "type": "internal/speech-to-text/submit"},
+ {"id": "hooli/composer", "type": "internal/audio-generations/submit"},
+ {"id": "initech/page-reader", "type": "internal/optical-character-recognition"},
+ # A multi-purpose router listed under every route it accepts.
+ {"id": "globex/router", "type": "vendor-a/image-generations"},
+ {"id": "globex/router", "type": "vendor-a/chat-completions"},
+]
+
+MIXED_GATEWAY_CHAT = {"acme/chat-large", "acme/chat-small", "globex/reasoner-2", "acme/voice-audio-preview", "globex/router"}
+
+
+class TestTheFilter:
+ def test_a_mixed_gateway_list_keeps_exactly_its_chat_models(self) -> None:
+ result = filter_chat_models(MIXED_GATEWAY_ROWS)
+
+ assert {m.id for m in result.models} == MIXED_GATEWAY_CHAT
+ assert result.listed == 15
+ assert result.excluded == 10
+ assert [m.id for m in result.models] == sorted(MIXED_GATEWAY_CHAT), "offered once each, in a stable order"
+
+ def test_a_plain_openai_shaped_list_is_judged_by_the_id(self) -> None:
+ """A self-hosted server declares nothing. Its rows are chat models
+ unless the id names a non-text family, as a whole word."""
+ rows = [
+ {"id": i, "object": "model", "owned_by": "local"}
+ for i in (
+ "example-chat-7b",
+ "example-instruct-70b:q4",
+ "example-embed-large",
+ "text-embedding-small",
+ "example-tts-1",
+ "example-whisper-small",
+ "example-image-xl",
+ "example-rerank-base",
+ "imagery-analyst-13b",
+ )
+ ]
+
+ assert _ids(rows) == {"example-chat-7b", "example-instruct-70b:q4", "imagery-analyst-13b"}
+
+ def test_declared_modalities_are_honoured(self) -> None:
+ rows = [
+ {"id": "a/text", "architecture": {"modality": "text->text"}},
+ {"id": "a/vision", "architecture": {"modality": "text+image->text"}},
+ {"id": "a/painter", "architecture": {"modality": "text->image"}},
+ {"id": "a/listener", "architecture": {"input_modalities": ["audio"], "output_modalities": ["text"]}},
+ {"id": "a/both", "architecture": {"input_modalities": ["text"], "output_modalities": ["text", "image"]}},
+ {"id": "a/split", "modalities": {"input": ["text"], "output": ["audio"]}},
+ {"id": "a/top", "input_modalities": ["text"], "output_modalities": ["text"]},
+ ]
+
+ assert _ids(rows) == {"a/text", "a/vision", "a/both", "a/top"}
+
+ def test_declared_capabilities_are_honoured(self) -> None:
+ rows = [
+ {"id": "a/one", "capabilities": {"completion_chat": True, "vision": False}},
+ {"id": "a/two", "capabilities": {"completion_chat": False}},
+ {"id": "a/three", "capabilities": ["chat", "tools"]},
+ {"id": "a/four-embed", "capabilities": {"function_calling": True}},
+ ]
+
+ # a/four-embed declares nothing about chat, so its id decides.
+ assert _ids(rows) == {"a/one", "a/three"}
+
+ def test_a_declared_negative_beats_a_declared_positive(self) -> None:
+ """Wrongly hidden can still be typed; wrongly offered fails at triage."""
+ rows = [{"id": "a/x", "type": "chat", "architecture": {"modality": "text->image"}}]
+
+ assert _ids(rows) == set()
+
+ def test_a_label_that_never_says_chat_is_not_a_classifier(self) -> None:
+ """Some endpoints use ``type`` for "base" vs "fine-tuned". Reading
+ that as "not chat" would empty the list."""
+ rows = [
+ {"id": "a/base-model", "type": "base"},
+ {"id": "a/tuned", "type": "fine-tuned"},
+ {"id": "a/embed-v1", "type": "base"},
+ ]
+
+ assert _ids(rows) == {"a/base-model", "a/tuned"}
+
+ def test_rows_without_a_usable_id_are_skipped(self) -> None:
+ rows: list[Any] = ["a-string", None, {"id": ""}, {"id": 7}, {"name": "x"}, {"id": "ok/model"}]
+
+ result = filter_chat_models(rows)
+
+ assert [m.id for m in result.models] == ["ok/model"]
+ assert result.listed == 1
+
+ def test_nothing_chat_capable_says_so_and_points_at_the_typed_id(self) -> None:
+ result = filter_chat_models([{"id": "a/embed", "type": "embeddings"}])
+
+ assert result.outcome == "ok"
+ assert result.models == []
+ assert "type the model id" in result.detail.lower()
+
+
+class TestTheUrl:
+ @pytest.mark.parametrize(
+ ("base", "expected"),
+ [
+ ("https://gw.example.com/v1", "https://gw.example.com/v1/models"),
+ ("https://gw.example.com/v1/", "https://gw.example.com/v1/models"),
+ ("https://gw.example.com", "https://gw.example.com/v1/models"),
+ ("https://gw.example.com/v1/chat/completions", "https://gw.example.com/v1/models"),
+ ],
+ )
+ def test_models_is_listed_beside_the_completions_url_triage_calls(self, base: str, expected: str) -> None:
+ assert models_url_for_base(base) == expected
+
+
+# --------------------------------------------------------------- the call
+
+
+class _Recorder:
+ """A ``MockTransport`` handler that records what reached the wire."""
+
+ def __init__(self, response: httpx.Response | None = None, *, delay: float = 0.0) -> None:
+ self.response = response or httpx.Response(200, json={"object": "list", "data": MIXED_GATEWAY_ROWS})
+ self.delay = delay
+ self.requests: list[httpx.Request] = []
+
+ async def __call__(self, request: httpx.Request) -> httpx.Response:
+ self.requests.append(request)
+ if self.delay:
+ await asyncio.sleep(self.delay)
+ return self.response
+
+ @property
+ def transport(self) -> httpx.MockTransport:
+ return httpx.MockTransport(self)
+
+
+class TestTheCall:
+ @pytest.mark.asyncio
+ async def test_it_asks_models_with_the_key_and_returns_only_chat(self) -> None:
+ wire = _Recorder()
+
+ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k-123", transport=wire.transport)
+
+ assert result.outcome == "ok", result.detail
+ assert {m.id for m in result.models} == MIXED_GATEWAY_CHAT
+ [request] = wire.requests
+ assert request.method == "GET"
+ assert str(request.url) == f"{PRIVATE_BASE}/models"
+ assert request.headers["authorization"] == "Bearer k-123"
+
+ @pytest.mark.asyncio
+ async def test_no_key_means_no_authorization_header(self) -> None:
+ wire = _Recorder()
+
+ await discover_chat_models(base_url=PRIVATE_BASE, api_key=None, transport=wire.transport)
+
+ assert "authorization" not in wire.requests[0].headers
+
+ @pytest.mark.parametrize(
+ ("status_code", "needle"),
+ [(401, "rejected the key"), (403, "rejected the key"), (404, "type the model id"), (500, "http 500")],
+ )
+ @pytest.mark.asyncio
+ async def test_each_refusal_is_a_reported_state(self, status_code: int, needle: str) -> None:
+ wire = _Recorder(httpx.Response(status_code, text="nope"))
+
+ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)
+
+ assert result.outcome == "refused"
+ assert needle in result.detail.lower()
+ assert result.models == []
+
+ @pytest.mark.asyncio
+ async def test_a_redirect_is_reported_not_followed(self) -> None:
+ """Following it would dial a host the SSRF guard never saw."""
+ wire = _Recorder(httpx.Response(302, headers={"location": "http://169.254.169.254/latest/meta-data/"}))
+
+ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)
+
+ assert result.outcome == "refused"
+ assert len(wire.requests) == 1
+
+ @pytest.mark.parametrize("body", [b"<html>not json</html>", b'{"models": []}', b'"a string"'])
+ @pytest.mark.asyncio
+ async def test_a_body_that_is_not_a_model_list_is_refused(self, body: bytes) -> None:
+ wire = _Recorder(httpx.Response(200, content=body))
+
+ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)
+
+ assert result.outcome == "refused"
+ assert result.models == []
+
+
+class TestTheBounds:
+ @pytest.mark.asyncio
+ async def test_a_body_over_the_byte_cap_is_not_read(self) -> None:
+ rows = [{"id": f"m/{i:07d}", "type": "chat", "pad": "x" * 200} for i in range(DISCOVERY_MAX_BYTES // 200 + 10)]
+ wire = _Recorder(httpx.Response(200, content=json.dumps({"data": rows}).encode()))
+
+ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)
+
+ assert result.outcome == "refused"
+ assert "larger than" in result.detail
+ assert result.models == []
+
+ @pytest.mark.asyncio
+ async def test_the_selector_is_capped_and_says_so(self) -> None:
+ rows = [{"id": f"m/{i:04d}", "type": "chat"} for i in range(DISCOVERY_MAX_MODELS + 25)]
+ rows += [{"id": f"img/{i:04d}", "type": "image"} for i in range(50)]
+ wire = _Recorder(httpx.Response(200, json={"data": rows}))
+
+ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)
+
+ assert result.outcome == "ok"
+ assert len(result.models) == DISCOVERY_MAX_MODELS
+ assert result.truncated is True
+ assert result.listed == DISCOVERY_MAX_MODELS + 75
+
+ @pytest.mark.asyncio
+ async def test_the_raw_row_count_does_not_cut_chat_models(self) -> None:
+ """Non-chat rows ahead of the chat ones must not use up the selector's
+ budget: capping raw rows at the selector size would drop these."""
+ rows = [{"id": f"img/{i:04d}", "type": "image"} for i in range(DISCOVERY_MAX_MODELS + 100)]
+ rows += [{"id": "acme/chat-late", "type": "chat"}]
+ wire = _Recorder(httpx.Response(200, json={"data": rows}))
+
+ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)
+
+ assert [m.id for m in result.models] == ["acme/chat-late"]
+ assert result.truncated is False
+
+ @pytest.mark.asyncio
+ async def test_the_deadline_covers_the_whole_exchange(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(discovery_mod, "DISCOVERY_TIMEOUT_SECONDS", 0.2)
+ wire = _Recorder(delay=2.0)
+
+ result = await discover_chat_models(base_url=PRIVATE_BASE, api_key="k", transport=wire.transport)
+
+ assert result.outcome == "unreachable"
+ assert "did not return" in result.detail
+
+
+class TestTheGuards:
+ @pytest.mark.parametrize(
+ "base",
+ ["http://169.254.169.254/v1", "http://127.0.0.1:8000/v1", "http://[::1]:8000/v1"],
+ )
+ @pytest.mark.asyncio
+ async def test_metadata_and_loopback_are_refused_before_anything_is_dialled(self, base: str) -> None:
+ wire = _Recorder()
+
+ result = await discover_chat_models(base_url=base, api_key="k", transport=wire.transport)
+
+ assert result.outcome == "refused"
+ assert "refused" in result.detail.lower()
+ assert wire.requests == []
+
+ @pytest.mark.asyncio
+ async def test_air_gap_reports_unverified_and_dials_nothing(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ from app.core import airgap as airgap_mod
+
+ monkeypatch.setattr(airgap_mod.settings, "AISOC_AIRGAPPED", True, raising=False)
+ wire = _Recorder()
+
+ result = await discover_chat_models(base_url="https://gw.example.com/v1", api_key="k", transport=wire.transport)
+
+ assert result.outcome == "unverified"
+ assert wire.requests == []
+
+
+# --------------------------------------------------------------- the route
+
+
+class _Session:
+ """Answers the endpoint's one SELECT with a fixed row."""
+
+ def __init__(self, row: Any) -> None:
+ self.row = row
+
+ async def execute(self, *_: Any, **__: Any) -> Any:
+ return SimpleNamespace(scalar_one_or_none=lambda: self.row)
+
+
+def _stored(provider: str = "custom", base_url: str = PRIVATE_BASE, key: str | None = "stored-key") -> SimpleNamespace:
+ now = datetime.now(UTC)
+ return SimpleNamespace(
+ tenant_id=uuid.uuid4(),
+ provider=provider,
+ base_url=base_url,
+ model=None,
+ api_key_vault=get_vault().encrypt(key) if key else None,
+ settings={},
+ enabled=True,
+ created_at=now,
+ updated_at=now,
+ last_rotated_at=now,
+ )
+
+
+def route(monkeypatch: pytest.MonkeyPatch):
+ """The real router, the real permission dependency and the real discovery
+ function; only the database row and the bytes on the wire are supplied."""
+ wire = _Recorder()
+ monkeypatch.setattr(
+ llm_credentials_module,
+ "discover_chat_models",
+ functools.partial(discover_chat_models, transport=wire.transport),
+ )
+ app = FastAPI()
+ app.include_router(llm_credentials_module.router, prefix="/api/v1")
+ state: dict[str, Any] = {"row": None, "scopes": None}
+ app.dependency_overrides[get_db] = lambda: _Session(state["row"])
+ app.dependency_overrides[get_current_user] = lambda: CurrentUser(
+ user_id=uuid.uuid4(),
+ tenant_id=uuid.uuid4(),
+ role="tenant_admin",
+ scopes=state["scopes"],
+ )
+ return TestClient(app), wire, state
+
+
+class TestTheRoute:
+ def test_it_is_published_where_the_console_asks_for_it(self) -> None:
+ from pathlib import Path
+
+ from app.main import app
+
+ client = (Path(__file__).resolve().parents[3] / "apps" / "web" / "src" / "lib" / "api.ts").read_text(encoding="utf-8")
+
+ assert "/api/v1/llm/credentials/models" in app.openapi()["paths"]
+ assert "/api/v1/llm/credentials/models" in client
+
+ def test_an_unsaved_form_lists_with_the_typed_key(self, route: Any) -> None:
+ client, wire, _ = route
+
+ response = client.post("/api/v1/llm/credentials/models", json={"base_url": PRIVATE_BASE, "api_key": "typed-key"})
+
+ assert response.status_code == 200, response.text
+ body = response.json()
+ assert body["outcome"] == "ok"
+ assert {m["id"] for m in body["models"]} == MIXED_GATEWAY_CHAT
+ assert (body["listed"], body["excluded"]) == (15, 10)
+ assert wire.requests[0].headers["authorization"] == "Bearer typed-key"
+
+ def test_a_saved_credential_lists_with_the_stored_key(self, route: Any) -> None:
+ client, wire, state = route
+ state["row"] = _stored()
+
+ response = client.post("/api/v1/llm/credentials/models", json={})
+
+ assert response.status_code == 200, response.text
+ assert str(wire.requests[0].url) == f"{PRIVATE_BASE}/models"
+ assert wire.requests[0].headers["authorization"] == "Bearer stored-key"
+
+ def test_the_stored_key_is_not_sent_to_a_different_base_url(self, route: Any) -> None:
+ """The form changed the base URL and left the key blank. Sending the
+ saved key there would hand it to a host it was never configured for."""
+ client, wire, state = route
+ state["row"] = _stored()
+
+ response = client.post("/api/v1/llm/credentials/models", json={"base_url": "http://10.99.99.99:9000/v1"})
+
+ assert response.status_code == 200, response.text
+ assert str(wire.requests[0].url) == "http://10.99.99.99:9000/v1/models"
+ assert "authorization" not in wire.requests[0].headers
+
+ def test_another_providers_key_is_never_used(self, route: Any) -> None:
+ client, wire, state = route
+ state["row"] = _stored(provider="local-litellm")
+
+ response = client.post("/api/v1/llm/credentials/models", json={"base_url": PRIVATE_BASE})
+
+ assert response.status_code == 200, response.text
+ assert "authorization" not in wire.requests[0].headers
+
+ def test_no_base_url_anywhere_is_a_422(self, route: Any) -> None:
+ client, wire, _ = route
+
+ response = client.post("/api/v1/llm/credentials/models", json={})
+
+ assert response.status_code == 422
+ assert wire.requests == []
+
+ def test_a_non_http_base_url_is_a_422(self, route: Any) -> None:
+ client, wire, _ = route
+
+ response = client.post("/api/v1/llm/credentials/models", json={"base_url": "file:///etc/passwd"})
+
+ assert response.status_code == 422
+ assert wire.requests == []
+
+ def test_a_read_scoped_key_cannot_make_the_server_dial_out(self, route: Any) -> None:
+ """`settings:write`, like `/test`: the route dials an operator-supplied
+ URL with private addresses permitted."""
+ client, wire, state = route
+ state["scopes"] = ["settings:read"]
+
+ response = client.post("/api/v1/llm/credentials/models", json={"base_url": PRIVATE_BASE})
+
+ assert response.status_code == 403
+ assert wire.requests == []
+
+ def test_a_write_scoped_key_is_not_refused(self, route: Any) -> None:
+ """The negative control for the refusal above."""
+ client, _, state = route
+ state["scopes"] = ["settings:write"]
+
+ response = client.post("/api/v1/llm/credentials/models", json={"base_url": PRIVATE_BASE})
+
+ assert response.status_code == 200, response.text