OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.
Python
fix: scope watsonx on-prem TLS verification (#2409)
Edwin Jose83348f8306c3
diff --git a/.env.example b/.env.exampleindex b1dd6dfd7..f1d640f8a 100644--- a/.env.example+++ b/.env.example@@ -258,6 +258,20 @@ OLLAMA_ENDPOINT=WATSONX_API_KEY=WATSONX_ENDPOINT=WATSONX_PROJECT_ID=++# --- IBM watsonx.ai on-prem (Cloud Pak for Data / Software Hub) ----------+# WATSONX_ONPREM_ENDPOINT=https://cpd-cluster.example.com+# WATSONX_ONPREM_USERNAME=+# WATSONX_ONPREM_API_KEY=+# Alternatively provide base64(username:api-key) instead of the two fields above.+# WATSONX_ONPREM_ZEN_API_KEY=+# WATSONX_ONPREM_SPACE_ID=+# WATSONX_ONPREM_PROJECT_ID=+# Provider-scoped TLS policy: false (default), true, or a mounted CA bundle path.+# SSL_VERIFY no longer controls this provider. SSL_CERT_FILE remains available+# as the process trust store when verification is enabled.+# WATSONX_ONPREM_TLS_VERIFY=false+# Tune the ibm-watsonx-ai SDK's native 429 retry used during ingestion embedding.# OPTIONAL: Max retries = retry attempts (default 10).# WATSONX_MAX_RETRIES=10diff --git a/docs/docs/reference/configuration.mdx b/docs/docs/reference/configuration.mdxindex 04f66e8e6..4faa93974 100644--- a/docs/docs/reference/configuration.mdx+++ b/docs/docs/reference/configuration.mdx@@ -63,6 +63,13 @@ These variables are initially set during the [application onboarding process](/i| `WATSONX_API_KEY` | Not set | API key for the IBM watsonx.ai model provider. || `WATSONX_ENDPOINT` | Not set | Custom provider endpoint for the IBM watsonx.ai model provider. || `WATSONX_PROJECT_ID` | Not set | Project ID for the IBM watsonx.ai model provider. |+| `WATSONX_ONPREM_ENDPOINT` | Not set | HTTPS base URL of the IBM Cloud Pak for Data or Software Hub cluster. |+| `WATSONX_ONPREM_USERNAME` | Not set | Cluster username used with `WATSONX_ONPREM_API_KEY`. |+| `WATSONX_ONPREM_API_KEY` | Not set | Cluster API key exchanged with the username for CPD authorization. |+| `WATSONX_ONPREM_ZEN_API_KEY` | Not set | Pre-encoded `base64(username:api-key)` alternative to the username and API-key variables. |+| `WATSONX_ONPREM_SPACE_ID` | Not set | Deployment space used for inference. |+| `WATSONX_ONPREM_PROJECT_ID` | Not set | Optional project alternative to a deployment space. |+| `WATSONX_ONPREM_TLS_VERIFY` | `false` | Provider-scoped TLS policy: `false`, `true`, or a mounted CA bundle path. `SSL_VERIFY` no longer controls this provider; `SSL_CERT_FILE` supplies the process trust store when verification is enabled. || `RHOAI_ENDPOINT` | Not set | Chat endpoint for the Red Hat OpenShift AI model provider: the predictor Service of the chat `InferenceService`, ending in `/v1`. || `RHOAI_EMBEDDINGS_ENDPOINT` | Not set | Embeddings endpoint for the Red Hat OpenShift AI model provider, a second `InferenceService`. Leave unset only if one endpoint serves both chat and embeddings. || `RHOAI_API_KEY` | Not set | Kubernetes bearer token for the Red Hat OpenShift AI model provider, sent to both endpoints. Its subject needs `get` on `inferenceservices` in the model namespace. |diff --git a/enhancements/providers/registry.py b/enhancements/providers/registry.pyindex e5683a1d1..d1a44aa05 100644--- a/enhancements/providers/registry.py+++ b/enhancements/providers/registry.py@@ -75,6 +75,15 @@ def credentials_for(return dict(translate(stored))+def runtime_kwargs_for(+ enhancement: ModuleType,+ stored: Mapping[str, Any],+) -> dict[str, Any]:+ """Build optional non-serializable kwargs only at a LiteLLM call boundary."""+ build = getattr(enhancement, "litellm_runtime_kwargs", None)+ return dict(build(stored)) if callable(build) else {}++@functools.cachedef _accepts_kind(enhancement: ModuleType) -> bool:"""Whether the module's `litellm_credentials` takes a `kind` keyword.diff --git a/enhancements/providers/watsonx/onprem.py b/enhancements/providers/watsonx/onprem.pyindex 3dc59b040..1808495cd 100644--- a/enhancements/providers/watsonx/onprem.py+++ b/enhancements/providers/watsonx/onprem.py@@ -34,42 +34,31 @@ Nothing here imports OpenRAG config — `config_manager`, `model_catalog` andVerified end to end against a Cloud Pak for Data 5.x cluster: ZenApiKey acceptedon `/ml/v1`, model listing, chat, streaming chat, embeddings and tool calling.-TLS: trusting a cluster's own CA----------------------------------A CPD cluster is usually fronted by an internal or self-signed CA, and the call-then fails before it leaves the process::-- litellm.InternalServerError: WatsonxException - Cannot connect to host- <cluster>:443 ssl:True [SSLCertVerificationError: ... certificate verify- failed: self-signed certificate in certificate chain]--There is no per-provider setting for this, and no OpenRAG one either. LiteLLM-resolves TLS trust from process-wide environment only — verified against-litellm 1.84 with a self-signed server, chat and embeddings both:--- ``SSL_CERT_FILE=<path>`` works. **This is the fix.**-- ``SSL_VERIFY=false`` works, but turns verification off for *every*- provider in the process, OpenAI and Anthropic included. Development only.-- ``ssl_verify=False`` as a per-call kwarg does **not** work: litellm threads it- through only some providers, and watsonx is not one of them. It is silently- ignored, so do not reach for it.--``SSL_CERT_FILE`` *replaces* certifi's roots rather than adding to them, so-pointing it at the cluster CA alone breaks every public provider. Concatenate::-- cat "$(python -m certifi)" cluster-ca.crt > /etc/ssl/certs/openrag-ca.pem- export SSL_CERT_FILE=/etc/ssl/certs/openrag-ca.pem--In Kubernetes, mount the cluster CA into the backend pod and build that bundle-in the entrypoint; the operator's ConfigMap is the usual home for the CA. A-failure that gets this far is reported by the gateway as a trust problem rather-than as an outage — see ``_UPSTREAM_TLS_MESSAGE`` in ``services/llm_gateway``.+TLS: provider-scoped certificate verification+---------------------------------------------+A CPD cluster is usually fronted by an internal or self-signed CA. TLS trust is+stored with this provider as ``ssl_verify``: ``true`` is the default and uses+OpenRAG's trust store, ``false`` disables verification, and any other value is+the path to a mounted CA bundle.++LiteLLM's watsonx adapter does not consistently consume an ``ssl_verify`` call+kwarg. It does accept an explicit HTTP client on both chat and embedding paths,+so ``litellm_runtime_kwargs`` creates a provider-scoped ``AsyncHTTPHandler`` at+the call boundary. Stored and translated credentials remain serializable data;+OpenRAG's health check and model discovery resolve the same TLS value for their+direct httpx calls.++The CA path is on the backend filesystem, not the browser's. In Kubernetes,+mount the cluster CA into the backend pod. Use a bundle containing both the+public roots and the cluster CA when this deployment also calls public+providers."""from __future__ import annotationsimport asyncioimport base64+import osimport timefrom collections.abc import Mappingfrom typing import Any, NamedTuple, cast@@ -161,12 +150,104 @@ CREDENTIAL_FIELDS: list[dict[str, Any]] = ["options": None,"default_value": None,},+ {+ "key": "ssl_verify",+ "label": "TLS certificate verification",+ "placeholder": "/etc/ssl/certs/openrag-ca.pem",+ "tooltip": "Verify the cluster certificate using OpenRAG's trust store, or provide "+ "the backend path to a mounted CA bundle. Disabling verification is for local "+ "development only.",+ "required": False,+ "field_type": "text",+ "options": None,+ "default_value": "true",+ },]#: Fields an operator fills in that are not LiteLLM kwargs. `username` is half-#: of the Zen key and nothing else — forwarding it would land it in the request-#: body, since LiteLLM passes kwargs it does not recognise straight through.-_LOCAL_ONLY_FIELDS = frozenset({"username"})+#: of the Zen key and nothing else. `ssl_verify` configures the provider's HTTP+#: client rather than the request body. Forwarding either would leak it into the+#: watsonx request payload.+_LOCAL_ONLY_FIELDS = frozenset({"username", "ssl_verify"})++_FALSE_TLS_VALUES = frozenset({"false", "0", "no", "off"})+_TRUE_TLS_VALUES = frozenset({"true", "1", "yes", "on"})+_reported_tls_settings: set[str] = set()++#: One source of truth for persisted and pending authentication fields.+AUTH_METHOD_FIELDS = {+ "username_api_key": frozenset({"username", "api_key"}),+ "zen_api_key": frozenset({"zen_api_key"}),+}+SHARED_CREDENTIAL_FIELDS = frozenset({"api_base", "space_id", "project_id", "ssl_verify"})+++def credential_fields_for_auth_method(auth_method: str) -> frozenset[str]:+ """Return every field valid for one on-prem authentication method."""+ active = AUTH_METHOD_FIELDS.get(auth_method)+ if active is None:+ raise ValueError("Choose a valid watsonx.ai on-prem authentication method")+ return SHARED_CREDENTIAL_FIELDS | active+++def _values(stored: Mapping[str, Any] | None) -> dict[str, str]:+ """Stored form values as trimmed strings, excluding runtime objects."""+ values: dict[str, str] = {}+ for name, value in (stored or {}).items():+ if name == "client":+ continue+ if isinstance(value, bool):+ text = "true" if value else "false"+ elif value is None or isinstance(value, (list, tuple, dict, set)):+ continue+ else:+ text = str(value).strip()+ if text:+ values[str(name)] = text+ return values+++def _warn_once(setting: str, message: str, **fields: Any) -> None:+ if setting in _reported_tls_settings:+ return+ _reported_tls_settings.add(setting)+ logger.warning(message, **fields)+++def resolve_ssl_verify(value: Any) -> bool | str:+ """Resolve a provider TLS value without consulting process-wide settings."""+ raw = str(value if value is not None else "").strip()+ if raw.lower() in _FALSE_TLS_VALUES:+ _warn_once(+ "disabled",+ "TLS verification is disabled for watsonx.ai on-prem. Credentials "+ "and model traffic can be intercepted; configure trust for deployed use.",+ )+ return False+ if not raw or raw.lower() in _TRUE_TLS_VALUES:+ return True+ if not os.path.isfile(raw):+ raise ValueError("The watsonx.ai on-prem CA bundle path is not usable")+ return raw+++def litellm_runtime_kwargs(stored: Mapping[str, Any]) -> dict[str, Any]:+ """Reuse an event-loop-safe LiteLLM transport for one resolved TLS policy."""+ import litellm+ from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler++ tls = ssl_verify(stored)+ revision = os.stat(tls).st_mtime_ns if isinstance(tls, str) else ""+ cache_key = f"openrag-watsonx-onprem:{tls!s}:{revision}"+ cache = litellm.in_memory_llm_clients_cache+ client = cache.get_cache(cache_key)+ if client is None:+ client = AsyncHTTPHandler(+ ssl_verify=tls,+ client_alias="openrag-watsonx-onprem",+ )+ cache.set_cache(cache_key, client, litellm_owned_client=True)+ return {"client": client}def zen_api_key(username: str | None, api_key: str | None) -> str:@@ -182,22 +263,21 @@ def zen_api_key(username: str | None, api_key: str | None) -> str:return base64.b64encode(f"{user}:{key}".encode()).decode("ascii")-def litellm_credentials(stored: Mapping[str, Any]) -> dict[str, str]:- """Stored form values as LiteLLM kwargs for `watsonx`.+def litellm_credentials(stored: Mapping[str, Any]) -> dict[str, Any]:+ """Translate stored form values into serializable LiteLLM credentials.`api_key` is set to the Zen key as well as `zen_api_key`: the embeddingspath rejects the call before it ever reads the auth header if `api_key` isunset, and the header it then builds comes from `zen_api_key`, so the twohave to travel together.++ Runtime transport objects deliberately stay out of this mapping. Callers+ that invoke LiteLLM merge `litellm_runtime_kwargs` at the final call seam."""- values = {- str(name): str(value).strip()- for name, value in (stored or {}).items()- if str(value or "").strip()- }+ values = _values(stored)zen = values.get("zen_api_key") or zen_api_key(values.get("username"), values.get("api_key"))- credentials = {+ credentials: dict[str, Any] = {name: valuefor name, value in values.items()if name not in _LOCAL_ONLY_FIELDS and name not in {"api_key", "zen_api_key"}@@ -225,23 +305,22 @@ MODEL_SPECS_PATH = "/ml/v1/foundation_model_specs"#: not serve.API_VERSION = "2024-03-13"+#: Cloud Pak for Data deployment-space discovery endpoint and page limits.+SPACES_PATH = "/v2/spaces"+CPD_AUTHORIZE_PATH = "/icp4d-api/v1/authorize"+SPACES_PAGE_LIMIT = 100+MAX_SPACE_PAGES = 10-def auth_header(stored: Mapping[str, Any]) -> str:- """The `Authorization` value for OpenRAG's own calls to the cluster.- The health check talks to `/ml/v1` directly rather than through LiteLLM, so- it has to build the same header LiteLLM would.- """- credentials = litellm_credentials(stored)- if credentials.get("zen_api_key"):- return f"ZenApiKey {credentials['zen_api_key']}"+def auth_header(stored: Mapping[str, Any]) -> str:+ """The `Authorization` value for OpenRAG's own calls to the cluster."""+ values = _values(stored)+ zen = values.get("zen_api_key") or zen_api_key(values.get("username"), values.get("api_key"))+ if zen:+ return f"ZenApiKey {zen}"# Deliberately no `Bearer <api_key>` fallback. A Cloud Pak for Data API key- # is not a bearer token — the cluster issues one from `/icp4d-api/v1/authorize`- # in exchange for a *username and* an API key. Sending the raw key gets a- # 401 that looks like bad credentials rather than incomplete ones, and the- # caller quietly falls back to the configured model list with nothing on- # screen to say why. Returning nothing lets the health check say what is- # missing.+ # is not a bearer token — returning nothing lets the health check report+ # incomplete credentials instead of a misleading rejected-key error.return ""@@ -262,6 +341,36 @@ def model_specs_params(**extra: Any) -> dict[str, Any]:return {"version": API_VERSION, **extra}+def spaces_url(api_base: str) -> str:+ """Deployment spaces visible to the configured CPD identity."""+ return f"{(api_base or '').rstrip('/')}{SPACES_PATH}"+++def cpd_authorize_url(api_base: str) -> str:+ """Endpoint that exchanges CPD user credentials for a bearer token."""+ return f"{(api_base or '').rstrip('/')}{CPD_AUTHORIZE_PATH}"+++def _cpd_user_credentials(values: Mapping[str, str]) -> tuple[str, str] | None:+ """Recover the username/API-key pair needed by CPD's token endpoint."""+ username = values.get("username", "")+ api_key = values.get("api_key", "")+ if username and api_key:+ return username, api_key++ encoded = values.get("zen_api_key", "")+ if not encoded:+ return None+ try:+ decoded = base64.b64decode(encoded, validate=True).decode("utf-8")+ username, api_key = decoded.split(":", 1)+ except (ValueError, UnicodeDecodeError):+ return None+ if not username or not api_key:+ return None+ return username, api_key++def _error_details(response: Any) -> str:"""Extract the concise error that Cloud Pak for Data returned."""try:@@ -305,6 +414,136 @@ async def _http_request_with_retry(raise RuntimeError(f"CPD request to {url} failed after retries")+class SpaceDiscoveryError(Exception):+ """A CPD spaces request failed with an actionable upstream status."""++ def __init__(self, status_code: int, message: str):+ super().__init__(message)+ self.status_code = status_code+++async def _cpd_bearer_token(+ client: Any,+ api_base: str,+ values: Mapping[str, str],+) -> str:+ """Exchange stored CPD credentials for APIs that reject ZenApiKey."""+ credentials = _cpd_user_credentials(values)+ if credentials is None:+ return ""+ username, api_key = credentials+ response = await _http_request_with_retry(+ "POST",+ cpd_authorize_url(api_base),+ client=client,+ headers={"Accept": "application/json", "Content-Type": "application/json"},+ json={"username": username, "api_key": api_key},+ timeout=15.0,+ )+ if response.status_code != 200:+ raise SpaceDiscoveryError(response.status_code, _error_details(response))+ body = response.json()+ if not isinstance(body, Mapping):+ raise ValueError("The watsonx.ai cluster returned an invalid authorization response")+ token = str(+ body.get("token") or body.get("accessToken") or body.get("access_token") or ""+ ).strip()+ if not token:+ raise ValueError("The watsonx.ai cluster authorization response contained no token")+ return token+++async def list_spaces(credentials: Mapping[str, Any]) -> list[dict[str, str]]:+ """List deployment spaces accessible to the configured CPD identity."""+ import httpx++ values = _values(credentials)+ api_base = values.get("api_base", "")+ header = auth_header(values)+ if not api_base:+ raise ValueError("Enter the watsonx.ai on-prem cluster URL first")+ parsed_base = urlsplit(api_base)+ if parsed_base.scheme != "https" or not parsed_base.netloc:+ raise ValueError("The watsonx.ai on-prem cluster URL must use HTTPS")+ if not header:+ raise ValueError(+ "Enter a username and API key, or a Zen API key, to load deployment spaces"+ )++ headers = {"Authorization": header, "Accept": "application/json"}+ url = spaces_url(api_base)+ params: dict[str, Any] | None = {+ "version": API_VERSION,+ "limit": SPACES_PAGE_LIMIT,+ }+ resources: list[Any] = []+ bearer_attempted = False+ async with httpx.AsyncClient(verify=ssl_verify(values), timeout=15.0) as client:+ token = await _cpd_bearer_token(client, api_base, values)+ if token:+ headers = {+ "Authorization": f"Bearer {token}",+ "Accept": "application/json",+ }+ bearer_attempted = True+ for _ in range(MAX_SPACE_PAGES):+ response = await _http_request_with_retry(+ "GET",+ url,+ client=client,+ headers=headers,+ params=params,+ timeout=15.0,+ )+ if response.status_code != 200:+ details = _error_details(response)+ missing_authorization = (+ response.status_code == 400 and "authorization header" in details.lower()+ )+ if (+ response.status_code in {401, 403} or missing_authorization+ ) and not bearer_attempted:+ bearer_attempted = True+ token = await _cpd_bearer_token(client, api_base, values)+ if token:+ headers = {+ "Authorization": f"Bearer {token}",+ "Accept": "application/json",+ }+ continue+ raise SpaceDiscoveryError(response.status_code, details)+ body = response.json()+ if not isinstance(body, dict):+ raise ValueError("The watsonx.ai cluster returned an invalid spaces response")+ resources.extend(body.get("resources") or [])+ next_page = body.get("next")+ href = next_page.get("href") if isinstance(next_page, Mapping) else None+ if not href:+ break+ parsed = urlsplit(href)+ url = f"{api_base.rstrip('/')}{urlunsplit(('', '', parsed.path, parsed.query, ''))}"+ params = None++ spaces: list[dict[str, str]] = []+ seen: set[str] = set()+ for resource in resources:+ if not isinstance(resource, Mapping):+ continue+ metadata = resource.get("metadata")+ entity = resource.get("entity")+ metadata = metadata if isinstance(metadata, Mapping) else {}+ entity = entity if isinstance(entity, Mapping) else {}+ space_id = str(metadata.get("id") or resource.get("id") or "").strip()+ if not space_id or space_id in seen:+ continue+ seen.add(space_id)+ name = str(+ metadata.get("name") or entity.get("name") or resource.get("name") or space_id+ ).strip()+ spaces.append({"id": space_id, "name": name})+ return spaces++async def lightweight_health_check(credentials: Mapping[str, Any]) -> None:"""Validate CPD connectivity and Zen credentials without selecting a model."""import httpx@@ -321,7 +560,7 @@ async def lightweight_health_check(credentials: Mapping[str, Any]) -> None:url = model_specs_url(api_base)try:- async with httpx.AsyncClient(verify=ssl_verify()) as client:+ async with httpx.AsyncClient(verify=ssl_verify(credentials)) as client:response = await _http_request_with_retry("GET",url,@@ -347,21 +586,9 @@ async def lightweight_health_check(credentials: Mapping[str, Any]) -> None:raise Exception(details)-def ssl_verify() -> bool | str:- """The TLS setting LiteLLM will use, so our own calls agree with real traffic.-- A cluster behind its own CA is reached by pointing `SSL_CERT_FILE` at a- bundle (or, in development, by `SSL_VERIFY=false`). Both are read by- LiteLLM, not by httpx, so without this a direct call here could fail on a- certificate that the gateway accepts, or the reverse.- """- try:- from litellm.llms.custom_httpx.http_handler import get_ssl_verify-- return get_ssl_verify()- except Exception:- logger.debug("Could not read LiteLLM's TLS setting; verifying normally", exc_info=True)- return True+def ssl_verify(stored: Mapping[str, Any] | None = None) -> bool | str:+ """The provider-scoped TLS setting used by direct and LiteLLM calls."""+ return resolve_ssl_verify(_values(stored).get("ssl_verify"))#: How long a fetched model list is reused. The set of deployed models changes@@ -400,9 +627,18 @@ def cached_models() -> ClusterModels | None:def _cache_key(credentials: Mapping[str, Any]) -> str:- """Identifies the cluster and the credentials a cached list was fetched with."""- values = litellm_credentials(credentials)- return f"{values.get('api_base', '')}|{values.get('api_key', '')}"+ """Identify the cluster, credentials, and TLS policy behind a model list."""+ values = _values(credentials)+ zen = values.get("zen_api_key") or zen_api_key(values.get("username"), values.get("api_key"))+ raw_tls = values.get("ssl_verify", "true")+ normalized_tls = raw_tls.lower()+ if normalized_tls in _TRUE_TLS_VALUES:+ tls = "true"+ elif normalized_tls in _FALSE_TLS_VALUES:+ tls = "false"+ else:+ tls = raw_tls+ return f"{values.get('api_base', '')}|{zen}|{tls}"def _resource_markers(resource: Mapping[str, Any]) -> set[str]:@@ -500,7 +736,7 @@ async def fetch_models(credentials: Mapping[str, Any]) -> ClusterModels | None:params: dict[str, Any] | None = model_specs_params(limit=MODELS_PAGE_LIMIT)resources: list[Any] = []try:- async with httpx.AsyncClient(verify=ssl_verify(), timeout=15.0) as client:+ async with httpx.AsyncClient(verify=ssl_verify(credentials), timeout=15.0) as client:for _ in range(MAX_MODEL_PAGES):response = await client.get(url, headers=headers, params=params)if response.status_code != 200:diff --git a/frontend/app/api/mutations/useOnboardingMutation.ts b/frontend/app/api/mutations/useOnboardingMutation.tsindex a5831dcf1..c0283811a 100644--- a/frontend/app/api/mutations/useOnboardingMutation.ts+++ b/frontend/app/api/mutations/useOnboardingMutation.ts@@ -23,6 +23,7 @@ export interface OnboardingVariables {watsonx_project_id?: string;ollama_endpoint?: string;provider_credentials?: Record<string, Record<string, string>>;+ provider_credential_removals?: Record<string, string[]>;provider_auth_methods?: Record<string, string>;}diff --git a/frontend/app/api/mutations/useUpdateSettingsMutation.ts b/frontend/app/api/mutations/useUpdateSettingsMutation.tsindex 2e6cae9fa..2c136274a 100644--- a/frontend/app/api/mutations/useUpdateSettingsMutation.ts+++ b/frontend/app/api/mutations/useUpdateSettingsMutation.ts@@ -54,6 +54,7 @@ export interface UpdateSettingsRequest {// Bypass the "this provider's embedding models are still in use" guard.force_remove?: boolean;provider_credentials?: Record<string, Record<string, string>>;+ provider_credential_removals?: Record<string, string[]>;provider_auth_methods?: Record<string, string>;remove_provider_config?: string;}diff --git a/frontend/app/onboarding/_components/advanced.tsx b/frontend/app/onboarding/_components/advanced.tsxindex a7cdabc7f..581241d6a 100644--- a/frontend/app/onboarding/_components/advanced.tsx+++ b/frontend/app/onboarding/_components/advanced.tsx@@ -44,6 +44,7 @@ export function AdvancedOnboarding({previewLimit={5}searchPlaceholder={searchPlaceholder}data-testid="embedding-model-selector"+ aria-label="Embedding model"icon={icon}value={embeddingModel}onValueChange={setEmbeddingModel}@@ -63,6 +64,7 @@ export function AdvancedOnboarding({previewLimit={5}searchPlaceholder={searchPlaceholder}data-testid="language-model-selector"+ aria-label="Language model"icon={icon}value={languageModel}onValueChange={setLanguageModel}diff --git a/frontend/app/onboarding/_components/generic-onboarding.test.tsx b/frontend/app/onboarding/_components/generic-onboarding.test.tsxindex 8206ef7cf..b56b62673 100644--- a/frontend/app/onboarding/_components/generic-onboarding.test.tsx+++ b/frontend/app/onboarding/_components/generic-onboarding.test.tsx@@ -32,6 +32,49 @@ const catalog = {{ model: "text-embedding-3-large" },],},+ {+ key: "watsonx_onprem",+ name: "IBM watsonx.ai",+ credential_fields: [+ {+ key: "api_base",+ label: "Cluster URL",+ required: true,+ field_type: "text",+ },+ {+ key: "username",+ label: "Username",+ required: true,+ field_type: "text",+ },+ {+ key: "api_key",+ label: "API key",+ required: true,+ field_type: "password",+ },+ {+ key: "space_id",+ label: "Deployment space ID",+ required: false,+ field_type: "text",+ },+ {+ key: "ssl_verify",+ label: "TLS certificate verification",+ required: false,+ field_type: "text",+ },+ ],+ models: [+ {+ model: "ibm/granite-3-8b-instruct",+ capabilities: ["function_calling"],+ },+ ],+ embedding_models: [],+ },],};@@ -56,6 +99,14 @@ function renderOnboarding(providers: ["tooltip"],handlers: [http.get("/api/models/catalog", () => HttpResponse.json(catalog)),+ http.post("/api/models/watsonx_onprem/spaces", () =>+ HttpResponse.json({+ spaces: [+ { id: "space-prod", name: "Production" },+ { id: "space-dev", name: "Development" },+ ],+ }),+ ),],},);@@ -67,14 +118,14 @@ describe("GenericOnboarding model selection", () => {{kind: "language",isEmbedding: false,- selector: "language-model-selector",+ selectorName: "Language model",modelKey: "llm_model",catalogModel: "gpt-5.5",},{kind: "embedding",isEmbedding: true,- selector: "embedding-model-selector",+ selectorName: "Embedding model",modelKey: "embedding_model",catalogModel: "text-embedding-3-small",},@@ -83,7 +134,9 @@ describe("GenericOnboarding model selection", () => {"azure",testCase.isEmbedding,);- const selector = screen.getByTestId(testCase.selector);+ const selector = await screen.findByRole("combobox", {+ name: testCase.selectorName,+ });await user.click(selector);const catalogOption = await screen.findByRole("option", {name: testCase.catalogModel,@@ -98,7 +151,9 @@ describe("GenericOnboarding model selection", () => {it("keeps automatic defaults for other generic providers", async () => {const { user, getSettings } = renderOnboarding("openai_like");- await user.click(screen.getByTestId("language-model-selector"));+ await user.click(+ await screen.findByRole("combobox", { name: "Language model" }),+ );await screen.findByRole("option", { name: "gpt-4.1" });expect(getSettings().llm_model).toBe("gpt-4.1");});@@ -111,7 +166,9 @@ describe("GenericOnboarding model selection", () => {},},});- await user.click(screen.getByTestId("language-model-selector"));+ await user.click(+ await screen.findByRole("combobox", { name: "Language model" }),+ );await screen.findByRole("option", { name: "gpt-4.1" });const apiBase = screen.getByLabelText("API base");@@ -127,9 +184,104 @@ describe("GenericOnboarding model selection", () => {);});+ it("keeps watsonx on-prem TLS configuration provider-scoped", async () => {+ const { user, getSettings } = renderOnboarding("watsonx_onprem");++ await user.click(+ await screen.findByRole("button", { name: "Advanced settings" }),+ );+ const verifyTls = screen.getByRole("switch", {+ name: "Verify TLS certificates",+ });++ expect(verifyTls).toBeChecked();+ const caPath = screen.getByRole("textbox", {+ name: "Custom CA bundle path",+ });+ await user.type(caPath, "/etc/ssl/certs/openrag-ca.pem");+ expect(getSettings().provider_credentials?.watsonx_onprem?.ssl_verify).toBe(+ "/etc/ssl/certs/openrag-ca.pem",+ );+ });++ it("marks a cleared saved field for removal during onboarding", async () => {+ const { user, getSettings } = renderOnboarding("watsonx_onprem", false, {+ custom: {+ watsonx_onprem: {+ credential_values: {+ api_base: "https://cpd.example.com",+ space_id: "deployment-space",+ ssl_verify: "true",+ },+ },+ },+ });++ await user.click(+ await screen.findByRole("button", { name: "Advanced settings" }),+ );+ expect(+ screen.getByRole("combobox", { name: "Deployment space ID" }),+ ).toHaveTextContent("deployment-space");++ await user.click(+ screen.getByRole("button", { name: "Clear deployment space" }),+ );+ expect(getSettings().provider_credential_removals?.watsonx_onprem).toEqual([+ "space_id",+ ]);+ });++ it("loads and selects an available deployment space during onboarding", async () => {+ const { user, getSettings } = renderOnboarding("watsonx_onprem", false, {+ custom: {+ watsonx_onprem: {+ auth_method: "username_api_key",+ credential_values: {+ api_base: "https://cpd.example.com",+ username: "cpd-user",+ ssl_verify: "true",+ },+ secret_fields: ["api_key"],+ },+ },+ });++ await user.click(+ await screen.findByRole("button", { name: "Advanced settings" }),+ );+ const spaceSelect = screen.getByRole("combobox", {+ name: "Deployment space ID",+ });+ await user.click(spaceSelect);+ await user.click(+ await screen.findByRole("option", { name: /Production.*space-prod/ }),+ );++ expect(getSettings().provider_credentials?.watsonx_onprem?.space_id).toBe(+ "space-prod",+ );++ await user.click(spaceSelect);+ await user.type(+ screen.getByPlaceholderText("Search or enter a deployment space ID…"),+ "manually-entered-space",+ );+ await user.click(+ await screen.findByRole("option", {+ name: /Use manually-entered-space as deployment space ID/,+ }),+ );+ expect(getSettings().provider_credentials?.watsonx_onprem?.space_id).toBe(+ "manually-entered-space",+ );+ });+it("updates the selected embedding model for another generic provider", async () => {const { user, getSettings } = renderOnboarding("openai_like", true);- await user.click(screen.getByTestId("embedding-model-selector"));+ await user.click(+ await screen.findByRole("combobox", { name: "Embedding model" }),+ );await user.click(await screen.findByRole("option", { name: "text-embedding-3-large" }),);diff --git a/frontend/app/onboarding/_components/generic-onboarding.tsx b/frontend/app/onboarding/_components/generic-onboarding.tsxindex 435274177..71b2913c5 100644--- a/frontend/app/onboarding/_components/generic-onboarding.tsx+++ b/frontend/app/onboarding/_components/generic-onboarding.tsx@@ -16,6 +16,8 @@ import {savedSecretFieldsForProvider,} from "@/components/models/catalog-models";import { getProviderChrome } from "@/components/models/model-helpers";+import { WatsonxSpaceSelect } from "@/components/models/watsonx-space-select";+import { WatsonxTlsSettings } from "@/components/models/watsonx-tls-settings";import type { OnboardingVariables } from "../../api/mutations/useOnboardingMutation";import { AdvancedOnboarding } from "./advanced";import { GenericProviderCredentialFields } from "./generic-provider-credential-fields";@@ -51,14 +53,17 @@ export function GenericOnboarding({() => onboardingCredentialFields(catalog, provider),[catalog, provider],);+ const savedCredentials = useMemo(+ () => savedCredentialValuesForProvider(providers, provider),+ [providers, provider],+ );const savedSecrets = useMemo(() => new Set(savedSecretFieldsForProvider(providers, provider)),[providers, provider],);- const [credentials, setCredentials] = useState<Record<string, string>>(() =>- savedCredentialValuesForProvider(providers, provider),- );+ const [credentials, setCredentials] =+ useState<Record<string, string>>(savedCredentials);const [model, setModel] = useState("");const [azureAuthMethod, setAzureAuthMethod] = useState(providers?.custom?.[provider]?.auth_method ?? "api_key",@@ -78,6 +83,7 @@ export function GenericOnboarding({nextModel: string,) => {const submitted: Record<string, string> = {};+ const removals: string[] = [];const activeAzureFields = new Set(["api_base","api_version",@@ -88,6 +94,7 @@ export function GenericOnboarding({"api_base","space_id","project_id",+ "ssl_verify",...(onPremAuthMethod === "zen_api_key"? ["zen_api_key"]: ["username", "api_key"]),@@ -99,33 +106,50 @@ export function GenericOnboarding({const trimmed = (value ?? "").trim();if (trimmed !== "") {submitted[key] = trimmed;+ } else if (!savedSecrets.has(key) && savedCredentials[key]) {+ removals.push(key);}}- setSettingsRef.current((prev) => ({- ...prev,- ...(isEmbedding- ? { embedding_provider: provider, embedding_model: nextModel }- : { llm_provider: provider, llm_model: nextModel }),- provider_credentials: Object.keys(submitted).length- ? { ...prev.provider_credentials, [provider]: submitted }- : prev.provider_credentials,- ...(provider === "azure"- ? {- provider_auth_methods: {- ...prev.provider_auth_methods,- azure: azureAuthMethod,- },- }- : {}),- ...(provider === "watsonx_onprem"- ? {- provider_auth_methods: {- ...prev.provider_auth_methods,- watsonx_onprem: onPremAuthMethod,- },- }- : {}),- }));+ setSettingsRef.current((prev) => {+ const providerCredentialRemovals = {+ ...prev.provider_credential_removals,+ };+ if (removals.length > 0) {+ providerCredentialRemovals[provider] = removals;+ } else {+ delete providerCredentialRemovals[provider];+ }++ return {+ ...prev,+ ...(isEmbedding+ ? { embedding_provider: provider, embedding_model: nextModel }+ : { llm_provider: provider, llm_model: nextModel }),+ provider_credentials: Object.keys(submitted).length+ ? { ...prev.provider_credentials, [provider]: submitted }+ : prev.provider_credentials,+ provider_credential_removals:+ Object.keys(providerCredentialRemovals).length > 0+ ? providerCredentialRemovals+ : undefined,+ ...(provider === "azure"+ ? {+ provider_auth_methods: {+ ...prev.provider_auth_methods,+ azure: azureAuthMethod,+ },+ }+ : {}),+ ...(provider === "watsonx_onprem"+ ? {+ provider_auth_methods: {+ ...prev.provider_auth_methods,+ watsonx_onprem: onPremAuthMethod,+ },+ }+ : {}),+ };+ });};const models = useMemo(@@ -190,6 +214,7 @@ export function GenericOnboarding({"api_base","space_id","project_id",+ "ssl_verify",...(method === "zen_api_key" ? ["zen_api_key"] : ["username", "api_key"]),]);const selected = Object.fromEntries(@@ -209,6 +234,33 @@ export function GenericOnboarding({};const renderField = (field: (typeof fields)[number]) => {+ if (provider === "watsonx_onprem" && field.key === "space_id") {+ return (+ <WatsonxSpaceSelect+ key={field.key}+ idPrefix={`onboarding-${provider}`}+ credentials={credentials}+ authMethod={onPremAuthMethod}+ hasSavedApiKey={savedSecrets.has("api_key")}+ hasSavedZenApiKey={savedSecrets.has("zen_api_key")}+ value={credentials.space_id}+ onValueChange={(value) => handleCredentialChange("space_id", value)}+ helperText={field.tooltip ?? undefined}+ />+ );+ }++ if (provider === "watsonx_onprem" && field.key === "ssl_verify") {+ return (+ <WatsonxTlsSettings+ key={field.key}+ idPrefix={`onboarding-${provider}`}+ value={credentials.ssl_verify}+ onValueChange={(value) => handleCredentialChange("ssl_verify", value)}+ />+ );+ }+const isSecret =field.field_type === "password" || field.field_type === "textarea";const hasSaved = isSecret && savedSecrets.has(field.key);diff --git a/frontend/app/onboarding/_components/generic-provider-credential-fields.tsx b/frontend/app/onboarding/_components/generic-provider-credential-fields.tsxindex a0d65efb6..52240560d 100644--- a/frontend/app/onboarding/_components/generic-provider-credential-fields.tsx+++ b/frontend/app/onboarding/_components/generic-provider-credential-fields.tsx@@ -96,9 +96,11 @@ export function GenericProviderCredentialFields({<AccordionItem value="advanced"><AccordionTrigger>Advanced settings</AccordionTrigger><AccordionContent className="space-y-4">- {fieldsForKeys(fields, ["space_id", "project_id"]).map(- renderField,- )}+ {fieldsForKeys(fields, [+ "space_id",+ "project_id",+ "ssl_verify",+ ]).map(renderField)}</AccordionContent></AccordionItem></Accordion>diff --git a/frontend/app/onboarding/_components/onboarding-card.tsx b/frontend/app/onboarding/_components/onboarding-card.tsxindex ced209fbc..282f88911 100644--- a/frontend/app/onboarding/_components/onboarding-card.tsx+++ b/frontend/app/onboarding/_components/onboarding-card.tsx@@ -548,6 +548,13 @@ const OnboardingCard = ({if (generic && Object.keys(generic).length > 0) {onboardingData.provider_credentials = { [currentProvider]: generic };}+ const credentialRemovals =+ settings.provider_credential_removals?.[currentProvider];+ if (credentialRemovals && credentialRemovals.length > 0) {+ onboardingData.provider_credential_removals = {+ [currentProvider]: credentialRemovals,+ };+ }const authMethod = settings.provider_auth_methods?.[currentProvider];if (authMethod) {onboardingData.provider_auth_methods = { [currentProvider]: authMethod };diff --git a/frontend/app/settings/_components/provider-settings-dialog.test.tsx b/frontend/app/settings/_components/provider-settings-dialog.test.tsxnew file mode 100644index 000000000..fc24de4ff--- /dev/null+++ b/frontend/app/settings/_components/provider-settings-dialog.test.tsx@@ -0,0 +1,275 @@+import { HttpResponse, http } from "msw";+import { describe, expect, it } from "vitest";+import type { UpdateSettingsRequest } from "@/app/api/mutations/useUpdateSettingsMutation";+import { makeSettings } from "@/test-utils/fixtures/settings";+import {+ renderWithProviders,+ screen,+ userEvent,+ waitFor,+} from "@/test-utils/render";+import ProviderSettingsDialog from "./provider-settings-dialog";++const catalog = {+ providers: [+ {+ key: "watsonx_onprem",+ name: "IBM watsonx.ai",+ credential_fields: [+ {+ key: "api_base",+ label: "Cluster URL",+ required: true,+ field_type: "text",+ },+ {+ key: "username",+ label: "Username",+ required: true,+ field_type: "text",+ },+ {+ key: "api_key",+ label: "API key",+ required: true,+ field_type: "password",+ },+ {+ key: "space_id",+ label: "Deployment space ID",+ required: false,+ field_type: "text",+ },+ {+ key: "ssl_verify",+ label: "TLS certificate verification",+ required: false,+ field_type: "text",+ },+ ],+ models: [],+ embedding_models: [],+ },+ ],+};++describe("ProviderSettingsDialog watsonx.ai on-prem TLS", () => {+ it("submits the advanced TLS policy with only the on-prem provider", async () => {+ let submitted: UpdateSettingsRequest | undefined;+ const settings = makeSettings({+ providers: {+ custom: {+ watsonx_onprem: {+ configured: true,+ auth_method: "username_api_key",+ credential_values: {+ api_base: "https://cpd.example.com",+ username: "cpd-user",+ ssl_verify: "true",+ },+ secret_fields: ["api_key"],+ },+ },+ },+ });++ renderWithProviders(+ <ProviderSettingsDialog+ provider="watsonx_onprem"+ displayName="IBM watsonx.ai (on-prem)"+ open+ setOpen={() => {}}+ />,+ {+ providers: ["auth", "tooltip"],+ handlers: [+ http.get("/api/settings", () => HttpResponse.json(settings)),+ http.get("/api/models/catalog", () => HttpResponse.json(catalog)),+ http.post("/api/models/watsonx_onprem/spaces", () =>+ HttpResponse.json({ spaces: [] }),+ ),+ http.post("/api/settings", async ({ request }) => {+ submitted = (await request.json()) as UpdateSettingsRequest;+ return HttpResponse.json({ message: "saved", settings });+ }),+ http.post("/api/models/openai", () =>+ HttpResponse.json({ models: [], embedding_models: [] }),+ ),+ ],+ },+ );++ const user = userEvent.setup();+ await user.click(+ await screen.findByRole("button", { name: "Advanced settings" }),+ );+ await user.click(+ screen.getByRole("switch", { name: "Verify TLS certificates" }),+ );+ await user.click(screen.getByRole("button", { name: "Save" }));++ await waitFor(() => {+ expect(submitted).toEqual({+ provider_credentials: {+ watsonx_onprem: {+ api_base: "https://cpd.example.com",+ username: "cpd-user",+ ssl_verify: "false",+ },+ },+ provider_auth_methods: { watsonx_onprem: "username_api_key" },+ });+ });+ });++ it("submits a cleared optional field so the backend can remove it", async () => {+ let submitted: UpdateSettingsRequest | undefined;+ const settings = makeSettings({+ providers: {+ custom: {+ watsonx_onprem: {+ configured: true,+ auth_method: "username_api_key",+ credential_values: {+ api_base: "https://cpd.example.com",+ username: "cpd-user",+ space_id: "deployment-space",+ ssl_verify: "true",+ },+ secret_fields: ["api_key"],+ },+ },+ },+ });++ renderWithProviders(+ <ProviderSettingsDialog+ provider="watsonx_onprem"+ displayName="IBM watsonx.ai (on-prem)"+ open+ setOpen={() => {}}+ />,+ {+ providers: ["auth", "tooltip"],+ handlers: [+ http.get("/api/settings", () => HttpResponse.json(settings)),+ http.get("/api/models/catalog", () => HttpResponse.json(catalog)),+ http.post("/api/models/watsonx_onprem/spaces", () =>+ HttpResponse.json({ spaces: [] }),+ ),+ http.post("/api/settings", async ({ request }) => {+ submitted = (await request.json()) as UpdateSettingsRequest;+ return HttpResponse.json({ message: "saved", settings });+ }),+ http.post("/api/models/openai", () =>+ HttpResponse.json({ models: [], embedding_models: [] }),+ ),+ ],+ },+ );++ const user = userEvent.setup();+ await user.click(+ await screen.findByRole("button", { name: "Advanced settings" }),+ );+ await user.click(+ screen.getByRole("button", { name: "Clear deployment space" }),+ );+ await user.click(screen.getByRole("button", { name: "Save" }));++ await waitFor(() => {+ expect(submitted?.provider_credentials?.watsonx_onprem).toEqual({+ api_base: "https://cpd.example.com",+ username: "cpd-user",+ ssl_verify: "true",+ });+ expect(submitted?.provider_credential_removals?.watsonx_onprem).toEqual([+ "space_id",+ ]);+ });+ });++ it("loads deployment spaces and submits the selected space", async () => {+ let submitted: UpdateSettingsRequest | undefined;+ let discoveryRequest: unknown;+ const settings = makeSettings({+ providers: {+ custom: {+ watsonx_onprem: {+ configured: true,+ auth_method: "username_api_key",+ credential_values: {+ api_base: "https://cpd.example.com",+ username: "cpd-user",+ ssl_verify: "true",+ },+ secret_fields: ["api_key"],+ },+ },+ },+ });++ renderWithProviders(+ <ProviderSettingsDialog+ provider="watsonx_onprem"+ displayName="IBM watsonx.ai (on-prem)"+ open+ setOpen={() => {}}+ />,+ {+ providers: ["auth", "tooltip"],+ handlers: [+ http.get("/api/settings", () => HttpResponse.json(settings)),+ http.get("/api/models/catalog", () => HttpResponse.json(catalog)),+ http.post(+ "/api/models/watsonx_onprem/spaces",+ async ({ request }) => {+ discoveryRequest = await request.json();+ return HttpResponse.json({+ spaces: [+ { id: "space-prod", name: "Production" },+ { id: "space-dev", name: "Development" },+ ],+ });+ },+ ),+ http.post("/api/settings", async ({ request }) => {+ submitted = (await request.json()) as UpdateSettingsRequest;+ return HttpResponse.json({ message: "saved", settings });+ }),+ http.post("/api/models/openai", () =>+ HttpResponse.json({ models: [], embedding_models: [] }),+ ),+ ],+ },+ );++ const user = userEvent.setup();+ await user.click(+ await screen.findByRole("button", { name: "Advanced settings" }),+ );+ const spaceSelect = screen.getByRole("combobox", {+ name: "Deployment space ID",+ });+ await waitFor(() => expect(spaceSelect).not.toBeDisabled());+ await user.click(spaceSelect);+ await user.click(+ await screen.findByRole("option", { name: /Production.*space-prod/ }),+ );+ await user.click(screen.getByRole("button", { name: "Save" }));++ await waitFor(() => {+ expect(discoveryRequest).toEqual({+ credentials: {+ api_base: "https://cpd.example.com",+ username: "cpd-user",+ ssl_verify: "true",+ },+ auth_method: "username_api_key",+ });+ expect(submitted?.provider_credentials?.watsonx_onprem?.space_id).toBe(+ "space-prod",+ );+ });+ });+});diff --git a/frontend/app/settings/_components/provider-settings-dialog.tsx b/frontend/app/settings/_components/provider-settings-dialog.tsxindex b04e24ec8..a606a620b 100644--- a/frontend/app/settings/_components/provider-settings-dialog.tsx+++ b/frontend/app/settings/_components/provider-settings-dialog.tsx@@ -150,9 +150,10 @@ const ProviderSettingsDialog = ({});const onSubmit = (data: ProviderSettingsFormData) => {- // Blank means "leave the stored value alone": the backend ignores empty- // values, and secrets are never echoed back for us to resubmit.+ // Secret inputs stay blank when a value is stored, so an empty secret means+ // "leave unchanged". Empty non-secret fields explicitly remove saved data.const credentials: Record<string, string> = {};+ const removals: string[] = [];const azureAuthFields: Record<string, Set<string>> = {api_key: new Set(["api_key"]),entra_token: new Set(["azure_ad_token"]),@@ -167,10 +168,12 @@ const ProviderSettingsDialog = ({"api_base","space_id","project_id",+ "ssl_verify",...(onPremAuthMethod === "zen_api_key"? ["zen_api_key"]: ["username", "api_key"]),]);+ const fieldsByKey = new Map(fields.map((field) => [field.key, field]));for (const [key, value] of Object.entries(data.credentials ?? {})) {if (provider === "azure" && !allowedAzureFields.has(key)) {continue;@@ -180,16 +183,26 @@ const ProviderSettingsDialog = ({const trimmed = (value ?? "").trim();if (trimmed !== "") {credentials[key] = trimmed;+ } else {+ const field = fieldsByKey.get(key);+ const isSecret =+ field?.field_type === "password" || field?.field_type === "textarea";+ if (!isSecret && saved?.credential_values?.[key]) {+ removals.push(key);+ }}}- if (Object.keys(credentials).length === 0) {+ if (Object.keys(credentials).length === 0 && removals.length === 0) {methods.setError("root", { message: "Enter at least one credential" });return;}settingsMutation.mutate({provider_credentials: { [provider]: credentials },+ ...(removals.length > 0+ ? { provider_credential_removals: { [provider]: removals } }+ : {}),...(provider === "azure"? { provider_auth_methods: { azure: azureAuthMethod } }: provider === "watsonx_onprem"diff --git a/frontend/app/settings/_components/provider-settings-form.tsx b/frontend/app/settings/_components/provider-settings-form.tsxindex dfc9cced2..0356255db 100644--- a/frontend/app/settings/_components/provider-settings-form.tsx+++ b/frontend/app/settings/_components/provider-settings-form.tsx@@ -1,6 +1,8 @@import { useFormContext } from "react-hook-form";import { LabelWrapper } from "@/components/label-wrapper";import type { CatalogCredentialField } from "@/components/models/catalog-models";+import { WatsonxSpaceSelect } from "@/components/models/watsonx-space-select";+import { WatsonxTlsSettings } from "@/components/models/watsonx-tls-settings";import {Accordion,AccordionContent,@@ -66,12 +68,46 @@ export function ProviderSettingsForm({}) {const {register,+ setValue,+ watch,formState: { errors },} = useFormContext<ProviderSettingsFormData>();const saved = new Set(savedSecretFields);+ const credentials = watch("credentials");const renderField = (field: CatalogCredentialField) => {+ if (provider === "watsonx_onprem" && field.key === "space_id") {+ return (+ <WatsonxSpaceSelect+ key={field.key}+ idPrefix="provider-watsonx-onprem"+ credentials={credentials}+ authMethod={onPremAuthMethod}+ hasSavedApiKey={saved.has("api_key")}+ hasSavedZenApiKey={saved.has("zen_api_key")}+ value={credentials.space_id}+ onValueChange={(value) =>+ setValue("credentials.space_id", value, { shouldDirty: true })+ }+ helperText={field.tooltip ?? undefined}+ />+ );+ }++ if (provider === "watsonx_onprem" && field.key === "ssl_verify") {+ return (+ <WatsonxTlsSettings+ key={field.key}+ idPrefix="provider-watsonx-onprem"+ value={watch("credentials.ssl_verify")}+ onValueChange={(value) =>+ setValue("credentials.ssl_verify", value, { shouldDirty: true })+ }+ />+ );+ }+const error = errors.credentials?.[field.key]?.message;const isSecret =field.field_type === "password" || field.field_type === "textarea";@@ -144,7 +180,7 @@ export function ProviderSettingsForm({(field) => field.key === "api_base",);const onPremAdvancedFields = fields.filter((field) =>- ["space_id", "project_id"].includes(field.key),+ ["space_id", "project_id", "ssl_verify"].includes(field.key),);return (diff --git a/frontend/components/models/model-selector.tsx b/frontend/components/models/model-selector.tsxindex 8ba6927fa..fa7124c28 100644--- a/frontend/components/models/model-selector.tsx+++ b/frontend/components/models/model-selector.tsx@@ -283,8 +283,13 @@ export function ModelSelector({return (<Popover open={open} onOpenChange={setOpen} modal={false}>- <PopoverTrigger asChild>+ <PopoverTrigger+ asChild+ role="combobox"+ aria-label={props["aria-label"] ?? placeholder}+ ><Button+ {...props}variant={variant}role="combobox"disabled={disabled || (allOptions.length === 0 && !allowCustomEntry)}@@ -295,7 +300,6 @@ export function ModelSelector({hasError && "!border-destructive",className,)}- {...props}>{value ? (<div className="flex items-center gap-2">diff --git a/frontend/components/models/watsonx-space-select.tsx b/frontend/components/models/watsonx-space-select.tsxnew file mode 100644index 000000000..bc791913a--- /dev/null+++ b/frontend/components/models/watsonx-space-select.tsx@@ -0,0 +1,283 @@+"use client";++import { Check, ChevronsUpDown, LoaderCircle, Plus, X } from "lucide-react";+import { useEffect, useId, useReducer, useState } from "react";+import { LabelWrapper } from "@/components/label-wrapper";+import { Button } from "@/components/ui/button";+import {+ Command,+ CommandEmpty,+ CommandGroup,+ CommandInput,+ CommandItem,+ CommandList,+} from "@/components/ui/command";+import {+ Popover,+ PopoverContent,+ PopoverTrigger,+} from "@/components/ui/popover";+import { cn } from "@/lib/utils";++interface WatsonxSpace {+ id: string;+ name: string;+}++interface WatsonxSpaceSelectProps {+ credentials: Record<string, string>;+ authMethod?: string;+ hasSavedApiKey?: boolean;+ hasSavedZenApiKey?: boolean;+ value?: string;+ onValueChange: (value: string) => void;+ helperText?: string;+ idPrefix: string;+}++type SpaceSelectState = {+ open: boolean;+ spaces: WatsonxSpace[];+ status: "idle" | "loading" | "loaded";+ error: string | null;+};++type SpaceSelectAction =+ | { type: "reset" }+ | { type: "loading" }+ | { type: "loaded"; spaces: WatsonxSpace[] }+ | { type: "failed"; error: string }+ | { type: "setOpen"; open: boolean };++const initialState: SpaceSelectState = {+ open: false,+ spaces: [],+ status: "idle",+ error: null,+};++function reduceSpaceSelect(+ state: SpaceSelectState,+ action: SpaceSelectAction,+): SpaceSelectState {+ switch (action.type) {+ case "reset":+ return initialState;+ case "loading":+ return { ...state, spaces: [], status: "loading", error: null };+ case "loaded":+ return { ...state, spaces: action.spaces, status: "loaded", error: null };+ case "failed":+ return { ...state, status: "loaded", error: action.error };+ case "setOpen":+ return { ...state, open: action.open };+ }+}++export function WatsonxSpaceSelect({+ credentials,+ authMethod = "username_api_key",+ hasSavedApiKey = false,+ hasSavedZenApiKey = false,+ value = "",+ onValueChange,+ helperText,+ idPrefix,+}: WatsonxSpaceSelectProps) {+ const generatedId = useId();+ const id = `${idPrefix}-space-${generatedId.replaceAll(":", "")}`;+ const [state, dispatch] = useReducer(reduceSpaceSelect, initialState);+ const [search, setSearch] = useState("");+ const { open, spaces, status, error } = state;++ const apiBase = credentials.api_base?.trim() ?? "";+ const username = credentials.username?.trim() ?? "";+ const apiKey = credentials.api_key ?? "";+ const zenApiKey = credentials.zen_api_key ?? "";+ const sslVerify = credentials.ssl_verify ?? "";+ const ready =+ Boolean(apiBase) &&+ (authMethod === "zen_api_key"+ ? Boolean(zenApiKey.trim() || hasSavedZenApiKey)+ : Boolean(username && (apiKey.trim() || hasSavedApiKey)));++ useEffect(() => {+ if (!ready) {+ dispatch({ type: "reset" });+ return;+ }++ const controller = new AbortController();+ dispatch({ type: "loading" });++ const timeout = window.setTimeout(async () => {+ try {+ const submittedCredentials = Object.fromEntries(+ Object.entries({+ api_base: apiBase,+ username,+ api_key: apiKey,+ zen_api_key: zenApiKey,+ ssl_verify: sslVerify,+ }).filter(([, credentialValue]) => credentialValue !== ""),+ );+ const response = await fetch("/api/models/watsonx_onprem/spaces", {+ method: "POST",+ headers: { "Content-Type": "application/json" },+ body: JSON.stringify({+ credentials: submittedCredentials,+ auth_method: authMethod,+ }),+ signal: controller.signal,+ });+ const result = (await response.json()) as {+ spaces?: WatsonxSpace[];+ error?: string;+ };+ if (!response.ok) {+ throw new Error(result.error || "Could not load deployment spaces");+ }+ dispatch({ type: "loaded", spaces: result.spaces ?? [] });+ } catch (requestError) {+ if (controller.signal.aborted) return;+ dispatch({+ type: "failed",+ error:+ requestError instanceof Error+ ? requestError.message+ : "Could not load deployment spaces",+ });+ }+ }, 350);++ return () => {+ window.clearTimeout(timeout);+ controller.abort();+ };+ }, [apiBase, username, apiKey, zenApiKey, sslVerify, authMethod, ready]);++ const selected = spaces.find((space) => space.id === value);+ const selectedLabel = selected?.name ?? value;+ const typedSpaceId = search.trim();+ const typedSpaceExists = spaces.some(+ (space) => space.id.toLowerCase() === typedSpaceId.toLowerCase(),+ );++ const selectSpace = (spaceId: string) => {+ onValueChange(spaceId);+ setSearch("");+ dispatch({ type: "setOpen", open: false });+ };++ return (+ <div className="min-w-0 space-y-2">+ <LabelWrapper label="Deployment space ID" helperText={helperText} id={id}>+ <div className="flex min-w-0 gap-2">+ <Popover+ open={open}+ onOpenChange={(nextOpen) => {+ dispatch({ type: "setOpen", open: nextOpen });+ if (!nextOpen) setSearch("");+ }}+ >+ <PopoverTrigger asChild>+ <Button+ id={id}+ type="button"+ variant="outline"+ role="combobox"+ aria-expanded={open}+ aria-busy={status === "loading"}+ disabled={!ready}+ className="min-w-0 flex-1 justify-between px-3 font-normal normal-case"+ ignoreTitleCase+ >+ <span className="min-w-0 truncate text-left">+ {status === "loading"+ ? "Loading deployment spaces…"+ : selectedLabel || "Select a deployment space"}+ </span>+ {status === "loading" ? (+ <LoaderCircle className="animate-spin opacity-60" />+ ) : (+ <ChevronsUpDown className="opacity-50" />+ )}+ </Button>+ </PopoverTrigger>+ <PopoverContent className="w-[var(--radix-popover-trigger-width)] p-0">+ <Command>+ <CommandInput+ placeholder="Search or enter a deployment space ID…"+ value={search}+ onValueChange={setSearch}+ />+ <CommandList>+ <CommandEmpty>+ No accessible deployment spaces. Enter an ID to use it.+ </CommandEmpty>+ <CommandGroup>+ {typedSpaceId && !typedSpaceExists && (+ <CommandItem+ value={typedSpaceId}+ onSelect={() => selectSpace(typedSpaceId)}+ className="gap-2 py-2"+ >+ <Plus />+ <span className="min-w-0 truncate">+ Use <span className="font-mono">{typedSpaceId}</span>{" "}+ as deployment space ID+ </span>+ </CommandItem>+ )}+ {spaces.map((space) => (+ <CommandItem+ key={space.id}+ value={`${space.name} ${space.id}`}+ onSelect={() => selectSpace(space.id)}+ className="items-start gap-2 py-2"+ >+ <Check+ className={cn(+ "mt-0.5",+ space.id === value ? "opacity-100" : "opacity-0",+ )}+ />+ <span className="min-w-0">+ <span className="block truncate">{space.name}</span>+ <span className="block truncate font-mono text-xs text-muted-foreground">+ {space.id}+ </span>+ </span>+ </CommandItem>+ ))}+ </CommandGroup>+ </CommandList>+ </Command>+ </PopoverContent>+ </Popover>+ {value && (+ <Button+ type="button"+ variant="ghost"+ size="iconMd"+ aria-label="Clear deployment space"+ onClick={() => onValueChange("")}+ >+ <X />+ </Button>+ )}+ </div>+ </LabelWrapper>+ {!ready && (+ <p className="text-sm text-muted-foreground">+ Enter connection credentials to load deployment spaces.+ </p>+ )}+ {error && (+ <p role="alert" className="text-sm text-destructive">+ {error}+ </p>+ )}+ </div>+ );+}diff --git a/frontend/components/models/watsonx-tls-settings.tsx b/frontend/components/models/watsonx-tls-settings.tsxnew file mode 100644index 000000000..9c65bfcb5--- /dev/null+++ b/frontend/components/models/watsonx-tls-settings.tsx@@ -0,0 +1,106 @@+"use client";++import { AlertTriangle } from "lucide-react";+import { useRef } from "react";+import { LabelWrapper } from "@/components/label-wrapper";+import { Input } from "@/components/ui/input";+import { Label } from "@/components/ui/label";+import { Switch } from "@/components/ui/switch";++const FALSE_VALUES: Record<string, true> = {+ false: true,+ "0": true,+ no: true,+ off: true,+};+const TRUE_VALUES: Record<string, true> = {+ true: true,+ "1": true,+ yes: true,+ on: true,+};++export function WatsonxTlsSettings({+ value,+ onValueChange,+ idPrefix,+}: {+ value?: string;+ onValueChange: (value: string) => void;+ idPrefix: string;+}) {+ const normalized = (value ?? "").trim().toLowerCase();+ const enabled = FALSE_VALUES[normalized] !== true;+ const caPath =+ !normalized ||+ TRUE_VALUES[normalized] === true ||+ FALSE_VALUES[normalized] === true+ ? ""+ : (value ?? "");+ const lastCaPath = useRef(caPath);+ if (caPath) lastCaPath.current = caPath;++ const switchId = `${idPrefix}-tls-verify`;+ const descriptionId = `${switchId}-description`;+ const caPathId = `${idPrefix}-tls-ca-path`;++ return (+ <div className="space-y-4">+ <div className="flex items-center justify-between gap-4 py-1">+ <div className="space-y-1">+ <Label htmlFor={switchId} className="text-mmd font-medium">+ Verify TLS certificates+ </Label>+ <p id={descriptionId} className="text-sm text-muted-foreground">+ Recommended. Validates the cluster certificate before sending+ credentials or model traffic.+ </p>+ </div>+ <Switch+ id={switchId}+ checked={enabled}+ aria-describedby={descriptionId}+ onCheckedChange={(checked) =>+ onValueChange(+ checked ? lastCaPath.current.trim() || "true" : "false",+ )+ }+ />+ </div>++ {enabled ? (+ <LabelWrapper+ id={caPathId}+ label="Custom CA bundle path"+ helperText="Optional backend filesystem path to a mounted CA bundle. Include public roots too if this OpenRAG deployment also calls public providers."+ >+ <Input+ id={caPathId}+ value={caPath}+ placeholder="/etc/ssl/certs/openrag-ca.pem"+ autoComplete="off"+ onChange={(event) => {+ const next = event.target.value;+ if (next) lastCaPath.current = next;+ onValueChange(next || "true");+ }}+ />+ </LabelWrapper>+ ) : (+ <div+ role="alert"+ className="flex gap-2 rounded-md bg-destructive/10 px-3 py-2 text-sm text-destructive"+ >+ <AlertTriangle+ className="mt-0.5 h-4 w-4 shrink-0"+ aria-hidden="true"+ />+ <p>+ Certificate verification is disabled. Credentials and model traffic+ can be intercepted. Use this only for local development.+ </p>+ </div>+ )}+ </div>+ );+}diff --git a/frontend/hooks/useChatStreaming.test.tsx b/frontend/hooks/useChatStreaming.test.tsxindex a5929feb5..1a0dc4526 100644--- a/frontend/hooks/useChatStreaming.test.tsx+++ b/frontend/hooks/useChatStreaming.test.tsx@@ -196,6 +196,25 @@ describe("useChatStreaming", () => {expect(onComplete.mock.calls[0][0].content).toBe("Hello there");});+ it("preserves a burst of deltas delivered in one network chunk", async () => {+ const deltas = Array.from({ length: 150 }, (_, index) =>+ textChunk(`${index},`),+ );+ serveStream([+ deltas.map((chunk) => JSON.stringify(chunk)).join("\n") + "\n",+ ]);+ const { result } = setup();++ const returned = await send(result, { prompt: "p" });++ expect(returned?.content).toBe(+ Array.from({ length: 150 }, (_, index) => `${index},`).join(""),+ );+ expect(console.error).not.toHaveBeenCalledWith(+ expect.stringContaining("Maximum update depth exceeded"),+ );+ });+it("reassembles a JSON object split across two network chunks", async () => {const line = JSON.stringify(textChunk("split"));serveStream([line.slice(0, 12), `${line.slice(12)}\n`]);diff --git a/frontend/hooks/useChatStreaming.ts b/frontend/hooks/useChatStreaming.tsindex 761a23ce5..d7782b3d4 100644--- a/frontend/hooks/useChatStreaming.ts+++ b/frontend/hooks/useChatStreaming.ts@@ -182,6 +182,7 @@ export function useChatStreaming({// Process complete lines (JSON objects)const lines = buffer.split("\n");buffer = lines.pop() || ""; // Keep incomplete line in buffer+ let parsedChunk = false;for (const line of lines) {if (line.trim()) {@@ -203,6 +204,7 @@ export function useChatStreaming({) ||parseOpenRAGChunk(chunk, content);detectImplicitToolCall(chunk, currentFunctionCalls);+ parsedChunk = true;const streamError = extractStreamProviderError(chunk);if (streamError) {@@ -210,27 +212,28 @@ export function useChatStreaming({providerStreamError = streamError;break streamLoop;}-- if (- !controller.signal.aborted &&- thisStreamId === streamIdRef.current- ) {- setStreamingMessage({- role: "assistant",- content: content.value,- functionCalls:- currentFunctionCalls.length > 0- ? [...currentFunctionCalls]- : undefined,- timestamp: new Date(),- isStreaming: true,- });- }} catch (parseError) {console.warn("Failed to parse chunk:", line, parseError);}}}++ if (+ parsedChunk &&+ !controller.signal.aborted &&+ thisStreamId === streamIdRef.current+ ) {+ setStreamingMessage({+ role: "assistant",+ content: content.value,+ functionCalls:+ currentFunctionCalls.length > 0+ ? [...currentFunctionCalls]+ : undefined,+ timestamp: new Date(),+ isStreaming: true,+ });+ }}} finally {reader.releaseLock();diff --git a/src/api/models.py b/src/api/models.pyindex bef65c3b0..86cdbfb11 100644--- a/src/api/models.py+++ b/src/api/models.py@@ -3,7 +3,7 @@ import reimport httpxfrom fastapi import Dependsfrom fastapi.responses import JSONResponse-from pydantic import BaseModel+from pydantic import BaseModel, Fieldfrom api.provider_validation import (is_provider_credential_error,@@ -47,6 +47,11 @@ class IBMBody(BaseModel):project_id: str | None = None+class WatsonxOnPremSpacesBody(BaseModel):+ credentials: dict[str, str] = Field(default_factory=dict)+ auth_method: str | None = None++def _models_error_response(exc: Exception) -> JSONResponse:"""Map model-route failures to client (400) vs upstream (502) vs server (500).@@ -218,6 +223,65 @@ async def get_ibm_models(return _models_error_response(e)+async def get_watsonx_onprem_spaces(+ body: WatsonxOnPremSpacesBody | None = None,+ user: User = Depends(require_permission("providers:write")),+):+ """Deployment spaces visible to the pending watsonx.ai on-prem credentials."""+ from enhancements.providers.watsonx import onprem++ request = body or WatsonxOnPremSpacesBody()+ try:+ config = get_openrag_config()+ stored_config = config.providers.get_provider_config(onprem.PROVIDER_KEY)+ stored_credentials = config.providers.stored_credentials(onprem.PROVIDER_KEY)+ auth_method = request.auth_method or getattr(stored_config, "auth_method", None)+ if auth_method is None:+ auth_method = (+ "zen_api_key" if stored_credentials.get("zen_api_key") else "username_api_key"+ )+ allowed = onprem.credential_fields_for_auth_method(auth_method)+ submitted = {+ name: str(value).strip()+ for name, value in request.credentials.items()+ if name in allowed and str(value).strip()+ }+ target_changed = any(+ name in submitted and submitted[name] != stored_credentials.get(name)+ for name in ("api_base", "ssl_verify")+ )+ base = {} if target_changed else stored_credentials+ credentials = {+ name: value for name, value in {**base, **submitted}.items() if name in allowed+ }+ spaces = await onprem.list_spaces(credentials)+ return JSONResponse(+ {"spaces": spaces},+ headers={"Cache-Control": "no-store"},+ )+ except onprem.SpaceDiscoveryError as exc:+ logger.warning(+ "watsonx.ai on-prem space discovery was rejected",+ status_code=exc.status_code,+ )+ status_code = exc.status_code if exc.status_code in {400, 401, 403, 404} else 502+ if status_code in {401, 403}:+ error = "The cluster rejected the configured credentials."+ elif status_code == 404:+ error = "The cluster deployment-space endpoint was not found."+ elif status_code == 400:+ error = "The cluster rejected the deployment-space request."+ else:+ error = "Unable to list deployment spaces from the cluster."+ return JSONResponse({"error": error}, status_code=status_code)+ except Exception as exc:+ logger.error("Failed to list watsonx.ai on-prem spaces", exc_info=exc)+ return JSONResponse(+ {"error": "Unable to list deployment spaces from the cluster."},+ status_code=500,+ )++async def get_model_providers(user: User = Depends(require_permission("providers:read")),):diff --git a/src/api/provider_validation.py b/src/api/provider_validation.pyindex 67e227692..c9afe1fc1 100644--- a/src/api/provider_validation.py+++ b/src/api/provider_validation.py@@ -727,9 +727,18 @@ async def validate_provider_setup())if probes_the_model:+ runtime_kwargs: dict[str, Any] = {}+ if enhancement is not None:+ from enhancements.providers.registry import runtime_kwargs_for++ runtime_kwargs = runtime_kwargs_for(+ enhancement,+ stored_credentials if stored_credentials is not None else supplied,+ )await _test_litellm_provider(provider=provider_lower,credentials=supplied,+ runtime_kwargs=runtime_kwargs,embedding_model=embedding_model,llm_model=llm_model,)@@ -788,7 +797,8 @@ async def validate_provider_setup(async def _test_litellm_provider(*,provider: str,- credentials: dict[str, str],+ credentials: Mapping[str, Any],+ runtime_kwargs: Mapping[str, Any],embedding_model: str | None,llm_model: str | None,) -> None:@@ -814,6 +824,7 @@ async def _test_litellm_provider(# wrongly-chosen one impossible to change.input=["OpenRAG provider validation"],**credentials,+ **(runtime_kwargs or {}),)returnawait litellm.acompletion(@@ -821,6 +832,7 @@ async def _test_litellm_provider(messages=[{"role": "user", "content": "Reply with OK."}],max_tokens=4,**credentials,+ **(runtime_kwargs or {}),)diff --git a/src/api/settings/endpoints.py b/src/api/settings/endpoints.pyindex 94e998823..5b4d89a6a 100644--- a/src/api/settings/endpoints.py+++ b/src/api/settings/endpoints.py@@ -421,9 +421,18 @@ async def update_settings("watsonx_project_id","ollama_endpoint","provider_credentials",+ "provider_credential_removals","remove_provider_config",]-+ submitted_credentials = {+ _provider_key(name): values+ for name, values in (body.provider_credentials or {}).items()+ }+ removals_by_provider = {+ _provider_key(name): set(fields)+ for name, fields in (body.provider_credential_removals or {}).items()+ }+ provider_update_keys = set(submitted_credentials) | set(removals_by_provider)should_validate = any(getattr(body, field) is not None for field in provider_fields)# Docling VLM settings reuse provider credentials, so they are gated@@ -468,10 +477,11 @@ async def update_settings(# a model-free `lightweight_health_check`, so a bad URL or# token is rejected here rather than on the first chat or# ingest.- for provider, submitted in (body.provider_credentials or {}).items():- provider_key = _provider_key(provider)+ for provider_key in provider_update_keys:+ submitted = submitted_credentials.get(provider_key, {})+ removals = removals_by_provider.get(provider_key, set())credentials = current_config.providers.pending_credentials(- provider_key, submitted+ provider_key, submitted, remove=removals)if provider_key == "azure":auth_method = (body.provider_auth_methods or {}).get(provider_key)@@ -505,7 +515,7 @@ async def update_settings(provider=provider_key,credentials=credentials,stored_credentials=current_config.providers.pending_stored_credentials(- provider_key, submitted+ provider_key, submitted, remove=removals),)@@ -530,12 +540,10 @@ async def update_settings(endpoint = getattr(llm_provider_config, "endpoint", None)project_id = getattr(llm_provider_config, "project_id", None)llm_provider_key = _provider_key(llm_provider)- submitted_credentials = {- _provider_key(name): values- for name, values in (body.provider_credentials or {}).items()- }credentials = current_config.providers.pending_credentials(- llm_provider_key, submitted_credentials.get(llm_provider_key, {})+ llm_provider_key,+ submitted_credentials.get(llm_provider_key, {}),+ remove=removals_by_provider.get(llm_provider_key, set()),)api_key = credentials.get("api_key", api_key)endpoint = credentials.get("api_base", endpoint)@@ -559,7 +567,9 @@ async def update_settings(project_id=project_id,credentials=credentials,stored_credentials=current_config.providers.pending_stored_credentials(- llm_provider_key, submitted_credentials.get(llm_provider_key, {})+ llm_provider_key,+ submitted_credentials.get(llm_provider_key, {}),+ remove=removals_by_provider.get(llm_provider_key, set()),),)logger.info(f"LLM provider validation successful for {llm_provider}")@@ -587,14 +597,11 @@ async def update_settings(endpoint = getattr(embedding_provider_config, "endpoint", None)project_id = getattr(embedding_provider_config, "project_id", None)embedding_provider_key = _provider_key(embedding_provider)- submitted_credentials = {- _provider_key(name): values- for name, values in (body.provider_credentials or {}).items()- }credentials = current_config.providers.pending_credentials(embedding_provider_key,submitted_credentials.get(embedding_provider_key, {}),kind="embedding",+ remove=removals_by_provider.get(embedding_provider_key, set()),)api_key = credentials.get("api_key", api_key)endpoint = credentials.get("api_base", endpoint)@@ -620,6 +627,7 @@ async def update_settings(stored_credentials=current_config.providers.pending_stored_credentials(embedding_provider_key,submitted_credentials.get(embedding_provider_key, {}),+ remove=removals_by_provider.get(embedding_provider_key, set()),),)logger.info(@@ -901,11 +909,12 @@ async def update_settings(# Update provider-specific settingsprovider_updated = False- for provider, credentials in (body.provider_credentials or {}).items():+ for provider in provider_update_keys:working_config.providers.set_credentials(provider,- credentials,- auth_method=(body.provider_auth_methods or {}).get(_provider_key(provider)),+ submitted_credentials.get(provider, {}),+ auth_method=(body.provider_auth_methods or {}).get(provider),+ remove=removals_by_provider.get(provider, set()),)config_updated = Trueprovider_updated = True@@ -1155,7 +1164,7 @@ async def onboarding(log_bootstrap_env(logger, "onboarding")# Get current configuration- current_config = get_openrag_config()+ current_config = copy.deepcopy(get_openrag_config())# Warn if config was already edited (onboarding being re-run)if current_config.edited:@@ -1251,11 +1260,20 @@ async def onboarding(current_config.providers.ollama.configured = Trueconfig_updated = True- for provider, credentials in (body.provider_credentials or {}).items():+ submitted_credentials = {+ _provider_key(name): values+ for name, values in (body.provider_credentials or {}).items()+ }+ removals_by_provider = {+ _provider_key(name): set(fields)+ for name, fields in (body.provider_credential_removals or {}).items()+ }+ for provider in set(submitted_credentials) | set(removals_by_provider):current_config.providers.set_credentials(provider,- credentials,- auth_method=(body.provider_auth_methods or {}).get(_provider_key(provider)),+ submitted_credentials.get(provider, {}),+ auth_method=(body.provider_auth_methods or {}).get(provider),+ remove=removals_by_provider.get(provider, set()),)config_updated = Truediff --git a/src/api/settings/models.py b/src/api/settings/models.pyindex 23a1dc639..ecf161ed1 100644--- a/src/api/settings/models.py+++ b/src/api/settings/models.py@@ -51,6 +51,7 @@ class SettingsUpdateBody(BaseModel):remove_anthropic_config: bool | None = Noneremove_watsonx_config: bool | None = Noneprovider_credentials: dict[str, dict[str, str]] | None = None+ provider_credential_removals: dict[str, list[str]] | None = Noneprovider_auth_methods: dict[str, str] | None = Noneremove_provider_config: str | None = None# Explicit confirmation that the caller accepts removing a provider whose@@ -71,6 +72,7 @@ class OnboardingBody(BaseModel):watsonx_project_id: str | None = Field(None, min_length=1)ollama_endpoint: str | None = Field(None, min_length=1)provider_credentials: dict[str, dict[str, str]] | None = None+ provider_credential_removals: dict[str, list[str]] | None = Noneprovider_auth_methods: dict[str, str] | None = Nonediff --git a/src/app/router_app.py b/src/app/router_app.pyindex 126626eba..1af32bd82 100644--- a/src/app/router_app.py+++ b/src/app/router_app.py@@ -24,6 +24,8 @@ from app.middleware import RequestLoggingMiddlewarefrom config.settings import (ACCESS_LOG_ENABLED,INGEST_CALLBACK_PATH,+ LANGFLOW_CONNECT_TIMEOUT,+ LANGFLOW_TIMEOUT,OPENRAG_BACKEND_ROUTER_HOST,OPENRAG_BACKEND_ROUTER_PORT,OPENRAG_BACKEND_ROUTER_UPSTREAM_URL,@@ -50,7 +52,14 @@ _LLM_PROXY_UPSTREAM_PATHS = {# hop-by-hop headers, etc.) is dropped so the router cannot be abused as an open# proxy. The ingest token travels in either Authorization or the custom header._FORWARDED_HEADERS = ("authorization", "x-openrag-ingest-token", "content-type", "accept")-_UPSTREAM_TIMEOUT = httpx.Timeout(60.0)+_INGEST_UPSTREAM_TIMEOUT = httpx.Timeout(60.0)+_LLM_UPSTREAM_TIMEOUT = httpx.Timeout(+ timeout=LANGFLOW_TIMEOUT,+ connect=LANGFLOW_CONNECT_TIMEOUT,+ read=LANGFLOW_TIMEOUT,+ write=LANGFLOW_CONNECT_TIMEOUT,+ pool=LANGFLOW_CONNECT_TIMEOUT,+)async def _proxy_ingest_chunks(request: Request) -> Response:@@ -61,7 +70,7 @@ async def _proxy_ingest_chunks(request: Request) -> Response:key: value for key, value in request.headers.items() if key.lower() in _FORWARDED_HEADERS}try:- async with httpx.AsyncClient(timeout=_UPSTREAM_TIMEOUT) as client:+ async with httpx.AsyncClient(timeout=_INGEST_UPSTREAM_TIMEOUT) as client:upstream = await client.post(upstream_url, content=body, headers=headers)except httpx.HTTPError as e:logger.error(@@ -102,7 +111,7 @@ async def _proxy_llm_request(request: Request) -> Response:headers = {key: value for key, value in request.headers.items() if key.lower() in _FORWARDED_HEADERS}- client = httpx.AsyncClient(timeout=_UPSTREAM_TIMEOUT)+ client = httpx.AsyncClient(timeout=_LLM_UPSTREAM_TIMEOUT)try:upstream = await client.send(client.build_request(diff --git a/src/app/routes/internal.py b/src/app/routes/internal.pyindex 22a435e84..a2ca7f93b 100644--- a/src/app/routes/internal.py+++ b/src/app/routes/internal.py@@ -459,6 +459,12 @@ def register_internal_routes(app: FastAPI):"/models/ollama", models.get_ollama_models, methods=["GET"], tags=["internal"])app.add_api_route("/models/ibm", models.get_ibm_models, methods=["POST"], tags=["internal"])+ app.add_api_route(+ "/models/watsonx_onprem/spaces",+ models.get_watsonx_onprem_spaces,+ methods=["POST"],+ tags=["internal"],+ )app.add_api_route("/models/catalog", models.get_model_catalog, methods=["GET"], tags=["internal"])diff --git a/src/config/config_manager.py b/src/config/config_manager.pyindex 583f5920f..2ddc12d15 100644--- a/src/config/config_manager.py+++ b/src/config/config_manager.py@@ -232,20 +232,25 @@ class ProvidersConfig:return self.custom.get(provider_lower, GenericProviderConfig())def set_credentials(- self, provider: str, credentials: dict[str, str], *, auth_method: str | None = None+ self,+ provider: str,+ credentials: dict[str, str],+ *,+ auth_method: str | None = None,+ remove: set[str] | None = None,) -> None:- """Upsert arbitrary LiteLLM credentials while preserving legacy config."""+ """Upsert credentials and apply explicitly requested field removals."""key = provider.strip().lower()clean = {str(name): str(value).strip()for name, value in credentials.items()if str(name).strip() and str(value).strip()}- if not clean:- # Every submitted value was blank. Creating the entry anyway would- # register a provider that reports `configured` with zero- # credentials, which then satisfies `any_configured()` and can be- # picked as a fallback provider and called with no key at all.+ removals = {str(name).strip() for name in remove or set() if str(name).strip()}+ if not clean and not removals:+ # Blank values remain "leave unchanged" because secret fields are+ # intentionally not echoed to forms. Deletion is an explicit,+ # separate operation so an empty password cannot erase a secret.returnprevious = self.custom.get(key, GenericProviderConfig())if key == "azure" and auth_method:@@ -267,19 +272,17 @@ class ProvidersConfig:}previous.auth_method = auth_methodif key == "watsonx_onprem" and auth_method:- methods = {- "username_api_key": {"username", "api_key"},- "zen_api_key": {"zen_api_key"},- }- active = methods.get(auth_method)- if active is None:- raise ValueError(f"Unknown watsonx.ai on-prem authentication method: {auth_method}")+ from enhancements.providers.watsonx.onprem import (+ credential_fields_for_auth_method,+ )++ allowed = credential_fields_for_auth_method(auth_method)previous.credentials = {- name: value- for name, value in previous.credentials.items()- if name in {"api_base", "space_id", "project_id"} or name in active+ name: value for name, value in previous.credentials.items() if name in allowed}previous.auth_method = auth_method+ for name in removals:+ previous.credentials.pop(name, None)previous.credentials.update(clean)# Complete against the form's required fields, not merely non-empty:# a submission of just `ssl_verify` must not make a provider look@@ -321,16 +324,18 @@ class ProvidersConfig:return dict(self.custom.get(provider.strip().lower(), GenericProviderConfig()).credentials)def pending_stored_credentials(- self, provider: str, submitted: dict[str, str] | None = None+ self,+ provider: str,+ submitted: dict[str, str] | None = None,+ *,+ remove: set[str] | None = None,) -> dict[str, str]:- """`stored_credentials()` as it would read once `submitted` is saved.-- The untranslated counterpart of `pending_credentials()`, for the pre-save- health check of a provider whose check needs every field — both of an- OpenShift AI deployment's endpoints — rather than the one LiteLLM call- the translated form is narrowed to.- """- return {**self.stored_credentials(provider), **_clean_submitted(submitted)}+ """`stored_credentials()` as it would read after one pending update."""+ pending = self.stored_credentials(provider)+ for name in remove or set():+ pending.pop(name, None)+ pending.update(_clean_submitted(submitted))+ return pendingdef pending_credentials(self,@@ -338,6 +343,7 @@ class ProvidersConfig:submitted: dict[str, str] | None = None,*,kind: str = "chat",+ remove: set[str] | None = None,) -> dict[str, Any]:"""LiteLLM kwargs for `provider` as it would be once `submitted` is saved.@@ -359,9 +365,11 @@ class ProvidersConfig:clean = _clean_submitted(submitted)enhancement = get_provider_enhancement(key)if enhancement:- stored = self.custom.get(key, GenericProviderConfig()).credentials- return credentials_for(enhancement, {**stored, **clean}, kind)+ stored = self.pending_stored_credentials(key, clean, remove=remove)+ return credentials_for(enhancement, stored, kind)values = self.credential_values(key, kind=kind)+ for name in remove or set():+ values.pop(name, None)values.update(clean)return values@@ -815,6 +823,32 @@ class ConfigManager:if os.getenv("WATSONX_PROJECT_ID"):config_data["providers"]["watsonx"]["project_id"] = os.getenv("WATSONX_PROJECT_ID")+ # IBM watsonx.ai on-prem (Cloud Pak for Data / Software Hub).+ onprem_credentials = {+ "api_base": os.getenv("WATSONX_ONPREM_ENDPOINT"),+ "username": os.getenv("WATSONX_ONPREM_USERNAME"),+ "api_key": os.getenv("WATSONX_ONPREM_API_KEY"),+ "zen_api_key": os.getenv("WATSONX_ONPREM_ZEN_API_KEY"),+ "space_id": os.getenv("WATSONX_ONPREM_SPACE_ID"),+ "project_id": os.getenv("WATSONX_ONPREM_PROJECT_ID"),+ "ssl_verify": os.getenv("WATSONX_ONPREM_TLS_VERIFY"),+ }+ if any(onprem_credentials.values()):+ self._seed_custom_provider_credentials(+ config_data,+ "watsonx_onprem",+ onprem_credentials,+ required=("api_base",),+ )+ entry = config_data["providers"]["custom"]["watsonx_onprem"]+ stored = entry["credentials"]+ has_zen = bool(stored.get("zen_api_key"))+ entry["auth_method"] = "zen_api_key" if has_zen else "username_api_key"+ entry["configured"] = bool(+ stored.get("api_base")+ and (has_zen or (stored.get("username") and stored.get("api_key")))+ )+# Ollama provider settingsif os.getenv("OLLAMA_ENDPOINT"):config_data["providers"]["ollama"]["endpoint"] = os.getenv("OLLAMA_ENDPOINT")diff --git a/src/services/llm_gateway.py b/src/services/llm_gateway.pyindex 530951b01..3dc42ea41 100644--- a/src/services/llm_gateway.py+++ b/src/services/llm_gateway.py@@ -198,6 +198,27 @@ def provider_credentials(return credentials+def _provider_runtime_kwargs(provider: str, config=None) -> dict[str, Any]:+ """Non-serializable transport kwargs for the immediate LiteLLM invocation."""+ cfg = config or _get_config()+ prov = cfg.providers+ if not hasattr(prov, "credential_values"):+ return {}++ from enhancements.providers.registry import get, runtime_kwargs_for++ key = (provider or "").strip().lower()+ enhancement = get(key)+ if enhancement is None:+ return {}+ stored = (+ prov.stored_credentials(key)+ if hasattr(prov, "stored_credentials")+ else prov.credential_values(key)+ )+ return runtime_kwargs_for(enhancement, stored)++def resolve_call(model: str | None,*,@@ -640,6 +661,7 @@ async def chat_completions("""OpenAI `POST /v1/chat/completions`. Streams SSE lines when `stream` is true."""cfg = config or _get_config()litellm_model, provider, credentials = resolve_call(body.get("model"), kind="chat", config=cfg)+ runtime_kwargs = _provider_runtime_kwargs(provider, cfg)kwargs = {key: body[key] for key in _LITELLM_FORWARDED_PARAMS if key in body}stream = bool(body.get("stream"))if litellm_model in _TOOLS_NEED_REASONING_OFF:@@ -661,6 +683,7 @@ async def chat_completions(# provider's capabilities instead of failing the request.drop_params=True,**credentials,+ **runtime_kwargs,**kwargs,)@@ -1017,6 +1040,9 @@ def _log_stream_shape(tally: _StreamTally, provider: str, model: str) -> None:logger.debug("Could not summarise stream shape", exc_info=True)+_WATSONX_ONPREM_EMBEDDING_BATCH_SIZE = 32++def _embedding_input(value: Any) -> Any:"""OpenAI's `input`, in the shape every provider behind us accepts.@@ -1036,14 +1062,56 @@ async def embeddings(body: Mapping[str, Any], *, config=None) -> dict[str, Any]:litellm_model, provider, credentials = resolve_call(body.get("model"), kind="embedding", config=cfg)+ runtime_kwargs = _provider_runtime_kwargs(provider, cfg)+ embedding_input = _embedding_input(body.get("input"))+ should_batch = (+ provider == "watsonx_onprem"+ and isinstance(embedding_input, list)+ and len(embedding_input) > _WATSONX_ONPREM_EMBEDDING_BATCH_SIZE+ and not all(isinstance(item, int) for item in embedding_input)+ )try:import litellm- result = await litellm.aembedding(- model=litellm_model,- input=_embedding_input(body.get("input")),- **credentials,- )+ if not should_batch:+ result = await litellm.aembedding(+ model=litellm_model,+ input=embedding_input,+ **credentials,+ **runtime_kwargs,+ )+ response = _to_openai_dict(result)+ else:+ response = {}+ data: list[dict[str, Any]] = []+ usage: dict[str, int | float] = {}+ for offset in range(+ 0,+ len(embedding_input),+ _WATSONX_ONPREM_EMBEDDING_BATCH_SIZE,+ ):+ batch = embedding_input[offset : offset + _WATSONX_ONPREM_EMBEDDING_BATCH_SIZE]+ result = await litellm.aembedding(+ model=litellm_model,+ input=batch,+ **credentials,+ **runtime_kwargs,+ )+ payload = _to_openai_dict(result)+ if not response:+ response = {+ key: value for key, value in payload.items() if key not in {"data", "usage"}+ }+ for position, item in enumerate(payload.get("data", [])):+ merged_item = dict(item)+ merged_item["index"] = offset + int(item.get("index", position))+ data.append(merged_item)+ for key, value in payload.get("usage", {}).items():+ if isinstance(value, (int, float)):+ usage[key] = usage.get(key, 0) + value+ response["data"] = data+ if usage:+ response["usage"] = usageexcept LlmGatewayError:raiseexcept Exception as exc:@@ -1057,4 +1125,4 @@ async def embeddings(body: Mapping[str, Any], *, config=None) -> dict[str, Any]:detail=detail,) from excprovider_error_log.record_success(provider, "embedding")- return _to_openai_dict(result)+ return responsediff --git a/src/services/model_catalog.py b/src/services/model_catalog.pyindex ed1575ae1..2fceffd9f 100644--- a/src/services/model_catalog.py+++ b/src/services/model_catalog.py@@ -467,7 +467,14 @@ async def refresh_live_models() -> None:)continueif credentials:- await enhancement.fetch_models(credentials)+ try:+ await enhancement.fetch_models(credentials)+ except Exception:+ logger.warning(+ "Provider model discovery failed; keeping configured fallback models",+ provider=enhancement.PROVIDER_KEY,+ exc_info=True,+ )def supported_provider_keys() -> frozenset[str]:diff --git a/tests/unit/api/test_settings_endpoints.py b/tests/unit/api/test_settings_endpoints.pyindex ba72b11a4..7e1c04b9e 100644--- a/tests/unit/api/test_settings_endpoints.py+++ b/tests/unit/api/test_settings_endpoints.py@@ -406,3 +406,118 @@ def test_settings_update_body_accepts_configurable_vlm_providers(provider):def test_settings_update_body_rejects_malformed_vlm_provider(provider):with pytest.raises(ValidationError):SettingsUpdateBody(vlm_provider=provider)+++async def test_update_settings_persists_case_insensitive_removal_only_request():+ from api.settings.endpoints import update_settings+ from config.config_manager import OpenRAGConfig++ config = OpenRAGConfig.from_dict({})+ config.edited = True+ config.providers.set_credentials(+ "gemini",+ {"api_key": "secret", "api_base": "https://gemini.example.com"},+ )+ body = SettingsUpdateBody(+ provider_credential_removals={"Gemini": ["api_key"]},+ )+ rbac = MagicMock()+ rbac.has_permission = AsyncMock(return_value=True)++ with (+ patch("api.settings.endpoints.get_openrag_config", return_value=config),+ patch(+ "api.settings.endpoints.config_manager.save_config_file",+ return_value=True,+ ) as save,+ patch(+ "api.settings.endpoints.clients.refresh_patched_client",+ new_callable=AsyncMock,+ ),+ ):+ response = await update_settings(+ body=body,+ session_manager=AsyncMock(),+ user=MagicMock(spec=User),+ models_service=MagicMock(),+ rbac=rbac,+ )+ assert getattr(response, "status_code", 200) == 200+ saved_config = save.call_args.args[0]+ assert saved_config.providers.stored_credentials("gemini") == {+ "api_base": "https://gemini.example.com"+ }+++async def test_removal_only_provider_update_requires_provider_write_permission():+ from api.settings.endpoints import update_settings+ from config.config_manager import OpenRAGConfig++ config = OpenRAGConfig.from_dict({})+ config.edited = True+ rbac = MagicMock()+ rbac.has_permission = AsyncMock(return_value=False)+ rbac.audit_denied = AsyncMock()++ with (+ patch("api.settings.endpoints.get_openrag_config", return_value=config),+ patch("api.settings.endpoints.is_rbac_enforced", return_value=True),+ pytest.raises(HTTPException) as exc_info,+ ):+ await update_settings(+ body=SettingsUpdateBody(+ provider_credential_removals={"gemini": ["api_key"]},+ ),+ session_manager=AsyncMock(),+ user=MagicMock(spec=User, db_user_id="user-1", user_id="user-1"),+ models_service=MagicMock(),+ rbac=rbac,+ )++ assert exc_info.value.status_code == 403+ rbac.audit_denied.assert_awaited_once_with("user-1", "providers:write")+++async def test_failed_onboarding_validation_does_not_mutate_cached_config():+ from api.settings.endpoints import onboarding+ from config.config_manager import OpenRAGConfig++ config = OpenRAGConfig.from_dict({})+ config.agent.llm_provider = "openai"+ config.agent.llm_model = "old-model"+ body = OnboardingBody(+ llm_provider="openai",+ llm_model="new-model",+ openai_api_key="sk-new",+ )++ with (+ patch("api.settings.endpoints.get_openrag_config", return_value=config),+ patch(+ "api.settings.endpoints.TelemetryClient.send_event",+ new_callable=AsyncMock,+ ),+ patch(+ "api.settings.endpoints.validate_provider_setup",+ new_callable=AsyncMock,+ side_effect=Exception("validation failed"),+ ),+ ):+ response = await onboarding(+ body=body,+ flows_service=MagicMock(),+ session_manager=AsyncMock(),+ document_service=MagicMock(),+ models_service=MagicMock(),+ task_service=MagicMock(),+ langflow_file_service=MagicMock(),+ knowledge_filter_service=MagicMock(),+ user=MagicMock(spec=User),+ )++ assert response.status_code == 400+ assert config.agent.llm_model == "old-model"+ assert config.providers.openai.api_key == ""diff --git a/tests/unit/config/test_generic_provider_config.py b/tests/unit/config/test_generic_provider_config.pyindex 2cf3f1202..b95396702 100644--- a/tests/unit/config/test_generic_provider_config.py+++ b/tests/unit/config/test_generic_provider_config.py@@ -230,6 +230,57 @@ def test_rhoai_tls_env_alone_reaches_a_stored_config(monkeypatch, tmp_path):}+def _clear_watsonx_onprem_env(monkeypatch):+ for name in (+ "WATSONX_ONPREM_ENDPOINT",+ "WATSONX_ONPREM_USERNAME",+ "WATSONX_ONPREM_API_KEY",+ "WATSONX_ONPREM_ZEN_API_KEY",+ "WATSONX_ONPREM_SPACE_ID",+ "WATSONX_ONPREM_PROJECT_ID",+ "WATSONX_ONPREM_TLS_VERIFY",+ ):+ monkeypatch.delenv(name, raising=False)+++def test_watsonx_onprem_env_seeds_credentials_and_tls(monkeypatch, tmp_path):+ from config.config_manager import ConfigManager++ _clear_rhoai_env(monkeypatch)+ _clear_watsonx_onprem_env(monkeypatch)+ monkeypatch.setenv("WATSONX_ONPREM_ENDPOINT", "https://cpd.example.com")+ monkeypatch.setenv("WATSONX_ONPREM_USERNAME", "cpduser")+ monkeypatch.setenv("WATSONX_ONPREM_API_KEY", "secret")+ monkeypatch.setenv("WATSONX_ONPREM_SPACE_ID", "space-1")+ monkeypatch.setenv("WATSONX_ONPREM_TLS_VERIFY", "/etc/openrag/cpd-ca.pem")++ config = ConfigManager(config_file=tmp_path / "config.yaml").load_config()+ provider = config.providers.custom["watsonx_onprem"]++ assert provider.configured is True+ assert provider.auth_method == "username_api_key"+ assert config.providers.stored_credentials("watsonx_onprem") == {+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "secret",+ "space_id": "space-1",+ "ssl_verify": "/etc/openrag/cpd-ca.pem",+ }+++def test_watsonx_onprem_env_without_complete_auth_stays_unconfigured(monkeypatch, tmp_path):+ from config.config_manager import ConfigManager++ _clear_rhoai_env(monkeypatch)+ _clear_watsonx_onprem_env(monkeypatch)+ monkeypatch.setenv("WATSONX_ONPREM_ENDPOINT", "https://cpd.example.com")+ monkeypatch.setenv("WATSONX_ONPREM_TLS_VERIFY", "false")++ config = ConfigManager(config_file=tmp_path / "config.yaml").load_config()++ assert config.providers.custom["watsonx_onprem"].configured is False++def test_rhoai_env_is_ignored_once_settings_have_been_edited(monkeypatch, tmp_path):"""The trap worth an explicit test: the first Settings save sets`config.edited` and silently freezes every environment override, which on adiff --git a/tests/unit/enhancements/providers/watsonx/test_onprem.py b/tests/unit/enhancements/providers/watsonx/test_onprem.pyindex 2cdb29ddb..b5dd44d09 100644--- a/tests/unit/enhancements/providers/watsonx/test_onprem.py+++ b/tests/unit/enhancements/providers/watsonx/test_onprem.py@@ -1,5 +1,6 @@"""watsonx.ai on a Cloud Pak for Data cluster, routed through LiteLLM's watsonx provider."""+import osfrom types import SimpleNamespacefrom typing import Any@@ -50,13 +51,21 @@ def test_a_half_filled_form_yields_no_zen_key(username, api_key) -> None:assert watsonx_onprem.zen_api_key(username, api_key) == ""-def test_username_never_reaches_litellm() -> None:- """LiteLLM forwards kwargs it does not recognise, so a stray field lands in the body."""- credentials = watsonx_onprem.litellm_credentials(- {"api_base": "https://cpd.example.com", "username": "cpduser", "api_key": "APIKEY"}- )+def test_username_never_reaches_serializable_credentials() -> None:+ """Credential data stays serializable; runtime transport remains call-local."""+ stored = {+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "APIKEY",+ }+ credentials = watsonx_onprem.litellm_credentials(stored)+ client = watsonx_onprem.litellm_runtime_kwargs(stored)["client"]assert "username" not in credentials+ assert "ssl_verify" not in credentials+ assert "client" not in credentials+ assert client.ssl_verify is True+ assert client is watsonx_onprem.litellm_runtime_kwargs(stored)["client"]assert credentials["api_base"] == "https://cpd.example.com"# Both, and the same value: the embeddings path refuses the call outright# when api_key is unset, and builds its auth header from zen_api_key.@@ -107,14 +116,28 @@ def test_a_deployment_scope_is_passed_through_untouched() -> None:assert credentials["project_id"] == "proj-1"-def test_credential_values_translates_the_stored_form() -> None:+def test_credential_values_translates_the_stored_form_as_data() -> None:+ from utils import provider_health_cache+providers = _providers(api_base="https://cpd.example.com", username="cpduser", api_key="APIKEY")- assert providers.credential_values(PROVIDER) == {+ credentials = providers.credential_values(PROVIDER)+ assert credentials == {"api_base": "https://cpd.example.com","zen_api_key": "Y3BkdXNlcjpBUElLRVk=","api_key": "Y3BkdXNlcjpBUElLRVk=",}+ provider_health_cache.cache_key(+ provider=PROVIDER,+ embedding_provider=PROVIDER,+ test_completion=False,+ llm_model="model",+ embedding_model="embedding",+ endpoint=None,+ project_id=None,+ api_key=None,+ credentials=credentials,+ )def test_pending_credentials_rebuilds_the_zen_key_from_a_submitted_change() -> None:@@ -149,6 +172,7 @@ def test_the_gateway_routes_it_as_watsonx() -> None:# The OpenRAG key is what the caller and the credential store still see.assert provider == PROVIDERassert credentials["zen_api_key"] == "Y3BkdXNlcjpBUElLRVk="+ assert "client" not in credentialsdef test_the_alias_is_routable_so_ids_are_not_billed_to_the_default_provider() -> None:@@ -182,7 +206,9 @@ def test_the_settings_form_asks_for_cluster_credentials_not_ibm_cloud_ones() ->"zen_api_key","space_id","project_id",+ "ssl_verify",}+ assert fields["ssl_verify"]["default_value"] == "true"assert model_catalog.secret_field_keys(PROVIDER) == {"api_key", "zen_api_key"}@@ -385,18 +411,259 @@ async def test_health_reports_a_rejected_zen_key_as_a_credential_problem(monkeypassert is_provider_credential_error(str(excinfo.value))-def test_the_health_probe_uses_the_same_tls_setting_as_real_traffic(monkeypatch) -> None:- """`SSL_CERT_FILE`/`SSL_VERIFY` are read by LiteLLM, not by httpx.-- Without this the banner could sit red on a certificate error while chat- through the gateway works, or the reverse.- """+def test_tls_setting_is_scoped_to_the_onprem_provider(monkeypatch) -> None:+ """The stored value wins without mutating LiteLLM's process-wide setting."""monkeypatch.setenv("SSL_VERIFY", "false")- assert watsonx_onprem.ssl_verify() is False- monkeypatch.setenv("SSL_VERIFY", "true")- monkeypatch.delenv("SSL_CERT_FILE", raising=False)- assert watsonx_onprem.ssl_verify() is True+ secure = watsonx_onprem.litellm_runtime_kwargs(+ {+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "APIKEY",+ "ssl_verify": "true",+ }+ )+ insecure = watsonx_onprem.litellm_runtime_kwargs(+ {+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "APIKEY",+ "ssl_verify": "false",+ }+ )++ assert secure["client"].ssl_verify is True+ assert insecure["client"].ssl_verify is False+ assert "ssl_verify" not in secure+ assert "ssl_verify" not in insecure+ assert watsonx_onprem.ssl_verify({"ssl_verify": "true"}) is True+ assert watsonx_onprem.ssl_verify({"ssl_verify": "false"}) is False+ assert watsonx_onprem.ssl_verify({}) is True+ assert os.environ["SSL_VERIFY"] == "false"+++def test_custom_ca_bundle_must_exist(tmp_path) -> None:+ ca = tmp_path / "openrag-ca.pem"+ ca.write_text("combined bundle", encoding="utf-8")++ assert watsonx_onprem.resolve_ssl_verify(str(ca)) == str(ca)+ with pytest.raises(ValueError, match="CA bundle path is not usable"):+ watsonx_onprem.resolve_ssl_verify(str(tmp_path / "missing.pem"))+++async def test_missing_ca_bundle_does_not_break_the_shared_model_catalog(tmp_path) -> None:+ models = await watsonx_onprem.fetch_models(+ {+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "APIKEY",+ "ssl_verify": str(tmp_path / "missing.pem"),+ }+ )++ assert models is None+++async def test_health_check_uses_the_saved_tls_policy(monkeypatch) -> None:+ seen: dict[str, Any] = {}++ class _Client:+ def __init__(self, *, verify):+ seen["verify"] = verify++ async def __aenter__(self):+ return self++ async def __aexit__(self, *exc):+ return False++ async def request(self, method, url, **kwargs):+ return SimpleNamespace(status_code=200, text="{}", json=lambda: {"resources": []})++ monkeypatch.setattr("httpx.AsyncClient", _Client)++ await watsonx_onprem.lightweight_health_check(+ {+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "APIKEY",+ "ssl_verify": "false",+ }+ )++ assert seen["verify"] is False+++async def test_space_listing_rejects_cleartext_credentials() -> None:+ with pytest.raises(ValueError, match="must use HTTPS"):+ await watsonx_onprem.list_spaces(+ {+ "api_base": "http://cpd.example.com",+ "username": "cpduser",+ "api_key": "APIKEY",+ }+ )+++async def test_space_listing_uses_bearer_tls_and_rebases_pagination(monkeypatch) -> None:+ seen: dict[str, Any] = {"requests": []}+ responses = [+ {"accessToken": "cpd-access-token"},+ {+ "resources": [+ {"metadata": {"id": "space-1", "name": "Production"}},+ {"metadata": {"id": "space-1", "name": "Duplicate"}},+ ],+ "next": {"href": "https://internal-cpd/v2/spaces?version=2024-03-13&start=2"},+ },+ {+ "resources": [+ {"metadata": {"id": "space-2", "name": "Development"}},+ ]+ },+ ]++ class _Client:+ def __init__(self, *, verify, timeout):+ seen["verify"] = verify+ seen["timeout"] = timeout++ async def __aenter__(self):+ return self++ async def __aexit__(self, *exc):+ return False++ async def request(self, method, url, **kwargs):+ seen["requests"].append((method, url, kwargs))+ body = responses.pop(0)+ return SimpleNamespace(status_code=200, text="{}", json=lambda: body)++ monkeypatch.setattr("httpx.AsyncClient", _Client)++ spaces = await watsonx_onprem.list_spaces(+ {+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "APIKEY",+ }+ )++ assert spaces == [+ {"id": "space-1", "name": "Production"},+ {"id": "space-2", "name": "Development"},+ ]+ assert seen["verify"] is True+ assert [(method, url) for method, url, _ in seen["requests"]] == [+ ("POST", "https://cpd.example.com/icp4d-api/v1/authorize"),+ ("GET", "https://cpd.example.com/v2/spaces"),+ ("GET", "https://cpd.example.com/v2/spaces?version=2024-03-13&start=2"),+ ]+ assert seen["requests"][0][2]["json"] == {+ "username": "cpduser",+ "api_key": "APIKEY",+ }+ assert all(+ kwargs["headers"]["Authorization"] == "Bearer cpd-access-token"+ for _, _, kwargs in seen["requests"][1:]+ )+++async def test_space_listing_exchanges_a_pasted_zen_key_for_bearer(monkeypatch) -> None:+ requests: list[tuple[str, str, dict[str, Any]]] = []+ responses = [+ (200, {"token": "cpd-access-token"}),+ (+ 200,+ {+ "resources": [+ {"metadata": {"id": "space-1", "name": "Production"}},+ ]+ },+ ),+ ]++ class _Client:+ def __init__(self, *, verify, timeout):+ assert verify is True++ async def __aenter__(self):+ return self++ async def __aexit__(self, *exc):+ return False++ async def request(self, method, url, **kwargs):+ requests.append((method, url, kwargs))+ status_code, body = responses.pop(0)+ return SimpleNamespace(+ status_code=status_code,+ text=str(body),+ json=lambda: body,+ )++ monkeypatch.setattr("httpx.AsyncClient", _Client)++ spaces = await watsonx_onprem.list_spaces(+ {+ "api_base": "https://cpd.example.com",+ "zen_api_key": "Y3BkdXNlcjpBUElLRVk=",+ }+ )++ assert spaces == [{"id": "space-1", "name": "Production"}]+ assert [(method, url) for method, url, _ in requests] == [+ ("POST", "https://cpd.example.com/icp4d-api/v1/authorize"),+ ("GET", "https://cpd.example.com/v2/spaces"),+ ]+ assert requests[0][2]["json"] == {+ "username": "cpduser",+ "api_key": "APIKEY",+ }+ assert requests[1][2]["headers"]["Authorization"] == "Bearer cpd-access-token"+++def test_auth_method_changes_preserve_the_tls_policy() -> None:+ providers = _providers(+ api_base="https://cpd.example.com",+ username="cpduser",+ api_key="APIKEY",+ ssl_verify="false",+ )++ providers.set_credentials(+ PROVIDER,+ {"zen_api_key": "cHJlOmVuY29kZWQ="},+ auth_method="zen_api_key",+ )++ assert providers.stored_credentials(PROVIDER) == {+ "api_base": "https://cpd.example.com",+ "zen_api_key": "cHJlOmVuY29kZWQ=",+ "ssl_verify": "false",+ }+++def test_cleared_optional_credential_is_removed() -> None:+ providers = _providers(+ api_base="https://cpd.example.com",+ username="cpduser",+ api_key="APIKEY",+ space_id="deployment-space",+ )+ providers.set_credentials(+ PROVIDER,+ {"api_base": "https://cpd.example.com"},+ auth_method="username_api_key",+ remove={"space_id"},+ )++ assert "space_id" not in providers.stored_credentials(PROVIDER)@pytest.mark.asyncio@@ -719,6 +986,7 @@ async def test_validating_an_embedding_model_sends_a_list_not_a_string(monkeypat)assert isinstance(sent["input"], list), sent["input"]+ assert "client" in sent@pytest.mark.asyncio@@ -752,6 +1020,7 @@ async def test_a_client_sending_a_bare_string_input_still_embeds(monkeypatch) ->)assert sent["input"] == ["hello"]+ assert "client" in sentdef test_a_token_array_input_is_left_alone() -> None:diff --git a/tests/unit/services/test_llm_gateway.py b/tests/unit/services/test_llm_gateway.pyindex a8be49969..67b4d166c 100644--- a/tests/unit/services/test_llm_gateway.py+++ b/tests/unit/services/test_llm_gateway.py@@ -197,6 +197,37 @@ def test_indexed_space_routes_through_its_configured_generic_provider():assert credentials == {"api_key": "gemini-secret"}+def test_onprem_resolved_credentials_remain_json_serializable():+ providers = ProvidersConfig(+ openai=OpenAIConfig(),+ anthropic=AnthropicConfig(),+ watsonx=WatsonXConfig(),+ ollama=OllamaConfig(),+ custom={+ "watsonx_onprem": GenericProviderConfig(+ credentials={+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "secret",+ "ssl_verify": "false",+ },+ configured=True,+ )+ },+ )+ cfg = SimpleNamespace(providers=providers)++ _, provider, credentials = resolve_call(+ "watsonx_onprem:test-model",+ kind="chat",+ config=cfg,+ )++ assert provider == "watsonx_onprem"+ assert "client" not in credentials+ json.dumps(credentials)++@pytest.mark.asyncioasync def test_chat_completions_calls_litellm_with_config_key(monkeypatch):captured = {}@@ -369,6 +400,60 @@ async def test_embeddings_calls_litellm(monkeypatch):assert captured["input"] == ["hello"]+async def test_onprem_embeddings_batch_large_document_inputs(monkeypatch):+ providers = ProvidersConfig(+ openai=OpenAIConfig(),+ anthropic=AnthropicConfig(),+ watsonx=WatsonXConfig(),+ ollama=OllamaConfig(),+ custom={+ "watsonx_onprem": GenericProviderConfig(+ credentials={+ "api_base": "https://cpd.example.com",+ "username": "cpduser",+ "api_key": "secret",+ },+ configured=True,+ )+ },+ )+ calls: list[list[str]] = []++ async def fake_aembedding(**kwargs):+ batch = kwargs["input"]+ calls.append(batch)+ return {+ "object": "list",+ "data": [+ {"object": "embedding", "embedding": [float(index)], "index": index}+ for index in range(len(batch))+ ],+ "usage": {+ "prompt_tokens": len(batch),+ "total_tokens": len(batch),+ },+ }++ monkeypatch.setattr("litellm.aembedding", fake_aembedding)+ monkeypatch.setattr(+ "services.llm_gateway._provider_runtime_kwargs",+ lambda provider, config: {},+ )++ result = await embeddings(+ {+ "model": "watsonx_onprem:ibm/slate-30m-english-rtrvr",+ "input": [f"chunk-{index}" for index in range(70)],+ },+ config=SimpleNamespace(providers=providers),+ )++ assert [len(batch) for batch in calls] == [32, 32, 6]+ assert [item["index"] for item in result["data"]] == list(range(70))+ assert result["usage"] == {"prompt_tokens": 70, "total_tokens": 70}++class _RecordingLogger:"""structlog-style logger that records calls instead of emitting them."""diff --git a/tests/unit/services/test_model_catalog.py b/tests/unit/services/test_model_catalog.pyindex 7d6258745..55ee9fb97 100644--- a/tests/unit/services/test_model_catalog.py+++ b/tests/unit/services/test_model_catalog.py@@ -9,6 +9,8 @@ the same payload.from __future__ import annotations+from types import SimpleNamespace+import pytestfrom config import model_providers@@ -491,3 +493,34 @@ def test_the_shipped_config_excludes_on_azure_as_well_as_openai(monkeypatch) ->ids = {row["id"] for row in model_catalog.openai_models_list()["data"]}assert not [model_id for model_id in ids if "gpt-3.5" in model_id or "luna" in model_id]+++async def test_live_model_failure_does_not_block_other_provider_refreshes(monkeypatch):+ calls: list[str] = []++ async def fail(_credentials):+ calls.append("broken")+ raise ValueError("missing CA bundle")++ async def succeed(_credentials):+ calls.append("healthy")++ broken = SimpleNamespace(PROVIDER_KEY="broken", fetch_models=fail)+ healthy = SimpleNamespace(PROVIDER_KEY="healthy", fetch_models=succeed)+ providers = SimpleNamespace(stored_credentials=lambda _provider: {"api_key": "secret"})++ monkeypatch.setattr(model_catalog, "provider_enhancements", lambda: (broken, healthy))+ monkeypatch.setattr(+ model_catalog,+ "supported_provider_keys",+ lambda: frozenset({"broken", "healthy"}),+ )+ monkeypatch.setattr(+ "config.settings.get_openrag_config",+ lambda: SimpleNamespace(providers=providers),+ )++ await model_catalog.refresh_live_models()++ assert calls == ["broken", "healthy"]diff --git a/tests/unit/test_backend_router.py b/tests/unit/test_backend_router.pyindex 0a3149b21..dc5845502 100644--- a/tests/unit/test_backend_router.py+++ b/tests/unit/test_backend_router.py@@ -192,6 +192,25 @@ def test_llm_proxy_forwards_only_allowlisted_headers_and_maps_private_path(monkeassert "x-evil" not in forwarded+def test_llm_proxy_uses_ingestion_scale_timeout(monkeypatch):+ captured: dict = {}++ def create_client(*args, **kwargs):+ captured["timeout"] = kwargs["timeout"]+ return _FakeLlmClient(captured)++ monkeypatch.setattr(router_app.httpx, "AsyncClient", create_client)++ response = TestClient(router_app.create_router_app()).post(+ "/embeddings",+ json={"model": "ibm/slate-30m-english-rtrvr", "input": ["chunk"]},+ )++ assert response.status_code == 200+ assert captured["timeout"].connect == settings.LANGFLOW_CONNECT_TIMEOUT+ assert captured["timeout"].read == settings.LANGFLOW_TIMEOUT++def test_router_exposes_only_allowlisted_paths_and_health():client = TestClient(router_app.create_router_app())assert client.get("/health").status_code == 200diff --git a/tests/unit/test_models_api_errors.py b/tests/unit/test_models_api_errors.pyindex 4bbb2a2b0..4d52ca864 100644--- a/tests/unit/test_models_api_errors.py+++ b/tests/unit/test_models_api_errors.py@@ -136,6 +136,177 @@ async def test_get_ibm_models_returns_project_configuration_error():assert payload["error"] == msg+async def test_watsonx_space_listing_merges_pending_fields_with_saved_secret(monkeypatch):+ from config.config_manager import (+ AnthropicConfig,+ GenericProviderConfig,+ OllamaConfig,+ OpenAIConfig,+ ProvidersConfig,+ WatsonXConfig,+ )+ from enhancements.providers.watsonx import onprem++ providers = ProvidersConfig(+ openai=OpenAIConfig(),+ anthropic=AnthropicConfig(),+ watsonx=WatsonXConfig(),+ ollama=OllamaConfig(),+ custom={+ onprem.PROVIDER_KEY: GenericProviderConfig(+ credentials={+ "api_base": "https://cpd.example.com",+ "username": "saved-user",+ "api_key": "saved-secret",+ "ssl_verify": "true",+ },+ auth_method="username_api_key",+ configured=True,+ )+ },+ )+ monkeypatch.setattr(+ models_api,+ "get_openrag_config",+ lambda: SimpleNamespace(providers=providers),+ )+ seen = {}++ async def _list_spaces(credentials):+ seen.update(credentials)+ return [{"id": "space-1", "name": "Production"}]++ monkeypatch.setattr(onprem, "list_spaces", _list_spaces)++ response = await models_api.get_watsonx_onprem_spaces(+ body=models_api.WatsonxOnPremSpacesBody(+ credentials={"username": "pending-user", "api_key": ""},+ auth_method="username_api_key",+ ),+ user=SimpleNamespace(),+ )++ assert response.status_code == 200+ assert json.loads(response.body) == {"spaces": [{"id": "space-1", "name": "Production"}]}+ assert seen == {+ "api_base": "https://cpd.example.com",+ "username": "pending-user",+ "api_key": "saved-secret",+ "ssl_verify": "true",+ }+++async def test_watsonx_space_listing_never_reuses_secrets_for_a_new_target(monkeypatch):+ from config.config_manager import (+ AnthropicConfig,+ GenericProviderConfig,+ OllamaConfig,+ OpenAIConfig,+ ProvidersConfig,+ WatsonXConfig,+ )+ from enhancements.providers.watsonx import onprem++ providers = ProvidersConfig(+ openai=OpenAIConfig(),+ anthropic=AnthropicConfig(),+ watsonx=WatsonXConfig(),+ ollama=OllamaConfig(),+ custom={+ onprem.PROVIDER_KEY: GenericProviderConfig(+ credentials={+ "api_base": "https://saved.example.com",+ "username": "saved-user",+ "api_key": "saved-secret",+ "ssl_verify": "true",+ },+ auth_method="username_api_key",+ configured=True,+ )+ },+ )+ monkeypatch.setattr(+ models_api,+ "get_openrag_config",+ lambda: SimpleNamespace(providers=providers),+ )+ seen = {}++ async def _list_spaces(credentials):+ seen.update(credentials)+ return []++ monkeypatch.setattr(onprem, "list_spaces", _list_spaces)++ response = await models_api.get_watsonx_onprem_spaces(+ body=models_api.WatsonxOnPremSpacesBody(+ credentials={"api_base": "https://new.example.com"},+ auth_method="username_api_key",+ ),+ user=SimpleNamespace(),+ )++ assert response.status_code == 200+ assert seen == {"api_base": "https://new.example.com"}+++async def test_watsonx_space_listing_hides_upstream_error_details(monkeypatch):+ from config.config_manager import (+ AnthropicConfig,+ GenericProviderConfig,+ OllamaConfig,+ OpenAIConfig,+ ProvidersConfig,+ WatsonXConfig,+ )+ from enhancements.providers.watsonx import onprem++ providers = ProvidersConfig(+ openai=OpenAIConfig(),+ anthropic=AnthropicConfig(),+ watsonx=WatsonXConfig(),+ ollama=OllamaConfig(),+ custom={+ onprem.PROVIDER_KEY: GenericProviderConfig(+ credentials={+ "api_base": "https://cpd.example.com",+ "username": "saved-user",+ "api_key": "saved-secret",+ },+ auth_method="username_api_key",+ configured=True,+ )+ },+ )+ monkeypatch.setattr(+ models_api,+ "get_openrag_config",+ lambda: SimpleNamespace(providers=providers),+ )++ async def _list_spaces(credentials):+ raise onprem.SpaceDiscoveryError(+ 401,+ "Traceback: Authorization Bearer exposed-secret",+ )++ monkeypatch.setattr(onprem, "list_spaces", _list_spaces)++ response = await models_api.get_watsonx_onprem_spaces(+ body=models_api.WatsonxOnPremSpacesBody(),+ user=SimpleNamespace(),+ )++ assert response.status_code == 401+ assert json.loads(response.body) == {+ "error": "The cluster rejected the configured credentials."+ }+ assert b"exposed-secret" not in response.body++@pytest.mark.asyncioasync def test_get_model_catalog_returns_only_supported_providers():"""The catalogue publishes exactly what this run mode's config exposes.