Skip to content

OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.

Python

test: live validation matrix for Azure AI Foundry, ahead of the flip (#2447)

Rico Furtado51bc93d50a01
diff --git a/scripts/README.md b/scripts/README.md
index ea289eefc..487cce993 100644
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -250,3 +250,89 @@ After running, **restart the backend** and reload the app.
This script does **not** delete ingested documents, knowledge filters, Langflow
flows, or conversations. For a full teardown, use the authenticated API endpoint
`POST /settings/rollback-onboarding` instead.
+
+## Azure AI Foundry live validation (`validate_azure_foundry_live.py`)
+
+The matrix that must pass against a **real** Azure AI Foundry resource before
+`azure_ai` is made visible in any run mode. The provider stays hidden while it
+runs: credentials come from the environment and the config object is built in
+memory, so nothing reads `model_providers.yaml` or the saved configuration.
+
+Every check goes through OpenRAG's own code — the provider enhancement, the
+LLM gateway, the Docling option builder — rather than reimplementing the
+requests. A script that built its own URLs would prove Azure works, not that
+OpenRAG calls it correctly, which is the question the flip depends on.
+
+```bash
+export AZURE_AI_API_BASE="https://<resource>.services.ai.azure.com/openai/v1"
+export AZURE_AI_API_KEY="<key>"
+export AZURE_AI_CHAT_DEPLOYMENT="<chat deployment>"
+export AZURE_AI_EMBEDDING_DEPLOYMENT="<embedding deployment>"
+
+# Optional; each unlocks more of the matrix.
+export AZURE_AI_PROJECT_API_BASE="https://<r>.services.ai.azure.com/api/projects/<p>/openai/v1"
+export AZURE_AI_LEGACY_API_BASE="https://<resource>.services.ai.azure.com/models"
+export AZURE_AI_VLM_DEPLOYMENT="<vision-capable deployment>"
+
+uv run python scripts/validate_azure_foundry_live.py
+```
+
+Exit status is non-zero if any **blocker** fails *or is skipped* — an
+unexercised blocker leaves the matrix incomplete, which is not a pass.
+
+| # | Check | |
+|---|---|---|
+| 1 | Save-time validation, valid credentials | blocker |
+| 2 | Invalid API key is rejected | blocker |
+| 3 | Chat | blocker |
+| 4 | Streaming | blocker |
+| 5 | Tool calling | blocker |
+| 6 | Embeddings | blocker |
+| 7 | Unknown deployment fails cleanly | blocker |
+| 8 | `/openai/v1` routing | blocker |
+| 9 | `/api/projects/<project>/openai/v1` routing | blocker when configured |
+| 10 | Docling VLM picture description | blocker |
+| 11 | No credential in errors or logs | blocker |
+| 12 | Legacy `/models` `api-key` vs `Bearer` | advisory |
+| 13 | Cost attribution | advisory |
+
+**The 403 question.** `foundry.lightweight_health_check` raises
+`PermissionError` for a 403 on the deployment listing, on the reasoning that
+listing can need a permission inference does not. The product does not act on
+that distinction: `api/settings/endpoints.py` catches `Exception` around
+provider validation and returns HTTP 400, so the provider cannot be saved
+either way. A 403 on a valid credential is therefore reported as
+**inconclusive and blocking**, not as a pass. Set
+`AZURE_AI_LISTING_DENIED_API_KEY` to capture what such a credential actually
+returns; the decision — require listing permission, or make a specific denial
+non-blocking — needs that response alongside the invalid-key response, and is
+only safe if the two are reliably distinguishable.
+
+Checks 3–7 run once per configured endpoint.
+
+Cost is advisory by design: OpenRAG does not consume cost data today, and the
+OpenAI-compatible transport's loss of LiteLLM's `azure_ai/*` price rows is
+documented in `enhancements/providers/azure/foundry.py`. Check 12 is advisory
+because the shipped health check sends both header styles; its answer only
+decides whether that can later be simplified.
+
+The API key is never printed. The run captures root logging, stdout, stderr
+and every recorded exception, and check 11 scans all of it. When a leak is
+found the captured material is **not** reproduced — only the fact and the
+number of matching fragments — and every printed line is redacted regardless,
+since several checks quote an upstream response body.
+
+Checks that pass by observing a rejection assert the *category* of failure:
+the invalid-key check passes only on a credential rejection, and the
+unknown-deployment check only on a not-found through the gateway's own error
+contract. Neither accepts a DNS, TLS, timeout or 5xx failure, which would
+otherwise let an unreachable endpoint look like a successful rejection.
+
+**Two things the script cannot cover**, both needed before the flip:
+
+- Check 10 issues the exact URL, headers and params OpenRAG hands
+ docling-serve, but does not run docling-serve. Do one real ingest of an
+ image-bearing document through the running stack with picture descriptions
+ enabled.
+- Nothing here exercises the settings UI. Save the provider once through
+ **Settings > Providers** to confirm the form and its validation behave.
diff --git a/scripts/validate_azure_foundry_live.py b/scripts/validate_azure_foundry_live.py
new file mode 100644
index 000000000..119c286e2
--- /dev/null
+++ b/scripts/validate_azure_foundry_live.py
@@ -0,0 +1,1378 @@
+#!/usr/bin/env python3
+"""Live validation matrix for the Azure AI Foundry provider.
+
+Runs the checks that must pass against a **real Foundry resource** before
+`azure_ai` is made visible in any run mode. Everything here is deliberately
+read through OpenRAG's own code — `enhancements.providers.azure.foundry`,
+`services.llm_gateway`, `services.docling_service` — rather than reimplemented
+with `curl`-equivalent requests. A script that built its own URLs would prove
+Azure works, not that OpenRAG calls it correctly, which is the open question.
+
+The provider stays hidden while this runs. Nothing here reads
+`model_providers.yaml` or the saved configuration: credentials come from the
+environment and a config object is assembled in memory, so the matrix can be
+run against production code with the feature still switched off.
+
+Usage
+-----
+ export AZURE_AI_API_BASE="https://<resource>.services.ai.azure.com/openai/v1"
+ export AZURE_AI_API_KEY="<key>"
+ export AZURE_AI_CHAT_DEPLOYMENT="<your chat deployment>"
+ export AZURE_AI_EMBEDDING_DEPLOYMENT="<your embedding deployment>"
+
+ # Optional, each unlocks more of the matrix:
+ export AZURE_AI_PROJECT_API_BASE="https://<r>.services.ai.azure.com/api/projects/<p>/openai/v1"
+ export AZURE_AI_LEGACY_API_BASE="https://<resource>.services.ai.azure.com/models"
+ export AZURE_AI_VLM_DEPLOYMENT="<a vision-capable deployment>"
+ # A credential that can run inference but NOT list deployments, if one can
+ # be created. See "The 403 question" below — this is the scenario that
+ # decides it.
+ export AZURE_AI_LISTING_DENIED_API_KEY="<key with inference but not listing>"
+
+ uv run python scripts/validate_azure_foundry_live.py
+
+Exit code is non-zero if any **blocker** fails. Blockers are the capabilities
+the product claims once the provider is visible: supported endpoint routing,
+chat, streaming, tool calling, embeddings, Docling VLM, and error
+sanitization. Cost attribution is reported but never blocks — OpenRAG does not
+consume cost data today (see the transport trade-off in `foundry.py`).
+
+The 403 question
+----------------
+`foundry.lightweight_health_check` raises `PermissionError` for a 403 on the
+deployment listing, on the reasoning that listing can need a permission
+inference does not. **But the product does not act on that distinction.**
+`api/settings/endpoints.py` catches `Exception` around provider validation and
+returns HTTP 400, so a `PermissionError` is indistinguishable from a bad key:
+the provider cannot be saved either way.
+
+So a 403 on a valid credential is *not* treated as a pass here. This script
+gathers the evidence needed to decide between two fixes — require listing
+permission and say so, or make a specifically-identified listing denial
+non-blocking in the settings endpoint — by recording exactly what Foundry
+returns for:
+
+ * a valid key with normal permissions;
+ * a deliberately invalid key;
+ * a valid credential that can infer but not list, when one is available
+ (`AZURE_AI_LISTING_DENIED_API_KEY`).
+
+Until that evidence exists, a generic 403 is reported as **inconclusive and
+blocking**. Do not relax it on the strength of a status code alone: the
+distinction is only safe if the responses are reliably distinguishable.
+
+Secrets
+-------
+The API key is never printed. The run captures root logging, stdout, stderr
+and every recorded exception, and check 11 scans all of it. When a secret is
+found the captured material is **not** printed — only the fact and where.
+
+What this cannot cover
+----------------------
+Check 10 exercises the exact URL, headers and params OpenRAG hands
+docling-serve, by issuing that request itself. It does not run docling-serve.
+Before the flip, also do one real ingest of an image-bearing document through
+the running stack with picture descriptions enabled, to confirm docling-serve
+accepts the block as given.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import base64
+import contextlib
+import json
+import logging
+import math
+import os
+import re
+import struct
+import sys
+import traceback
+import zlib
+from dataclasses import dataclass, field
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any
+from urllib.parse import urlsplit, urlunsplit
+
+ROOT = Path(__file__).resolve().parent.parent
+SRC = ROOT / "src"
+for path in (str(SRC), str(ROOT)):
+ if path not in sys.path:
+ sys.path.insert(0, path)
+
+
+# --------------------------------------------------------------------------
+# Result collection
+# --------------------------------------------------------------------------
+
+BLOCKER = "blocker"
+ADVISORY = "advisory"
+
+
+@dataclass
+class Check:
+ number: int
+ name: str
+ severity: str
+ status: str = "skipped"
+ detail: str = ""
+
+ @property
+ def ok(self) -> bool:
+ return self.status in ("pass", "skipped")
+
+
+@dataclass
+class Matrix:
+ checks: list[Check] = field(default_factory=list)
+ #: Everything the run produced that a secret could hide in.
+ captured_text: list[str] = field(default_factory=list)
+
+ def record(self, number: int, name: str, severity: str, status: str, detail: str = "") -> Check:
+ check = Check(number, name, severity, status, detail)
+ self.checks.append(check)
+ self.captured_text.append(detail)
+ return check
+
+ def capture(self, text: str) -> None:
+ self.captured_text.append(text)
+
+
+def _unusable_chat_reply(text: str) -> str:
+ """Why `text` is not a usable completion, or "" if it is."""
+ return "" if text.strip() else "the completion succeeded with empty content"
+
+
+def _unusable_embedding(vector: Any) -> str:
+ """Why `vector` cannot be used as an embedding, or "" if it can."""
+ if not isinstance(vector, list) or not vector:
+ return "the call succeeded but returned an empty embedding"
+ for value in vector:
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ return f"the embedding contains a non-numeric value ({type(value).__name__})"
+ if not math.isfinite(value):
+ return "the embedding contains a non-finite value (nan or inf)"
+ return ""
+
+
+def _describe_exception(exc: BaseException) -> str:
+ return f"{type(exc).__name__}: {exc}"
+
+
+#: Every real credential this run was given. Populated once from the
+#: environment and used to redact anything on its way to a terminal — the
+#: in-memory capture keeps the raw text so check 11 can still detect a leak.
+_REAL_SECRETS: tuple[str, ...] = ()
+
+
+def _set_real_secrets(*secrets: str) -> None:
+ global _REAL_SECRETS
+ # Short values are not redacted: a two-character "secret" would blank out
+ # unrelated text and make the output useless.
+ _REAL_SECRETS = tuple(dict.fromkeys(s for s in secrets if s and len(s) >= 8))
+
+
+def _safe_endpoint(api_base: str) -> str:
+ """An endpoint with anything secret-bearing taken off, safe to display.
+
+ `_redact` only knows the credentials this run was handed, so it cannot
+ help with a secret carried *inside* the endpoint: `user:password@` in the
+ netloc, or a SAS token in the query. The provider module refuses or strips
+ both, but that happens after the operator has already seen this line.
+ """
+ parts = urlsplit(api_base or "")
+ if not parts.scheme or not parts.netloc:
+ return api_base or ""
+ host = parts.hostname or ""
+ if parts.port:
+ host = f"{host}:{parts.port}"
+ cleaned = urlunsplit((parts.scheme, host, parts.path, "", ""))
+ dropped = []
+ if parts.username or parts.password:
+ dropped.append("userinfo")
+ if parts.query:
+ dropped.append("query")
+ return f"{cleaned} (dropped: {', '.join(dropped)})" if dropped else cleaned
+
+
+def _redact(text: str) -> str:
+ """Remove every known real credential from text about to be displayed.
+
+ Check 11 *detects* a leak after the fact, which is too late on its own:
+ by then a library has already written the secret to stdout or stderr and,
+ in CI, into a build log that outlives the run. So nothing reaches a real
+ stream without passing through here first.
+ """
+ cleaned = text
+ for secret in _REAL_SECRETS:
+ cleaned = cleaned.replace(secret, "<redacted>")
+ return cleaned
+
+
+#: A failure that says nothing about credentials or deployments — the request
+#: did not get a considered answer. Any check that "passes on an error" has to
+#: exclude these, or an unplugged network reads as a successful rejection.
+_CONNECTIVITY_MARKERS = (
+ "nodename nor servname",
+ "name or service not known",
+ "temporary failure in name resolution",
+ "cannot connect to host",
+ "connection refused",
+ "connection reset",
+ "connect call failed",
+ "connecterror",
+ "connecttimeout",
+ "readtimeout",
+ "timeout",
+ "timed out",
+ "certificate",
+ "ssl",
+ "tlsv1",
+ "proxyerror",
+)
+
+#: Upstream said the deployment or model is not there. Azure words this
+#: differently across its two surfaces, so both vocabularies are covered.
+_NOT_FOUND_MARKERS = (
+ "deploymentnotfound",
+ "deployment not found",
+ "does not exist",
+ "model_not_found",
+ "model not found",
+ "unknown model",
+ "no deployment",
+ "resourcenotfound",
+ "404 not found",
+)
+
+#: A bare "404" is not a marker: those three digits appear in request ids,
+#: timestamps and token counts, and matching them anywhere would let an
+#: unrelated response pass the unknown-deployment check. Recognised only where
+#: the surrounding text makes it a status.
+_HTTP_404_PATTERN = re.compile(
+ r"(?:http[/ ]?[\d.]*\s*|status(?:_?code)?\s*[:=]\s*|code\s*[:=]\s*)404\b",
+ re.IGNORECASE,
+)
+
+#: And a bare "401" is not one either, for the same reason: check 2 runs this
+#: classifier over 300 characters of response body, where a request id like
+#: `8a401b2c` would otherwise read as a rejected credential and let a
+#: non-credential failure pass. The real status is already covered by the
+#: `status == 401` test at the call site.
+_HTTP_401_PATTERN = re.compile(
+ r"(?:http[/ ]?[\d.]*\s*|status(?:_?code)?\s*[:=]\s*|code\s*[:=]\s*)401\b",
+ re.IGNORECASE,
+)
+
+
+def _looks_like_connectivity_failure(text: str) -> bool:
+ lowered = (text or "").lower()
+ return any(marker in lowered for marker in _CONNECTIVITY_MARKERS)
+
+
+def _looks_like_not_found(text: str) -> bool:
+ lowered = (text or "").lower()
+ if any(marker in lowered for marker in _NOT_FOUND_MARKERS):
+ return True
+ return bool(_HTTP_404_PATTERN.search(lowered))
+
+
+def _looks_like_credential_failure(text: str) -> bool:
+ """Upstream rejected the credential, per OpenRAG's own classifier."""
+ from api.provider_validation import is_provider_credential_error
+
+ lowered = (text or "").lower()
+ if is_provider_credential_error(lowered):
+ return True
+ # Azure's own vocabulary for a rejected key, which the shared marker list
+ # does not have to know about.
+ if _HTTP_401_PATTERN.search(lowered):
+ return True
+ return any(
+ marker in lowered
+ for marker in (
+ "unauthorized",
+ "access denied due to invalid subscription key",
+ "invalid subscription key",
+ "permissiondenied",
+ "invalid authentication",
+ "accessdenied",
+ )
+ )
+
+
+# --------------------------------------------------------------------------
+# In-memory config, so nothing depends on the provider being enabled or saved
+# --------------------------------------------------------------------------
+
+
+def _build_config(api_base: str, api_key: str, chat_model: str, embedding_model: str):
+ from config.config_manager import (
+ AnthropicConfig,
+ GenericProviderConfig,
+ OllamaConfig,
+ OpenAIConfig,
+ ProvidersConfig,
+ WatsonXConfig,
+ )
+
+ providers = ProvidersConfig(
+ openai=OpenAIConfig(),
+ anthropic=AnthropicConfig(),
+ watsonx=WatsonXConfig(),
+ ollama=OllamaConfig(),
+ custom={
+ "azure_ai": GenericProviderConfig(
+ credentials={"api_base": api_base, "api_key": api_key},
+ configured=True,
+ )
+ },
+ )
+ return SimpleNamespace(
+ providers=providers,
+ agent=SimpleNamespace(llm_model=chat_model, llm_provider="azure_ai"),
+ knowledge=SimpleNamespace(embedding_model=embedding_model, embedding_provider="azure_ai"),
+ )
+
+
+# --------------------------------------------------------------------------
+# Per-endpoint matrix
+# --------------------------------------------------------------------------
+
+
+async def _run_endpoint_matrix(
+ matrix: Matrix,
+ label: str,
+ api_base: str,
+ settings: dict[str, str],
+ *,
+ base_number: int,
+ expected_profile: str | None = None,
+) -> None:
+ """Save check, chat, streaming, tools, embeddings and a bad deployment."""
+ from enhancements.providers.azure import foundry
+ from services.llm_gateway import chat_completions, embeddings
+
+ api_key = settings["api_key"]
+ chat_model = settings["chat_deployment"]
+ embedding_model = settings["embedding_deployment"]
+ config = _build_config(api_base, api_key, chat_model, embedding_model)
+
+ # Classification is a blocker in its own right, so it has to be able to
+ # *fail*. Recording an unconditional pass meant a legacy `/models`
+ # endpoint pasted into AZURE_AI_API_BASE still satisfied the `/openai/v1`
+ # check, which is the one thing this matrix exists to prove.
+ profile = foundry.endpoint_profile(api_base)
+ try:
+ route = foundry.litellm_route({"api_base": api_base})
+ except Exception as exc:
+ # `litellm_route` refuses an endpoint it cannot classify, by design.
+ # Uncaught it would abort the whole run before `_print_report`, so the
+ # operator would get a traceback instead of the matrix — the least
+ # useful outcome for a release gate.
+ matrix.record(
+ base_number,
+ f"[{label}] endpoint classified and routed",
+ BLOCKER,
+ "fail",
+ f"profile={profile} route=<unresolved> {_describe_exception(exc)}",
+ )
+ return
+
+ mismatch = expected_profile is not None and profile != expected_profile
+ matrix.record(
+ base_number,
+ f"[{label}] endpoint classified and routed",
+ BLOCKER,
+ "fail" if mismatch else "pass",
+ f"profile={profile} route={route}"
+ + (f" (expected {expected_profile})" if mismatch else ""),
+ )
+ if mismatch:
+ # Everything after this would be measuring a different surface than
+ # the one this block of the matrix claims to cover.
+ return
+
+ # 1. Save-time validation with valid credentials. Delegated to the
+ # credential-semantics probe so the raw response is on record, not just
+ # whether an exception was raised.
+ await _probe_credential(
+ matrix,
+ number=1,
+ name=f"[{label}] save-time check with valid credentials",
+ api_base=api_base,
+ api_key=api_key,
+ scenario="valid",
+ )
+
+ # 3. Chat.
+ try:
+ response = await chat_completions(
+ {
+ "model": f"azure_ai:{chat_model}",
+ "messages": [{"role": "user", "content": "Reply with the word OK."}],
+ "max_tokens": 16,
+ },
+ config=config,
+ )
+ text = (response["choices"][0]["message"].get("content") or "").strip()
+ # A completion that succeeds and says nothing is not a working
+ # deployment. Recorded as a pass it made the release matrix green
+ # without usable inference, which is the one thing a blocker here
+ # exists to rule out.
+ problem = _unusable_chat_reply(text)
+ matrix.record(
+ 3,
+ f"[{label}] chat",
+ BLOCKER,
+ "fail" if problem else "pass",
+ problem or f"reply={text[:40]!r}",
+ )
+ except Exception as exc:
+ matrix.record(3, f"[{label}] chat", BLOCKER, "fail", _describe_exception(exc))
+
+ # 4. Streaming. The contract is `data: {...}` frames terminated by
+ # `data: [DONE]`, so what is asserted is a well-formed stream that
+ # completes — not a frame count. A short completion may legitimately
+ # arrive in a single content chunk, and counting frames would also have
+ # passed a *failed* stream, because the gateway reports failure as an
+ # error frame followed by [DONE].
+ try:
+ stream = await chat_completions(
+ {
+ "model": f"azure_ai:{chat_model}",
+ "messages": [{"role": "user", "content": "Count from one to five."}],
+ "max_tokens": 64,
+ "stream": True,
+ },
+ config=config,
+ )
+ frames = 0
+ content_chunks = 0
+ saw_done = False
+ error_frame = ""
+ async for frame in stream:
+ frames += 1
+ payload = (
+ frame[len("data: ") :].strip() if frame.startswith("data: ") else frame.strip()
+ )
+ if payload == "[DONE]":
+ saw_done = True
+ continue
+ try:
+ parsed = json.loads(payload)
+ except (ValueError, TypeError):
+ continue
+ if isinstance(parsed, dict) and parsed.get("error"):
+ error_frame = str(parsed["error"])[:160]
+ continue
+ for choice in (parsed or {}).get("choices") or []:
+ delta = choice.get("delta") or {}
+ if delta.get("content") or delta.get("tool_calls"):
+ content_chunks += 1
+
+ if error_frame:
+ matrix.record(4, f"[{label}] streaming", BLOCKER, "fail", f"error frame: {error_frame}")
+ elif not content_chunks:
+ matrix.record(
+ 4,
+ f"[{label}] streaming",
+ BLOCKER,
+ "fail",
+ f"{frames} frames but none carried content or tool calls",
+ )
+ elif not saw_done:
+ matrix.record(
+ 4,
+ f"[{label}] streaming",
+ BLOCKER,
+ "fail",
+ f"{content_chunks} content chunk(s) but the stream never terminated with [DONE]",
+ )
+ else:
+ matrix.record(
+ 4,
+ f"[{label}] streaming",
+ BLOCKER,
+ "pass",
+ f"{content_chunks} content chunk(s) of {frames} frames, terminated cleanly",
+ )
+ except Exception as exc:
+ matrix.record(4, f"[{label}] streaming", BLOCKER, "fail", _describe_exception(exc))
+
+ # 5. Tool calling. Forced with `tool_choice`, because the question here
+ # is whether tool calls survive the transport — not whether this
+ # particular model independently decides a tool is warranted. Leaving
+ # that to the model makes the gate flaky and provider-dependent.
+ tools = [
+ {
+ "type": "function",
+ "function": {
+ "name": "get_weather",
+ "description": "Look up the current weather for a city.",
+ "parameters": {
+ "type": "object",
+ "properties": {"city": {"type": "string", "description": "City name"}},
+ "required": ["city"],
+ },
+ },
+ }
+ ]
+ try:
+ response = await chat_completions(
+ {
+ "model": f"azure_ai:{chat_model}",
+ "messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],
+ "tools": tools,
+ "tool_choice": {"type": "function", "function": {"name": "get_weather"}},
+ "max_tokens": 128,
+ },
+ config=config,
+ )
+ calls = response["choices"][0]["message"].get("tool_calls") or []
+ named = [c for c in calls if (c.get("function") or {}).get("name") == "get_weather"]
+ if not named:
+ matrix.record(
+ 5,
+ f"[{label}] tool calling",
+ BLOCKER,
+ "fail",
+ "tool_choice forced get_weather but it was not returned; "
+ f"tool_calls={[(c.get('function') or {}).get('name') for c in calls]}",
+ )
+ else:
+ raw_arguments = (named[0].get("function") or {}).get("arguments")
+ try:
+ arguments = json.loads(raw_arguments or "")
+ except (ValueError, TypeError):
+ matrix.record(
+ 5,
+ f"[{label}] tool calling",
+ BLOCKER,
+ "fail",
+ "get_weather was called but its arguments are not valid JSON; "
+ f"type={type(raw_arguments).__name__}",
+ )
+ else:
+ city = (arguments or {}).get("city") if isinstance(arguments, dict) else None
+ if isinstance(city, str) and city.strip():
+ matrix.record(
+ 5,
+ f"[{label}] tool calling",
+ BLOCKER,
+ "pass",
+ f"get_weather(city={city.strip()[:40]!r}) with valid JSON arguments",
+ )
+ else:
+ matrix.record(
+ 5,
+ f"[{label}] tool calling",
+ BLOCKER,
+ "fail",
+ f"get_weather returned JSON without a usable city: {arguments!r}",
+ )
+ except Exception as exc:
+ matrix.record(5, f"[{label}] tool calling", BLOCKER, "fail", _describe_exception(exc))
+
+ # 6. Embeddings.
+ try:
+ response = await embeddings(
+ {"model": f"azure_ai:{embedding_model}", "input": ["live validation probe"]},
+ config=config,
+ )
+ vector = response["data"][0]["embedding"]
+ # Same reasoning as the chat check: an empty vector recorded
+ # "0 dimensions" and passed. The finiteness test matters too — a
+ # vector of NaN has the right length and is useless for retrieval,
+ # and OpenSearch rejects it at index time rather than here.
+ problem = _unusable_embedding(vector)
+ matrix.record(
+ 6,
+ f"[{label}] embeddings",
+ BLOCKER,
+ "fail" if problem else "pass",
+ problem or f"{len(vector)} dimensions",
+ )
+ except Exception as exc:
+ matrix.record(6, f"[{label}] embeddings", BLOCKER, "fail", _describe_exception(exc))
+
+ # 7. A deployment that does not exist must fail as a *not found*, through
+ # the gateway's own error contract. Passing on any exception would let a
+ # DNS failure or a 500 stand in for the rejection this proves.
+ from services.llm_gateway import LlmGatewayError
+
+ try:
+ await chat_completions(
+ {
+ "model": "azure_ai:openrag-live-validation-no-such-deployment",
+ "messages": [{"role": "user", "content": "hello"}],
+ "max_tokens": 8,
+ },
+ config=config,
+ )
+ matrix.record(
+ 7,
+ f"[{label}] unknown deployment is rejected",
+ BLOCKER,
+ "fail",
+ "the call succeeded against a deployment that should not exist",
+ )
+ except LlmGatewayError as exc:
+ # `detail` is the internal text; `message` is what a caller is shown.
+ # Upstream 404s are deliberately mapped to 502 by the gateway, so the
+ # status is not the discriminator — the cause is.
+ detail = f"{exc.detail}"
+ if _looks_like_connectivity_failure(detail):
+ matrix.record(
+ 7,
+ f"[{label}] unknown deployment is rejected",
+ BLOCKER,
+ "fail",
+ f"could not reach the endpoint, so nothing was proved: {detail[:160]}",
+ )
+ elif _looks_like_not_found(detail):
+ matrix.record(
+ 7,
+ f"[{label}] unknown deployment is rejected",
+ BLOCKER,
+ "pass",
+ f"HTTP {exc.status_code} via the gateway; sanitized message={exc.message[:90]!r}",
+ )
+ else:
+ matrix.record(
+ 7,
+ f"[{label}] unknown deployment is rejected",
+ BLOCKER,
+ "fail",
+ "rejected, but not as a missing deployment — the error does not "
+ f"identify the cause: {detail[:160]}",
+ )
+ except Exception as exc:
+ matrix.record(
+ 7,
+ f"[{label}] unknown deployment is rejected",
+ BLOCKER,
+ "fail",
+ "the failure did not come through the gateway's error contract: "
+ + _describe_exception(exc)[:160],
+ )
+
+
+# --------------------------------------------------------------------------
+# Standalone checks
+# --------------------------------------------------------------------------
+
+
+async def _listing_response(api_base: str, api_key: str) -> tuple[int | None, str]:
+ """Raw status and body of the deployment listing, for the record.
+
+ The health check collapses the response into an exception type; deciding
+ the 403 question needs the response itself.
+ """
+ import httpx
+
+ from enhancements.providers.azure import foundry
+
+ try:
+ async with httpx.AsyncClient(timeout=30.0) as client:
+ response = await client.get(
+ foundry.models_url(api_base), headers=foundry.health_headers({"api_key": api_key})
+ )
+ return response.status_code, (response.text or "")[:300]
+ except Exception as exc:
+ return None, _describe_exception(exc)
+
+
+async def _probe_credential(
+ matrix: Matrix,
+ *,
+ number: int,
+ name: str,
+ api_base: str,
+ api_key: str,
+ scenario: str,
+) -> None:
+ """One credential scenario, recorded as evidence rather than a verdict.
+
+ `scenario` is "valid", "invalid" or "listing_denied". Each has a different
+ expected outcome, and a 403 is never silently treated as success — see
+ "The 403 question" in the module docstring.
+ """
+ from enhancements.providers.azure import foundry
+
+ status, body = await _listing_response(api_base, api_key)
+
+ outcome = "accepted"
+ detail = ""
+ try:
+ await foundry.lightweight_health_check({"api_base": api_base, "api_key": api_key})
+ except PermissionError as exc:
+ outcome, detail = "PermissionError", str(exc)
+ except Exception as exc:
+ outcome, detail = type(exc).__name__, str(exc)
+
+ # What the product would do: settings/endpoints.py wraps provider
+ # validation in `except Exception` and returns 400, so anything raised
+ # here — PermissionError included — blocks the save.
+ would_save = outcome == "accepted"
+ evidence = (
+ f"listing HTTP {status if status is not None else 'unreachable'}; "
+ f"health check -> {outcome}; provider saveable={would_save}"
+ )
+
+ if status is None and _looks_like_connectivity_failure(body):
+ matrix.record(number, name, BLOCKER, "fail", f"endpoint unreachable: {body[:140]}")
+ return
+
+ if scenario == "valid":
+ if would_save:
+ matrix.record(number, name, BLOCKER, "pass", evidence)
+ elif status == 403:
+ matrix.record(
+ number,
+ name,
+ BLOCKER,
+ "fail",
+ "INCONCLUSIVE, and blocking by design: the credential was accepted "
+ "for listing purposes but denied. `lightweight_health_check` raises "
+ "PermissionError, and settings/endpoints.py turns that into HTTP 400, "
+ "so the provider cannot be saved. Decide whether listing permission "
+ "is required or whether this specific denial should be non-blocking, "
+ f"using the response below. {evidence}. body={body[:160]!r}",
+ )
+ else:
+ matrix.record(number, name, BLOCKER, "fail", f"{evidence}. body={body[:160]!r}")
+ return
+
+ if scenario == "invalid":
+ if would_save:
+ matrix.record(
+ number, name, BLOCKER, "fail", f"an invalid key passed the check. {evidence}"
+ )
+ return
+ combined = f"{detail} {body}"
+ if _looks_like_connectivity_failure(combined):
+ matrix.record(
+ number,
+ name,
+ BLOCKER,
+ "fail",
+ f"rejected for a transport reason, which proves nothing: {evidence}",
+ )
+ elif status is not None and 500 <= status < 600:
+ matrix.record(
+ number,
+ name,
+ BLOCKER,
+ "fail",
+ f"upstream error rather than a credential rejection: {evidence}",
+ )
+ elif _looks_like_credential_failure(combined) or status == 401:
+ matrix.record(number, name, BLOCKER, "pass", evidence)
+ else:
+ matrix.record(
+ number,
+ name,
+ BLOCKER,
+ "fail",
+ "rejected, but not identifiably as a bad credential — so a bad key "
+ f"is indistinguishable from other failures. {evidence}. "
+ f"body={body[:160]!r}",
+ )
+ return
+
+ # listing_denied: pure evidence gathering, never a pass/fail verdict.
+ matrix.record(
+ number,
+ name,
+ ADVISORY,
+ "pass",
+ "EVIDENCE for the 403 decision — compare with the invalid-key response "
+ f"above; they must be reliably distinguishable. {evidence}. "
+ f"body={body[:200]!r}",
+ )
+
+
+def _sample_png(size: int = 128) -> bytes:
+ """A small PNG with enough structure for a vision model to describe.
+
+ Written here rather than committed as a fixture or a base64 blob so it
+ stays readable, and generated at a realistic size: a 1x1 pixel is a valid
+ PNG but some vision deployments reject images below a minimum dimension,
+ which would show up as an integration failure that is really a fixture
+ problem.
+
+ Four coloured quadrants, a dark border, and a white disc in the middle —
+ unambiguous enough that a description can be sanity-checked by eye.
+ """
+ quadrants = ((220, 60, 60), (60, 120, 220), (240, 200, 60), (60, 180, 110))
+ centre = size / 2
+ radius = size / 5
+ rows = []
+ for y in range(size):
+ row = bytearray()
+ for x in range(size):
+ if x < 2 or y < 2 or x >= size - 2 or y >= size - 2:
+ pixel = (30, 30, 30)
+ elif (x - centre) ** 2 + (y - centre) ** 2 <= radius**2:
+ pixel = (250, 250, 250)
+ else:
+ pixel = quadrants[(1 if x >= centre else 0) + (2 if y >= centre else 0)]
+ row.extend(pixel)
+ rows.append(bytes(row))
+
+ def _chunk(tag: bytes, data: bytes) -> bytes:
+ body = tag + data
+ return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body))
+
+ raw = b"".join(b"\x00" + row for row in rows)
+ return (
+ b"\x89PNG\r\n\x1a\n"
+ + _chunk(b"IHDR", struct.pack(">IIBBBBB", size, size, 8, 2, 0, 0, 0))
+ + _chunk(b"IDAT", zlib.compress(raw, 9))
+ + _chunk(b"IEND", b"")
+ )
+
+
+async def _check_docling_vlm(matrix: Matrix, api_base: str, settings: dict[str, str]) -> None:
+ """Build the block OpenRAG hands docling-serve, then issue that request.
+
+ docling-serve is not involved: the point is that the URL, headers and
+ params OpenRAG produces are accepted by Foundry as given.
+ """
+ import httpx
+
+ from services.docling_service import DoclingService
+
+ deployment = settings.get("vlm_deployment")
+ if not deployment:
+ matrix.record(
+ 10,
+ "Docling VLM picture description",
+ BLOCKER,
+ "skipped",
+ "set AZURE_AI_VLM_DEPLOYMENT to run this — it is a blocker for the flip",
+ )
+ return
+
+ knowledge = SimpleNamespace(
+ ocr=False,
+ ocr_languages=[],
+ table_structure=False,
+ picture_descriptions=True,
+ vlm_enabled=True,
+ vlm_provider="azure_ai",
+ vlm_model=f"azure_ai:{deployment}",
+ vlm_prompt="Describe this image in one short sentence.",
+ vlm_max_tokens=64,
+ vlm_watsonx_api_version="2024-01-01",
+ )
+ config = SimpleNamespace(
+ knowledge=knowledge,
+ providers=SimpleNamespace(
+ credential_values=lambda _key, **_kw: {
+ "api_base": api_base,
+ "api_key": settings["api_key"],
+ }
+ ),
+ )
+
+ import services.docling_service as docling_module
+
+ original = docling_module.get_openrag_config
+ docling_module.get_openrag_config = lambda: config
+ try:
+ options = await DoclingService()._build_docling_options_async()
+ except Exception as exc:
+ matrix.record(
+ 10,
+ "Docling VLM picture description",
+ BLOCKER,
+ "fail",
+ f"could not build the request: {_describe_exception(exc)}",
+ )
+ return
+ finally:
+ docling_module.get_openrag_config = original
+
+ api = options.get("picture_description_api")
+ if not api:
+ matrix.record(
+ 10, "Docling VLM picture description", BLOCKER, "fail", "no request was built"
+ )
+ return
+
+ # The model name must have shed its provider tag before it goes upstream.
+ if api["params"].get("model") != deployment:
+ matrix.record(
+ 10,
+ "Docling VLM picture description",
+ BLOCKER,
+ "fail",
+ f"model sent as {api['params'].get('model')!r}, expected {deployment!r}",
+ )
+ return
+
+ image = base64.b64encode(_sample_png()).decode()
+ payload = {
+ **api["params"],
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {"type": "text", "text": api["prompt"]},
+ {
+ "type": "image_url",
+ "image_url": {"url": f"data:image/png;base64,{image}"},
+ },
+ ],
+ }
+ ],
+ }
+ try:
+ async with httpx.AsyncClient(timeout=60.0) as client:
+ response = await client.post(api["url"], headers=api["headers"], json=payload)
+ if response.status_code == 200:
+ matrix.record(
+ 10,
+ "Docling VLM picture description",
+ BLOCKER,
+ "pass",
+ "Foundry accepted the request OpenRAG builds for docling-serve",
+ )
+ else:
+ matrix.record(
+ 10,
+ "Docling VLM picture description",
+ BLOCKER,
+ "fail",
+ f"HTTP {response.status_code}: {response.text[:200]}",
+ )
+ except Exception as exc:
+ matrix.record(
+ 10, "Docling VLM picture description", BLOCKER, "fail", _describe_exception(exc)
+ )
+
+
+async def _check_legacy_header_behaviour(
+ matrix: Matrix, legacy_base: str | None, api_key: str
+) -> None:
+ """The question open since the routing design: does `/models` take Bearer?
+
+ `foundry.health_headers` sends both styles because litellm itself is
+ inconsistent — its `azure_ai` chat path sends `api-key` while its embedding
+ path sends `Bearer` to the same endpoint. This measures what the endpoint
+ actually accepts, one header at a time.
+
+ Advisory: the shipped code sends both, so it works either way. The answer
+ decides whether that belt-and-braces can be simplified later.
+ """
+ import httpx
+
+ if not legacy_base:
+ matrix.record(
+ 12,
+ "legacy /models header behaviour",
+ ADVISORY,
+ "skipped",
+ "set AZURE_AI_LEGACY_API_BASE to measure this",
+ )
+ return
+
+ from enhancements.providers.azure import foundry
+
+ url = foundry.models_url(legacy_base)
+ results = []
+ for style, headers in (
+ ("api-key", {"api-key": api_key}),
+ ("Bearer", {"Authorization": f"Bearer {api_key}"}),
+ ):
+ try:
+ async with httpx.AsyncClient(timeout=30.0) as client:
+ response = await client.get(url, headers=headers)
+ results.append(f"{style}={response.status_code}")
+ except Exception as exc:
+ results.append(f"{style}=error({type(exc).__name__})")
+ matrix.record(12, "legacy /models header behaviour", ADVISORY, "pass", ", ".join(results))
+
+
+def _check_secret_leak(matrix: Matrix, secrets: tuple[str, ...]) -> None:
+ """Nothing this run produced may contain the key.
+
+ Scans captured root logging, stdout, stderr, OpenRAG's structlog output
+ and every recorded exception. On a hit the material is deliberately **not**
+ printed — only the fact, how many fragments matched, and the first source
+ kind — because printing it is the leak all over again, into a terminal and
+ very likely into CI output.
+ """
+ if not secrets:
+ matrix.record(
+ 11, "no credential in errors or logs", BLOCKER, "fail", "no credential to scan for"
+ )
+ return
+
+ scanned = sum(len(fragment) for fragment in matrix.captured_text if fragment)
+ leaked = 0
+ for index, secret in enumerate(secrets):
+ hits = sum(1 for fragment in matrix.captured_text if fragment and secret in fragment)
+ if hits:
+ leaked += 1
+ matrix.record(
+ 11,
+ "no credential in errors or logs",
+ BLOCKER,
+ "fail",
+ # Neither the secret nor the surrounding material is named:
+ # printing either is the leak again. The index identifies
+ # which credential without disclosing it.
+ f"credential #{index + 1} of {len(secrets)} appeared in {hits} captured "
+ "fragment(s). Nothing is reproduced here — re-run with scratch "
+ "credentials and inspect locally to find the source.",
+ )
+ if leaked:
+ return
+ matrix.record(
+ 11,
+ "no credential in errors or logs",
+ BLOCKER,
+ "pass",
+ f"scanned {scanned} characters of captured logging, stdout, stderr and errors "
+ f"for {len(secrets)} credential(s)",
+ )
+
+
+def _check_cost_attribution(matrix: Matrix, api_base: str, chat_model: str) -> None:
+ """Advisory by agreement: OpenRAG does not consume cost data today."""
+ from enhancements.providers.azure import foundry
+
+ route = foundry.litellm_route({"api_base": api_base})
+ if route == foundry.LITELLM_PROVIDER:
+ matrix.record(
+ 13,
+ "cost attribution",
+ ADVISORY,
+ "pass",
+ "native route keeps litellm's azure_ai price rows",
+ )
+ return
+ try:
+ import litellm
+
+ mapped = f"azure_ai/{chat_model}" in litellm.model_cost
+ except Exception:
+ mapped = False
+ matrix.record(
+ 13,
+ "cost attribution",
+ ADVISORY,
+ "pass",
+ (
+ f"transport route {route!r} has no price row; "
+ + (
+ f"base_model='azure_ai/{chat_model}' would restore it"
+ if mapped
+ else "the deployment name is not in any price table either"
+ )
+ + " — not consumed by OpenRAG today"
+ ),
+ )
+
+
+# --------------------------------------------------------------------------
+# Log capture
+# --------------------------------------------------------------------------
+
+
+class _Recorder:
+ """Collects every structlog call so check 11 can scan it."""
+
+ def __init__(self, sink: list[str]) -> None:
+ self._sink = sink
+
+ def __getattr__(self, _name):
+ def _log(message="", **fields):
+ self._sink.append(f"{message} {fields}")
+
+ return _log
+
+
+class _Tee:
+ """A stream that passes writes through and keeps a copy.
+
+ Progress stays visible while everything printed — including anything a
+ library writes directly rather than through logging — is available to the
+ secret scan. What reaches the real stream is redacted; what reaches the
+ sink is not, because detection needs the original.
+ """
+
+ def __init__(self, stream, sink: list[str]) -> None:
+ self._stream = stream
+ self._sink = sink
+
+ def write(self, text: str) -> int:
+ # Raw into the capture so check 11 can still see a leak...
+ self._sink.append(text)
+ # ...redacted on the way to the terminal. Forwarding the raw text and
+ # relying on the final report to redact would be too late: the secret
+ # is already on screen, and in CI already in a retained build log.
+ return self._stream.write(_redact(text))
+
+ def flush(self) -> None:
+ self._stream.flush()
+
+ def isatty(self) -> bool:
+ return False
+
+ def __getattr__(self, name):
+ return getattr(self._stream, name)
+
+
+class _SinkHandler(logging.Handler):
+ """Root logging handler: catches litellm, httpx and anything else."""
+
+ def __init__(self, sink: list[str]) -> None:
+ super().__init__(level=logging.DEBUG)
+ self._sink = sink
+
+ def emit(self, record: logging.LogRecord) -> None:
+ try:
+ self._sink.append(self.format(record))
+ except Exception:
+ # A handler that raises would be worse than one that misses a
+ # line, but a miss weakens the scan — so record that it happened.
+ self._sink.append(f"<unformattable log record from {record.name}>")
+
+
+def _console_handlers() -> list[logging.StreamHandler]:
+ """Every StreamHandler currently attached anywhere in the logging tree.
+
+ Walked rather than taken from the root logger alone: the handlers that
+ matter here are attached to named library loggers, not to root.
+ """
+ seen: list[logging.StreamHandler] = []
+ loggers: list[Any] = [logging.getLogger()]
+ loggers.extend(
+ obj for obj in logging.root.manager.loggerDict.values() if isinstance(obj, logging.Logger)
+ )
+ for logger_obj in loggers:
+ for candidate in getattr(logger_obj, "handlers", []):
+ if isinstance(candidate, logging.StreamHandler) and candidate not in seen:
+ seen.append(candidate)
+ return seen
+
+
+def _capturing(sink: list[str]):
+ """Capture root logging, stdout, stderr and OpenRAG's structlog loggers.
+
+ Everything here feeds check 11. Anything outside it — a library writing to
+ a file descriptor directly, or a subprocess — is not covered, which is why
+ check 11 reports how much it scanned rather than claiming completeness.
+ """
+ from enhancements.providers.azure import foundry
+ from services import docling_service, llm_gateway
+
+ # Annotated `Any` so the swap below type-checks. These are module objects,
+ # which mypy otherwise infers as `Module | Any` and then rejects assigning
+ # to; `setattr` silences that but trips ruff's B010, so the type is widened
+ # once here instead of suppressing a linter at each line.
+ modules: tuple[Any, ...] = (foundry, llm_gateway, docling_service)
+ original_loggers = [module.logger for module in modules]
+ for module in modules:
+ module.logger = _Recorder(sink)
+
+ root = logging.getLogger()
+ handler = _SinkHandler(sink)
+ handler.setFormatter(logging.Formatter("%(name)s %(levelname)s %(message)s"))
+ previous_level = root.level
+ root.addHandler(handler)
+ root.setLevel(logging.DEBUG)
+
+ real_stdout, real_stderr = sys.stdout, sys.stderr
+ tee_stdout = _Tee(real_stdout, sink)
+ tee_stderr = _Tee(real_stderr, sink)
+ sys.stdout = tee_stdout
+ sys.stderr = tee_stderr
+
+ # Swapping `sys.stderr` is not enough on its own. A StreamHandler attached
+ # before this block holds the stream *object* it was given, so it keeps
+ # writing to the real terminal no matter what `sys.stderr` now names —
+ # litellm attaches three such handlers at import, and litellm is the
+ # library most likely to log a request detail carrying a credential. Point
+ # them at the redacting stream for the duration and put them back after.
+ redirected: list[tuple[logging.StreamHandler, Any]] = []
+ for existing in _console_handlers():
+ if existing.stream is real_stdout:
+ redirected.append((existing, existing.stream))
+ existing.setStream(tee_stdout)
+ elif existing.stream is real_stderr:
+ redirected.append((existing, existing.stream))
+ existing.setStream(tee_stderr)
+
+ try:
+ yield
+ finally:
+ for existing, original_stream in redirected:
+ existing.setStream(original_stream)
+ sys.stdout, sys.stderr = real_stdout, real_stderr
+ root.removeHandler(handler)
+ root.setLevel(previous_level)
+ for module, logger in zip(modules, original_loggers, strict=True):
+ module.logger = logger
+
+
+# --------------------------------------------------------------------------
+# Entry point
+# --------------------------------------------------------------------------
+
+
+def _settings() -> dict[str, str]:
+ required = {
+ "api_base": "AZURE_AI_API_BASE",
+ "api_key": "AZURE_AI_API_KEY",
+ "chat_deployment": "AZURE_AI_CHAT_DEPLOYMENT",
+ "embedding_deployment": "AZURE_AI_EMBEDDING_DEPLOYMENT",
+ }
+ optional = {
+ "project_api_base": "AZURE_AI_PROJECT_API_BASE",
+ "legacy_api_base": "AZURE_AI_LEGACY_API_BASE",
+ "vlm_deployment": "AZURE_AI_VLM_DEPLOYMENT",
+ "listing_denied_api_key": "AZURE_AI_LISTING_DENIED_API_KEY",
+ }
+ values = {
+ name: os.environ.get(env, "").strip() for name, env in {**required, **optional}.items()
+ }
+ missing = [env for name, env in required.items() if not values[name]]
+ if missing:
+ print("Missing required environment variables:", ", ".join(missing), file=sys.stderr)
+ print(__doc__ or "", file=sys.stderr)
+ raise SystemExit(2)
+ return values
+
+
+async def _main() -> int:
+ settings = _settings()
+ # Before anything can print: every real credential the run was given,
+ # so `_redact` can keep all of them off the terminal.
+ _set_real_secrets(settings["api_key"], settings.get("listing_denied_api_key", ""))
+ matrix = Matrix()
+
+ # The profile each endpoint variable is *supposed* to name. Checked rather
+ # than assumed: the three are set by hand, and pasting the wrong surface
+ # into one of them would otherwise be reported as a pass for the surface
+ # it was not.
+ endpoints: list[tuple[str, str, int, str]] = [
+ ("resource v1", settings["api_base"], 8, "resource_openai_v1")
+ ]
+ if settings.get("project_api_base"):
+ endpoints.append(("project v1", settings["project_api_base"], 9, "project_openai_v1"))
+ if settings.get("legacy_api_base"):
+ endpoints.append(("legacy /models", settings["legacy_api_base"], 14, "legacy_models"))
+
+ with _capturing(matrix.captured_text):
+ await _run_all_checks(matrix, settings, endpoints)
+
+ _print_report(matrix)
+ return _exit_code(matrix)
+
+
+async def _run_all_checks(matrix: Matrix, settings: dict[str, str], endpoints) -> None:
+ for label, api_base, number, expected_profile in endpoints:
+ print(f"\n--- {label}: {_safe_endpoint(api_base)} ---", flush=True)
+ await _run_endpoint_matrix(
+ matrix,
+ label,
+ api_base,
+ settings,
+ base_number=number,
+ expected_profile=expected_profile,
+ )
+
+ if not settings.get("project_api_base"):
+ matrix.record(
+ 9,
+ "project /openai/v1",
+ ADVISORY,
+ "skipped",
+ "set AZURE_AI_PROJECT_API_BASE if this resource has a project endpoint",
+ )
+
+ await _probe_credential(
+ matrix,
+ number=2,
+ name="invalid API key is rejected at save time",
+ api_base=settings["api_base"],
+ api_key="openrag-live-validation-invalid-key",
+ scenario="invalid",
+ )
+ if settings.get("listing_denied_api_key"):
+ await _probe_credential(
+ matrix,
+ number=2,
+ name="listing-denied credential: what Foundry actually returns",
+ api_base=settings["api_base"],
+ api_key=settings["listing_denied_api_key"],
+ scenario="listing_denied",
+ )
+ else:
+ matrix.record(
+ 2,
+ "listing-denied credential: what Foundry actually returns",
+ ADVISORY,
+ "skipped",
+ "set AZURE_AI_LISTING_DENIED_API_KEY to settle the 403 question; "
+ "without it, a 403 on a valid key stays inconclusive and blocking",
+ )
+ await _check_docling_vlm(matrix, settings["api_base"], settings)
+ await _check_legacy_header_behaviour(
+ matrix, settings.get("legacy_api_base"), settings["api_key"]
+ )
+ _check_cost_attribution(matrix, settings["api_base"], settings["chat_deployment"])
+ # Last: it scans everything the run produced.
+ _check_secret_leak(matrix, _REAL_SECRETS)
+
+
+def _print_report(matrix: Matrix) -> None:
+ print("\n" + "=" * 78)
+ print("Azure AI Foundry live validation")
+ print("=" * 78)
+ symbols = {"pass": "PASS", "fail": "FAIL", "skipped": "SKIP"}
+ for check in sorted(matrix.checks, key=lambda c: (c.number, c.name)):
+ tag = "blocker " if check.severity == BLOCKER else "advisory"
+ print(f" [{symbols[check.status]}] ({tag}) {check.number:>2}. {check.name}")
+ if check.detail:
+ print(f" {_redact(check.detail)}")
+ print("-" * 78)
+
+
+def _exit_code(matrix: Matrix) -> int:
+ blocking_failures = [c for c in matrix.checks if c.severity == BLOCKER and c.status == "fail"]
+ blocking_skips = [c for c in matrix.checks if c.severity == BLOCKER and c.status == "skipped"]
+ if blocking_failures:
+ print(f"{len(blocking_failures)} blocker(s) failed. Do not enable the provider.")
+ return 1
+ if blocking_skips:
+ print(
+ f"{len(blocking_skips)} blocker(s) were not exercised. "
+ "The matrix is incomplete; do not enable the provider yet."
+ )
+ return 1
+ print("All blockers passed. The visibility flip can proceed.")
+ return 0
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.parse_args()
+ try:
+ raise SystemExit(asyncio.run(_main()))
+ except KeyboardInterrupt:
+ raise SystemExit(130) from None
+ except SystemExit:
+ raise
+ except Exception: # pragma: no cover - operator-facing script
+ traceback.print_exc()
+ raise SystemExit(1) from None
diff --git a/tests/unit/test_validate_azure_foundry_live.py b/tests/unit/test_validate_azure_foundry_live.py
new file mode 100644
index 000000000..766d2ddad
--- /dev/null
+++ b/tests/unit/test_validate_azure_foundry_live.py
@@ -0,0 +1,418 @@
+"""The live-validation harness's own safety properties.
+
+`scripts/validate_azure_foundry_live.py` is run by hand against a real Foundry
+resource, so its failure modes are not otherwise exercised by CI. Two of them
+are worth pinning: it must never put a real credential on a terminal, and the
+checks that pass by observing a rejection must not pass on an unrelated one.
+
+The leak tests matter because detection alone is not enough. Check 11 finds a
+leak *after* the run, by which point anything written to stdout or stderr is
+already on screen and, in CI, already in a retained build log.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import io
+import sys
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent.parent
+SCRIPT = ROOT / "scripts" / "validate_azure_foundry_live.py"
+
+REAL_KEY = "primary-credential-must-never-be-printed"
+LISTING_DENIED_KEY = "listing-denied-credential-also-secret"
+
+
+def harness():
+ """The script as an importable module, with its secret list reset."""
+ spec = importlib.util.spec_from_file_location("azure_foundry_live_validation", SCRIPT)
+ assert spec and spec.loader
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[spec.name] = module
+ spec.loader.exec_module(module)
+ try:
+ yield module
+ finally:
+ module._set_real_secrets()
+ sys.modules.pop(spec.name, None)
+
+
+class TestRedaction:
+ def test_every_supplied_credential_is_redacted(self, harness) -> None:
+ harness._set_real_secrets(REAL_KEY, LISTING_DENIED_KEY)
+ text = f"key={REAL_KEY} other={LISTING_DENIED_KEY}"
+
+ cleaned = harness._redact(text)
+
+ assert REAL_KEY not in cleaned
+ assert LISTING_DENIED_KEY not in cleaned
+ assert cleaned.count("<redacted>") == 2
+
+ def test_a_short_value_is_not_treated_as_a_secret(self, harness) -> None:
+ """Blanking a two-character value would destroy the output instead."""
+ harness._set_real_secrets("ab")
+ assert harness._redact("a table of absolute values") == "a table of absolute values"
+
+ def test_blank_credentials_are_ignored(self, harness) -> None:
+ harness._set_real_secrets("", None or "")
+ assert harness._REAL_SECRETS == ()
+
+
+class TestTeeDoesNotLeakToTheTerminal:
+ """The stream forwards redacted text and keeps the original for detection."""
+
+ @pytest.mark.parametrize("secret", [REAL_KEY, LISTING_DENIED_KEY])
+ def test_the_real_stream_never_receives_the_secret(self, harness, secret) -> None:
+ harness._set_real_secrets(REAL_KEY, LISTING_DENIED_KEY)
+ terminal = io.StringIO()
+ sink: list[str] = []
+ tee = harness._Tee(terminal, sink)
+
+ tee.write(f"litellm debug: authorization=Bearer {secret}\n")
+
+ assert secret not in terminal.getvalue()
+ assert "<redacted>" in terminal.getvalue()
+ # ...but the capture keeps the original, or check 11 could not detect it.
+ assert any(secret in fragment for fragment in sink)
+
+
+class TestCheckElevenDetectsEitherCredential:
+ @pytest.mark.parametrize(
+ "leaked, index",
+ [(REAL_KEY, 1), (LISTING_DENIED_KEY, 2)],
+ )
+ def test_a_forced_leak_of_either_credential_fails_the_check(
+ self, harness, leaked, index
+ ) -> None:
+ matrix = harness.Matrix()
+ matrix.capture(f"some library wrote {leaked} to stderr")
+
+ harness._check_secret_leak(matrix, (REAL_KEY, LISTING_DENIED_KEY))
+
+ check = matrix.checks[-1]
+ assert check.status == "fail"
+ assert check.severity == harness.BLOCKER
+ assert f"credential #{index}" in check.detail
+
+ def test_the_failure_detail_never_contains_the_secret(self, harness) -> None:
+ """Reporting a leak must not be the leak."""
+ matrix = harness.Matrix()
+ matrix.capture(f"leaked {REAL_KEY} here")
+
+ harness._check_secret_leak(matrix, (REAL_KEY,))
+
+ assert REAL_KEY not in matrix.checks[-1].detail
+
+ def test_a_clean_run_passes_and_says_how_much_it_scanned(self, harness) -> None:
+ matrix = harness.Matrix()
+ matrix.capture("nothing sensitive at all")
+
+ harness._check_secret_leak(matrix, (REAL_KEY, LISTING_DENIED_KEY))
+
+ check = matrix.checks[-1]
+ assert check.status == "pass"
+ assert "2 credential(s)" in check.detail
+
+ def test_no_credentials_to_scan_is_a_failure_not_a_pass(self, harness) -> None:
+ """An empty secret list would otherwise pass vacuously."""
+ matrix = harness.Matrix()
+
+ harness._check_secret_leak(matrix, ())
+
+ assert matrix.checks[-1].status == "fail"
+
+
+class TestTheReportCannotPrintASecret:
+ @pytest.mark.parametrize("secret", [REAL_KEY, LISTING_DENIED_KEY])
+ def test_a_secret_echoed_into_a_check_detail_is_redacted_on_print(
+ self, harness, secret, capsys
+ ) -> None:
+ """A provider can echo the credential back in an error body.
+
+ That body ends up in a check's detail, so the report has to redact on
+ the way out even though check 11 has already flagged it.
+ """
+ harness._set_real_secrets(REAL_KEY, LISTING_DENIED_KEY)
+ matrix = harness.Matrix()
+ matrix.record(
+ 2,
+ "invalid API key is rejected at save time",
+ harness.BLOCKER,
+ "fail",
+ f'upstream echoed the credential: body={{"key": "{secret}"}}',
+ )
+
+ harness._print_report(matrix)
+
+ assert secret not in capsys.readouterr().out
+
+
+class TestNotFoundClassification:
+ """A bare 404 is not evidence: those digits appear in ids and counters."""
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "DeploymentNotFound",
+ "The API deployment for this resource does not exist",
+ "model_not_found",
+ "HTTP 404",
+ "HTTP/1.1 404",
+ "status_code: 404",
+ "status=404",
+ "404 Not Found",
+ ],
+ )
+ def test_a_real_not_found_is_recognised(self, harness, text) -> None:
+ assert harness._looks_like_not_found(text)
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "request id 8a404b2c",
+ "used 404 prompt tokens",
+ "completed in 404ms",
+ "x-ms-request-id: 404aa1",
+ "rate limit exceeded",
+ ],
+ )
+ def test_an_incidental_404_is_not_mistaken_for_one(self, harness, text) -> None:
+ assert not harness._looks_like_not_found(text)
+
+
+class TestFailureCategories:
+ """Checks that pass on a rejection must not pass on an unrelated failure."""
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "ConnectError: nodename nor servname provided, or not known",
+ "Cannot connect to host contoso.services.ai.azure.com:443",
+ "ReadTimeout",
+ "certificate verify failed",
+ ],
+ )
+ def test_transport_failures_are_identified_as_such(self, harness, text) -> None:
+ assert harness._looks_like_connectivity_failure(text)
+ assert not harness._looks_like_credential_failure(text)
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "Access denied due to invalid subscription key",
+ "401 Unauthorized",
+ "Incorrect API key provided",
+ "HTTP 401",
+ "status_code: 401",
+ ],
+ )
+ def test_credential_rejections_are_identified_as_such(self, harness, text) -> None:
+ assert harness._looks_like_credential_failure(text)
+ assert not harness._looks_like_connectivity_failure(text)
+
+ @pytest.mark.parametrize(
+ "text",
+ [
+ "request id 8a401b2c",
+ "completed in 401 ms",
+ "prompt_tokens: 401",
+ ],
+ )
+ def test_an_incidental_401_is_not_mistaken_for_a_rejection(self, harness, text) -> None:
+ """Same reasoning as the 404 case above.
+
+ Check 2 runs this classifier over 300 characters of response body, so
+ a request id carrying those three digits would otherwise let a
+ non-credential rejection pass as a credential one.
+ """
+ assert not harness._looks_like_credential_failure(text)
+
+
+class TestTheRoutingCheckCanFail:
+ """A blocker that always passes is not a check.
+
+ Both paths below return before any network call, which is what makes them
+ testable without a live resource.
+ """
+
+ @pytest.fixture
+ def settings(self):
+ return {
+ "api_key": REAL_KEY,
+ "chat_deployment": "chat-dep",
+ "embedding_deployment": "embed-dep",
+ }
+
+ @pytest.mark.asyncio
+ async def test_the_wrong_surface_is_reported_rather_than_passed(
+ self, harness, settings, monkeypatch
+ ) -> None:
+ """A legacy `/models` URL in AZURE_AI_API_BASE used to satisfy the
+ `/openai/v1` blocker, which is the one thing the matrix exists to
+ prove."""
+ from enhancements.providers.azure import foundry
+
+ monkeypatch.setattr(foundry, "endpoint_profile", lambda _b: "legacy_models")
+ monkeypatch.setattr(foundry, "litellm_route", lambda _s: "azure_ai")
+
+ matrix = harness.Matrix()
+ await harness._run_endpoint_matrix(
+ matrix,
+ "resource v1",
+ "https://contoso.services.ai.azure.com/models",
+ settings,
+ base_number=8,
+ expected_profile="resource_openai_v1",
+ )
+
+ assert [c.status for c in matrix.checks] == ["fail"]
+ assert "expected resource_openai_v1" in matrix.checks[0].detail
+
+ @pytest.mark.asyncio
+ async def test_an_unclassifiable_endpoint_does_not_abort_the_run(
+ self, harness, settings, monkeypatch
+ ) -> None:
+ """`litellm_route` refuses what it cannot classify, by design.
+
+ Uncaught, that exception escaped `_run_all_checks` and the report was
+ never printed — a traceback instead of the matrix, which is the least
+ useful thing a release gate can hand an operator.
+ """
+ from enhancements.providers.azure import foundry
+
+ def _refuse(_stored):
+ raise foundry.UnsupportedEndpointError(foundry.UNSUPPORTED_ENDPOINT_MESSAGE)
+
+ monkeypatch.setattr(foundry, "endpoint_profile", lambda _b: "unknown")
+ monkeypatch.setattr(foundry, "litellm_route", _refuse)
+
+ matrix = harness.Matrix()
+ await harness._run_endpoint_matrix(
+ matrix,
+ "resource v1",
+ "https://contoso.services.ai.azure.com/anthropic/v1",
+ settings,
+ base_number=8,
+ expected_profile="resource_openai_v1",
+ )
+
+ assert [c.status for c in matrix.checks] == ["fail"]
+ assert "UnsupportedEndpointError" in matrix.checks[0].detail
+
+
+class TestBlockersRejectUnusableSuccesses:
+ """A 200 is not the same as a working deployment.
+
+ Both of these recorded a pass on a successful-but-empty response, which
+ made the release matrix green without usable inference — the one outcome
+ a blocker here has to rule out.
+ """
+
+ @pytest.mark.parametrize("text", ["", " ", "\n"])
+ def test_an_empty_completion_is_not_a_pass(self, harness, text) -> None:
+ assert harness._unusable_chat_reply(text)
+
+ def test_a_real_completion_passes(self, harness) -> None:
+ assert harness._unusable_chat_reply("OK") == ""
+
+ @pytest.mark.parametrize(
+ "vector",
+ [
+ [],
+ None,
+ "not-a-vector",
+ [0.1, float("nan")],
+ [float("inf"), 0.2],
+ [0.1, "0.2"],
+ [0.1, True],
+ ],
+ )
+ def test_an_unusable_vector_is_not_a_pass(self, harness, vector) -> None:
+ """Length alone is not enough: a vector of NaN has the right shape and
+ is useless for retrieval, and OpenSearch rejects it at index time
+ rather than here."""
+ assert harness._unusable_embedding(vector)
+
+ def test_a_real_vector_passes(self, harness) -> None:
+ assert harness._unusable_embedding([0.1, -0.2, 3]) == ""
+
+
+class TestTheEndpointIsNeverPrintedRaw:
+ """`_redact` only knows the credentials this run was handed.
+
+ A secret carried *inside* the endpoint is invisible to it, and the
+ provider module strips or refuses one only after this line is on screen.
+ """
+
+ def test_userinfo_and_query_are_removed(self, harness) -> None:
+ shown = harness._safe_endpoint(
+ "https://operator:[email protected]/openai/v1?sig=abc123"
+ )
+ assert "s3cr3t" not in shown
+ assert "operator" not in shown
+ assert "abc123" not in shown
+ assert "sig" not in shown
+ assert shown.startswith("https://contoso.services.ai.azure.com/openai/v1")
+ assert "userinfo" in shown and "query" in shown
+
+ def test_an_ordinary_endpoint_is_left_alone(self, harness) -> None:
+ plain = "https://contoso.services.ai.azure.com/openai/v1"
+ assert harness._safe_endpoint(plain) == plain
+
+ def test_a_non_url_is_passed_through(self, harness) -> None:
+ assert harness._safe_endpoint("not-a-url") == "not-a-url"
+
+
+class TestPreboundLogHandlersCannotLeak:
+ """Replacing `sys.stderr` does not retarget a handler that already holds it.
+
+ litellm attaches three such handlers at import, and it is the library most
+ likely to log a request detail carrying a credential — so without this the
+ harness's central promise, that nothing reaches a terminal unredacted,
+ does not hold for the most likely source.
+ """
+
+ def test_a_handler_bound_before_the_block_is_redirected(self, harness, monkeypatch) -> None:
+ import logging
+
+ # The handler must hold the *same object* `sys.stderr` names when
+ # `_capturing` is entered — that is the situation litellm creates at
+ # import, and the one swapping `sys.stderr` fails to cover. A handler
+ # pointed somewhere else entirely is a log file, not a terminal, and
+ # is deliberately left alone.
+ terminal = io.StringIO()
+ monkeypatch.setattr(sys, "stderr", terminal)
+ logger = logging.getLogger("test.prebound.handler")
+ logger.propagate = False
+ handler = logging.StreamHandler(terminal)
+ logger.addHandler(handler)
+ logger.setLevel(logging.INFO)
+
+ harness._set_real_secrets(REAL_KEY)
+ sink: list[str] = []
+ try:
+ with harness._capturing(sink):
+ logger.info("leaking %s now", REAL_KEY)
+ finally:
+ logger.removeHandler(handler)
+
+ assert REAL_KEY not in terminal.getvalue(), "the secret reached the real stream"
+ assert handler.stream is terminal, "the handler's stream was not restored"
+ assert any(REAL_KEY in entry for entry in sink), "detection still needs the original"
+
+
+class TestImageFixture:
+ def test_the_probe_image_is_large_enough_to_be_accepted(self, harness) -> None:
+ """A 1x1 PNG is valid but some vision deployments reject it on size,
+ which would read as an integration failure rather than a fixture one."""
+ import struct
+
+ png = harness._sample_png()
+
+ assert png[:8] == b"\x89PNG\r\n\x1a\n"
+ width, height = struct.unpack(">II", png[16:24])
+ assert width == height == 128