OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.
Python
test: live validation matrix for Azure AI Foundry, ahead of the flip (#2447)
Rico Furtado51bc93d50a01
diff --git a/scripts/README.md b/scripts/README.mdindex ea289eefc..487cce993 100644--- a/scripts/README.md+++ b/scripts/README.md@@ -250,3 +250,89 @@ After running, **restart the backend** and reload the app.This script does **not** delete ingested documents, knowledge filters, Langflowflows, or conversations. For a full teardown, use the authenticated API endpoint`POST /settings/rollback-onboarding` instead.++## Azure AI Foundry live validation (`validate_azure_foundry_live.py`)++The matrix that must pass against a **real** Azure AI Foundry resource before+`azure_ai` is made visible in any run mode. The provider stays hidden while it+runs: credentials come from the environment and the config object is built in+memory, so nothing reads `model_providers.yaml` or the saved configuration.++Every check goes through OpenRAG's own code — the provider enhancement, the+LLM gateway, the Docling option builder — rather than reimplementing the+requests. A script that built its own URLs would prove Azure works, not that+OpenRAG calls it correctly, which is the question the flip depends on.++```bash+export AZURE_AI_API_BASE="https://<resource>.services.ai.azure.com/openai/v1"+export AZURE_AI_API_KEY="<key>"+export AZURE_AI_CHAT_DEPLOYMENT="<chat deployment>"+export AZURE_AI_EMBEDDING_DEPLOYMENT="<embedding deployment>"++# Optional; each unlocks more of the matrix.+export AZURE_AI_PROJECT_API_BASE="https://<r>.services.ai.azure.com/api/projects/<p>/openai/v1"+export AZURE_AI_LEGACY_API_BASE="https://<resource>.services.ai.azure.com/models"+export AZURE_AI_VLM_DEPLOYMENT="<vision-capable deployment>"++uv run python scripts/validate_azure_foundry_live.py+```++Exit status is non-zero if any **blocker** fails *or is skipped* — an+unexercised blocker leaves the matrix incomplete, which is not a pass.++| # | Check | |+|---|---|---|+| 1 | Save-time validation, valid credentials | blocker |+| 2 | Invalid API key is rejected | blocker |+| 3 | Chat | blocker |+| 4 | Streaming | blocker |+| 5 | Tool calling | blocker |+| 6 | Embeddings | blocker |+| 7 | Unknown deployment fails cleanly | blocker |+| 8 | `/openai/v1` routing | blocker |+| 9 | `/api/projects/<project>/openai/v1` routing | blocker when configured |+| 10 | Docling VLM picture description | blocker |+| 11 | No credential in errors or logs | blocker |+| 12 | Legacy `/models` `api-key` vs `Bearer` | advisory |+| 13 | Cost attribution | advisory |++**The 403 question.** `foundry.lightweight_health_check` raises+`PermissionError` for a 403 on the deployment listing, on the reasoning that+listing can need a permission inference does not. The product does not act on+that distinction: `api/settings/endpoints.py` catches `Exception` around+provider validation and returns HTTP 400, so the provider cannot be saved+either way. A 403 on a valid credential is therefore reported as+**inconclusive and blocking**, not as a pass. Set+`AZURE_AI_LISTING_DENIED_API_KEY` to capture what such a credential actually+returns; the decision — require listing permission, or make a specific denial+non-blocking — needs that response alongside the invalid-key response, and is+only safe if the two are reliably distinguishable.++Checks 3–7 run once per configured endpoint.++Cost is advisory by design: OpenRAG does not consume cost data today, and the+OpenAI-compatible transport's loss of LiteLLM's `azure_ai/*` price rows is+documented in `enhancements/providers/azure/foundry.py`. Check 12 is advisory+because the shipped health check sends both header styles; its answer only+decides whether that can later be simplified.++The API key is never printed. The run captures root logging, stdout, stderr+and every recorded exception, and check 11 scans all of it. When a leak is+found the captured material is **not** reproduced — only the fact and the+number of matching fragments — and every printed line is redacted regardless,+since several checks quote an upstream response body.++Checks that pass by observing a rejection assert the *category* of failure:+the invalid-key check passes only on a credential rejection, and the+unknown-deployment check only on a not-found through the gateway's own error+contract. Neither accepts a DNS, TLS, timeout or 5xx failure, which would+otherwise let an unreachable endpoint look like a successful rejection.++**Two things the script cannot cover**, both needed before the flip:++- Check 10 issues the exact URL, headers and params OpenRAG hands+ docling-serve, but does not run docling-serve. Do one real ingest of an+ image-bearing document through the running stack with picture descriptions+ enabled.+- Nothing here exercises the settings UI. Save the provider once through+ **Settings > Providers** to confirm the form and its validation behave.diff --git a/scripts/validate_azure_foundry_live.py b/scripts/validate_azure_foundry_live.pynew file mode 100644index 000000000..119c286e2--- /dev/null+++ b/scripts/validate_azure_foundry_live.py@@ -0,0 +1,1378 @@+#!/usr/bin/env python3+"""Live validation matrix for the Azure AI Foundry provider.++Runs the checks that must pass against a **real Foundry resource** before+`azure_ai` is made visible in any run mode. Everything here is deliberately+read through OpenRAG's own code — `enhancements.providers.azure.foundry`,+`services.llm_gateway`, `services.docling_service` — rather than reimplemented+with `curl`-equivalent requests. A script that built its own URLs would prove+Azure works, not that OpenRAG calls it correctly, which is the open question.++The provider stays hidden while this runs. Nothing here reads+`model_providers.yaml` or the saved configuration: credentials come from the+environment and a config object is assembled in memory, so the matrix can be+run against production code with the feature still switched off.++Usage+-----+ export AZURE_AI_API_BASE="https://<resource>.services.ai.azure.com/openai/v1"+ export AZURE_AI_API_KEY="<key>"+ export AZURE_AI_CHAT_DEPLOYMENT="<your chat deployment>"+ export AZURE_AI_EMBEDDING_DEPLOYMENT="<your embedding deployment>"++ # Optional, each unlocks more of the matrix:+ export AZURE_AI_PROJECT_API_BASE="https://<r>.services.ai.azure.com/api/projects/<p>/openai/v1"+ export AZURE_AI_LEGACY_API_BASE="https://<resource>.services.ai.azure.com/models"+ export AZURE_AI_VLM_DEPLOYMENT="<a vision-capable deployment>"+ # A credential that can run inference but NOT list deployments, if one can+ # be created. See "The 403 question" below — this is the scenario that+ # decides it.+ export AZURE_AI_LISTING_DENIED_API_KEY="<key with inference but not listing>"++ uv run python scripts/validate_azure_foundry_live.py++Exit code is non-zero if any **blocker** fails. Blockers are the capabilities+the product claims once the provider is visible: supported endpoint routing,+chat, streaming, tool calling, embeddings, Docling VLM, and error+sanitization. Cost attribution is reported but never blocks — OpenRAG does not+consume cost data today (see the transport trade-off in `foundry.py`).++The 403 question+----------------+`foundry.lightweight_health_check` raises `PermissionError` for a 403 on the+deployment listing, on the reasoning that listing can need a permission+inference does not. **But the product does not act on that distinction.**+`api/settings/endpoints.py` catches `Exception` around provider validation and+returns HTTP 400, so a `PermissionError` is indistinguishable from a bad key:+the provider cannot be saved either way.++So a 403 on a valid credential is *not* treated as a pass here. This script+gathers the evidence needed to decide between two fixes — require listing+permission and say so, or make a specifically-identified listing denial+non-blocking in the settings endpoint — by recording exactly what Foundry+returns for:++ * a valid key with normal permissions;+ * a deliberately invalid key;+ * a valid credential that can infer but not list, when one is available+ (`AZURE_AI_LISTING_DENIED_API_KEY`).++Until that evidence exists, a generic 403 is reported as **inconclusive and+blocking**. Do not relax it on the strength of a status code alone: the+distinction is only safe if the responses are reliably distinguishable.++Secrets+-------+The API key is never printed. The run captures root logging, stdout, stderr+and every recorded exception, and check 11 scans all of it. When a secret is+found the captured material is **not** printed — only the fact and where.++What this cannot cover+----------------------+Check 10 exercises the exact URL, headers and params OpenRAG hands+docling-serve, by issuing that request itself. It does not run docling-serve.+Before the flip, also do one real ingest of an image-bearing document through+the running stack with picture descriptions enabled, to confirm docling-serve+accepts the block as given.+"""++from __future__ import annotations++import argparse+import asyncio+import base64+import contextlib+import json+import logging+import math+import os+import re+import struct+import sys+import traceback+import zlib+from dataclasses import dataclass, field+from pathlib import Path+from types import SimpleNamespace+from typing import Any+from urllib.parse import urlsplit, urlunsplit++ROOT = Path(__file__).resolve().parent.parent+SRC = ROOT / "src"+for path in (str(SRC), str(ROOT)):+ if path not in sys.path:+ sys.path.insert(0, path)+++# --------------------------------------------------------------------------+# Result collection+# --------------------------------------------------------------------------++BLOCKER = "blocker"+ADVISORY = "advisory"+++@dataclass+class Check:+ number: int+ name: str+ severity: str+ status: str = "skipped"+ detail: str = ""++ @property+ def ok(self) -> bool:+ return self.status in ("pass", "skipped")+++@dataclass+class Matrix:+ checks: list[Check] = field(default_factory=list)+ #: Everything the run produced that a secret could hide in.+ captured_text: list[str] = field(default_factory=list)++ def record(self, number: int, name: str, severity: str, status: str, detail: str = "") -> Check:+ check = Check(number, name, severity, status, detail)+ self.checks.append(check)+ self.captured_text.append(detail)+ return check++ def capture(self, text: str) -> None:+ self.captured_text.append(text)+++def _unusable_chat_reply(text: str) -> str:+ """Why `text` is not a usable completion, or "" if it is."""+ return "" if text.strip() else "the completion succeeded with empty content"+++def _unusable_embedding(vector: Any) -> str:+ """Why `vector` cannot be used as an embedding, or "" if it can."""+ if not isinstance(vector, list) or not vector:+ return "the call succeeded but returned an empty embedding"+ for value in vector:+ if isinstance(value, bool) or not isinstance(value, (int, float)):+ return f"the embedding contains a non-numeric value ({type(value).__name__})"+ if not math.isfinite(value):+ return "the embedding contains a non-finite value (nan or inf)"+ return ""+++def _describe_exception(exc: BaseException) -> str:+ return f"{type(exc).__name__}: {exc}"+++#: Every real credential this run was given. Populated once from the+#: environment and used to redact anything on its way to a terminal — the+#: in-memory capture keeps the raw text so check 11 can still detect a leak.+_REAL_SECRETS: tuple[str, ...] = ()+++def _set_real_secrets(*secrets: str) -> None:+ global _REAL_SECRETS+ # Short values are not redacted: a two-character "secret" would blank out+ # unrelated text and make the output useless.+ _REAL_SECRETS = tuple(dict.fromkeys(s for s in secrets if s and len(s) >= 8))+++def _safe_endpoint(api_base: str) -> str:+ """An endpoint with anything secret-bearing taken off, safe to display.++ `_redact` only knows the credentials this run was handed, so it cannot+ help with a secret carried *inside* the endpoint: `user:password@` in the+ netloc, or a SAS token in the query. The provider module refuses or strips+ both, but that happens after the operator has already seen this line.+ """+ parts = urlsplit(api_base or "")+ if not parts.scheme or not parts.netloc:+ return api_base or ""+ host = parts.hostname or ""+ if parts.port:+ host = f"{host}:{parts.port}"+ cleaned = urlunsplit((parts.scheme, host, parts.path, "", ""))+ dropped = []+ if parts.username or parts.password:+ dropped.append("userinfo")+ if parts.query:+ dropped.append("query")+ return f"{cleaned} (dropped: {', '.join(dropped)})" if dropped else cleaned+++def _redact(text: str) -> str:+ """Remove every known real credential from text about to be displayed.++ Check 11 *detects* a leak after the fact, which is too late on its own:+ by then a library has already written the secret to stdout or stderr and,+ in CI, into a build log that outlives the run. So nothing reaches a real+ stream without passing through here first.+ """+ cleaned = text+ for secret in _REAL_SECRETS:+ cleaned = cleaned.replace(secret, "<redacted>")+ return cleaned+++#: A failure that says nothing about credentials or deployments — the request+#: did not get a considered answer. Any check that "passes on an error" has to+#: exclude these, or an unplugged network reads as a successful rejection.+_CONNECTIVITY_MARKERS = (+ "nodename nor servname",+ "name or service not known",+ "temporary failure in name resolution",+ "cannot connect to host",+ "connection refused",+ "connection reset",+ "connect call failed",+ "connecterror",+ "connecttimeout",+ "readtimeout",+ "timeout",+ "timed out",+ "certificate",+ "ssl",+ "tlsv1",+ "proxyerror",+)++#: Upstream said the deployment or model is not there. Azure words this+#: differently across its two surfaces, so both vocabularies are covered.+_NOT_FOUND_MARKERS = (+ "deploymentnotfound",+ "deployment not found",+ "does not exist",+ "model_not_found",+ "model not found",+ "unknown model",+ "no deployment",+ "resourcenotfound",+ "404 not found",+)++#: A bare "404" is not a marker: those three digits appear in request ids,+#: timestamps and token counts, and matching them anywhere would let an+#: unrelated response pass the unknown-deployment check. Recognised only where+#: the surrounding text makes it a status.+_HTTP_404_PATTERN = re.compile(+ r"(?:http[/ ]?[\d.]*\s*|status(?:_?code)?\s*[:=]\s*|code\s*[:=]\s*)404\b",+ re.IGNORECASE,+)++#: And a bare "401" is not one either, for the same reason: check 2 runs this+#: classifier over 300 characters of response body, where a request id like+#: `8a401b2c` would otherwise read as a rejected credential and let a+#: non-credential failure pass. The real status is already covered by the+#: `status == 401` test at the call site.+_HTTP_401_PATTERN = re.compile(+ r"(?:http[/ ]?[\d.]*\s*|status(?:_?code)?\s*[:=]\s*|code\s*[:=]\s*)401\b",+ re.IGNORECASE,+)+++def _looks_like_connectivity_failure(text: str) -> bool:+ lowered = (text or "").lower()+ return any(marker in lowered for marker in _CONNECTIVITY_MARKERS)+++def _looks_like_not_found(text: str) -> bool:+ lowered = (text or "").lower()+ if any(marker in lowered for marker in _NOT_FOUND_MARKERS):+ return True+ return bool(_HTTP_404_PATTERN.search(lowered))+++def _looks_like_credential_failure(text: str) -> bool:+ """Upstream rejected the credential, per OpenRAG's own classifier."""+ from api.provider_validation import is_provider_credential_error++ lowered = (text or "").lower()+ if is_provider_credential_error(lowered):+ return True+ # Azure's own vocabulary for a rejected key, which the shared marker list+ # does not have to know about.+ if _HTTP_401_PATTERN.search(lowered):+ return True+ return any(+ marker in lowered+ for marker in (+ "unauthorized",+ "access denied due to invalid subscription key",+ "invalid subscription key",+ "permissiondenied",+ "invalid authentication",+ "accessdenied",+ )+ )+++# --------------------------------------------------------------------------+# In-memory config, so nothing depends on the provider being enabled or saved+# --------------------------------------------------------------------------+++def _build_config(api_base: str, api_key: str, chat_model: str, embedding_model: str):+ from config.config_manager import (+ AnthropicConfig,+ GenericProviderConfig,+ OllamaConfig,+ OpenAIConfig,+ ProvidersConfig,+ WatsonXConfig,+ )++ providers = ProvidersConfig(+ openai=OpenAIConfig(),+ anthropic=AnthropicConfig(),+ watsonx=WatsonXConfig(),+ ollama=OllamaConfig(),+ custom={+ "azure_ai": GenericProviderConfig(+ credentials={"api_base": api_base, "api_key": api_key},+ configured=True,+ )+ },+ )+ return SimpleNamespace(+ providers=providers,+ agent=SimpleNamespace(llm_model=chat_model, llm_provider="azure_ai"),+ knowledge=SimpleNamespace(embedding_model=embedding_model, embedding_provider="azure_ai"),+ )+++# --------------------------------------------------------------------------+# Per-endpoint matrix+# --------------------------------------------------------------------------+++async def _run_endpoint_matrix(+ matrix: Matrix,+ label: str,+ api_base: str,+ settings: dict[str, str],+ *,+ base_number: int,+ expected_profile: str | None = None,+) -> None:+ """Save check, chat, streaming, tools, embeddings and a bad deployment."""+ from enhancements.providers.azure import foundry+ from services.llm_gateway import chat_completions, embeddings++ api_key = settings["api_key"]+ chat_model = settings["chat_deployment"]+ embedding_model = settings["embedding_deployment"]+ config = _build_config(api_base, api_key, chat_model, embedding_model)++ # Classification is a blocker in its own right, so it has to be able to+ # *fail*. Recording an unconditional pass meant a legacy `/models`+ # endpoint pasted into AZURE_AI_API_BASE still satisfied the `/openai/v1`+ # check, which is the one thing this matrix exists to prove.+ profile = foundry.endpoint_profile(api_base)+ try:+ route = foundry.litellm_route({"api_base": api_base})+ except Exception as exc:+ # `litellm_route` refuses an endpoint it cannot classify, by design.+ # Uncaught it would abort the whole run before `_print_report`, so the+ # operator would get a traceback instead of the matrix — the least+ # useful outcome for a release gate.+ matrix.record(+ base_number,+ f"[{label}] endpoint classified and routed",+ BLOCKER,+ "fail",+ f"profile={profile} route=<unresolved> {_describe_exception(exc)}",+ )+ return++ mismatch = expected_profile is not None and profile != expected_profile+ matrix.record(+ base_number,+ f"[{label}] endpoint classified and routed",+ BLOCKER,+ "fail" if mismatch else "pass",+ f"profile={profile} route={route}"+ + (f" (expected {expected_profile})" if mismatch else ""),+ )+ if mismatch:+ # Everything after this would be measuring a different surface than+ # the one this block of the matrix claims to cover.+ return++ # 1. Save-time validation with valid credentials. Delegated to the+ # credential-semantics probe so the raw response is on record, not just+ # whether an exception was raised.+ await _probe_credential(+ matrix,+ number=1,+ name=f"[{label}] save-time check with valid credentials",+ api_base=api_base,+ api_key=api_key,+ scenario="valid",+ )++ # 3. Chat.+ try:+ response = await chat_completions(+ {+ "model": f"azure_ai:{chat_model}",+ "messages": [{"role": "user", "content": "Reply with the word OK."}],+ "max_tokens": 16,+ },+ config=config,+ )+ text = (response["choices"][0]["message"].get("content") or "").strip()+ # A completion that succeeds and says nothing is not a working+ # deployment. Recorded as a pass it made the release matrix green+ # without usable inference, which is the one thing a blocker here+ # exists to rule out.+ problem = _unusable_chat_reply(text)+ matrix.record(+ 3,+ f"[{label}] chat",+ BLOCKER,+ "fail" if problem else "pass",+ problem or f"reply={text[:40]!r}",+ )+ except Exception as exc:+ matrix.record(3, f"[{label}] chat", BLOCKER, "fail", _describe_exception(exc))++ # 4. Streaming. The contract is `data: {...}` frames terminated by+ # `data: [DONE]`, so what is asserted is a well-formed stream that+ # completes — not a frame count. A short completion may legitimately+ # arrive in a single content chunk, and counting frames would also have+ # passed a *failed* stream, because the gateway reports failure as an+ # error frame followed by [DONE].+ try:+ stream = await chat_completions(+ {+ "model": f"azure_ai:{chat_model}",+ "messages": [{"role": "user", "content": "Count from one to five."}],+ "max_tokens": 64,+ "stream": True,+ },+ config=config,+ )+ frames = 0+ content_chunks = 0+ saw_done = False+ error_frame = ""+ async for frame in stream:+ frames += 1+ payload = (+ frame[len("data: ") :].strip() if frame.startswith("data: ") else frame.strip()+ )+ if payload == "[DONE]":+ saw_done = True+ continue+ try:+ parsed = json.loads(payload)+ except (ValueError, TypeError):+ continue+ if isinstance(parsed, dict) and parsed.get("error"):+ error_frame = str(parsed["error"])[:160]+ continue+ for choice in (parsed or {}).get("choices") or []:+ delta = choice.get("delta") or {}+ if delta.get("content") or delta.get("tool_calls"):+ content_chunks += 1++ if error_frame:+ matrix.record(4, f"[{label}] streaming", BLOCKER, "fail", f"error frame: {error_frame}")+ elif not content_chunks:+ matrix.record(+ 4,+ f"[{label}] streaming",+ BLOCKER,+ "fail",+ f"{frames} frames but none carried content or tool calls",+ )+ elif not saw_done:+ matrix.record(+ 4,+ f"[{label}] streaming",+ BLOCKER,+ "fail",+ f"{content_chunks} content chunk(s) but the stream never terminated with [DONE]",+ )+ else:+ matrix.record(+ 4,+ f"[{label}] streaming",+ BLOCKER,+ "pass",+ f"{content_chunks} content chunk(s) of {frames} frames, terminated cleanly",+ )+ except Exception as exc:+ matrix.record(4, f"[{label}] streaming", BLOCKER, "fail", _describe_exception(exc))++ # 5. Tool calling. Forced with `tool_choice`, because the question here+ # is whether tool calls survive the transport — not whether this+ # particular model independently decides a tool is warranted. Leaving+ # that to the model makes the gate flaky and provider-dependent.+ tools = [+ {+ "type": "function",+ "function": {+ "name": "get_weather",+ "description": "Look up the current weather for a city.",+ "parameters": {+ "type": "object",+ "properties": {"city": {"type": "string", "description": "City name"}},+ "required": ["city"],+ },+ },+ }+ ]+ try:+ response = await chat_completions(+ {+ "model": f"azure_ai:{chat_model}",+ "messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],+ "tools": tools,+ "tool_choice": {"type": "function", "function": {"name": "get_weather"}},+ "max_tokens": 128,+ },+ config=config,+ )+ calls = response["choices"][0]["message"].get("tool_calls") or []+ named = [c for c in calls if (c.get("function") or {}).get("name") == "get_weather"]+ if not named:+ matrix.record(+ 5,+ f"[{label}] tool calling",+ BLOCKER,+ "fail",+ "tool_choice forced get_weather but it was not returned; "+ f"tool_calls={[(c.get('function') or {}).get('name') for c in calls]}",+ )+ else:+ raw_arguments = (named[0].get("function") or {}).get("arguments")+ try:+ arguments = json.loads(raw_arguments or "")+ except (ValueError, TypeError):+ matrix.record(+ 5,+ f"[{label}] tool calling",+ BLOCKER,+ "fail",+ "get_weather was called but its arguments are not valid JSON; "+ f"type={type(raw_arguments).__name__}",+ )+ else:+ city = (arguments or {}).get("city") if isinstance(arguments, dict) else None+ if isinstance(city, str) and city.strip():+ matrix.record(+ 5,+ f"[{label}] tool calling",+ BLOCKER,+ "pass",+ f"get_weather(city={city.strip()[:40]!r}) with valid JSON arguments",+ )+ else:+ matrix.record(+ 5,+ f"[{label}] tool calling",+ BLOCKER,+ "fail",+ f"get_weather returned JSON without a usable city: {arguments!r}",+ )+ except Exception as exc:+ matrix.record(5, f"[{label}] tool calling", BLOCKER, "fail", _describe_exception(exc))++ # 6. Embeddings.+ try:+ response = await embeddings(+ {"model": f"azure_ai:{embedding_model}", "input": ["live validation probe"]},+ config=config,+ )+ vector = response["data"][0]["embedding"]+ # Same reasoning as the chat check: an empty vector recorded+ # "0 dimensions" and passed. The finiteness test matters too — a+ # vector of NaN has the right length and is useless for retrieval,+ # and OpenSearch rejects it at index time rather than here.+ problem = _unusable_embedding(vector)+ matrix.record(+ 6,+ f"[{label}] embeddings",+ BLOCKER,+ "fail" if problem else "pass",+ problem or f"{len(vector)} dimensions",+ )+ except Exception as exc:+ matrix.record(6, f"[{label}] embeddings", BLOCKER, "fail", _describe_exception(exc))++ # 7. A deployment that does not exist must fail as a *not found*, through+ # the gateway's own error contract. Passing on any exception would let a+ # DNS failure or a 500 stand in for the rejection this proves.+ from services.llm_gateway import LlmGatewayError++ try:+ await chat_completions(+ {+ "model": "azure_ai:openrag-live-validation-no-such-deployment",+ "messages": [{"role": "user", "content": "hello"}],+ "max_tokens": 8,+ },+ config=config,+ )+ matrix.record(+ 7,+ f"[{label}] unknown deployment is rejected",+ BLOCKER,+ "fail",+ "the call succeeded against a deployment that should not exist",+ )+ except LlmGatewayError as exc:+ # `detail` is the internal text; `message` is what a caller is shown.+ # Upstream 404s are deliberately mapped to 502 by the gateway, so the+ # status is not the discriminator — the cause is.+ detail = f"{exc.detail}"+ if _looks_like_connectivity_failure(detail):+ matrix.record(+ 7,+ f"[{label}] unknown deployment is rejected",+ BLOCKER,+ "fail",+ f"could not reach the endpoint, so nothing was proved: {detail[:160]}",+ )+ elif _looks_like_not_found(detail):+ matrix.record(+ 7,+ f"[{label}] unknown deployment is rejected",+ BLOCKER,+ "pass",+ f"HTTP {exc.status_code} via the gateway; sanitized message={exc.message[:90]!r}",+ )+ else:+ matrix.record(+ 7,+ f"[{label}] unknown deployment is rejected",+ BLOCKER,+ "fail",+ "rejected, but not as a missing deployment — the error does not "+ f"identify the cause: {detail[:160]}",+ )+ except Exception as exc:+ matrix.record(+ 7,+ f"[{label}] unknown deployment is rejected",+ BLOCKER,+ "fail",+ "the failure did not come through the gateway's error contract: "+ + _describe_exception(exc)[:160],+ )+++# --------------------------------------------------------------------------+# Standalone checks+# --------------------------------------------------------------------------+++async def _listing_response(api_base: str, api_key: str) -> tuple[int | None, str]:+ """Raw status and body of the deployment listing, for the record.++ The health check collapses the response into an exception type; deciding+ the 403 question needs the response itself.+ """+ import httpx++ from enhancements.providers.azure import foundry++ try:+ async with httpx.AsyncClient(timeout=30.0) as client:+ response = await client.get(+ foundry.models_url(api_base), headers=foundry.health_headers({"api_key": api_key})+ )+ return response.status_code, (response.text or "")[:300]+ except Exception as exc:+ return None, _describe_exception(exc)+++async def _probe_credential(+ matrix: Matrix,+ *,+ number: int,+ name: str,+ api_base: str,+ api_key: str,+ scenario: str,+) -> None:+ """One credential scenario, recorded as evidence rather than a verdict.++ `scenario` is "valid", "invalid" or "listing_denied". Each has a different+ expected outcome, and a 403 is never silently treated as success — see+ "The 403 question" in the module docstring.+ """+ from enhancements.providers.azure import foundry++ status, body = await _listing_response(api_base, api_key)++ outcome = "accepted"+ detail = ""+ try:+ await foundry.lightweight_health_check({"api_base": api_base, "api_key": api_key})+ except PermissionError as exc:+ outcome, detail = "PermissionError", str(exc)+ except Exception as exc:+ outcome, detail = type(exc).__name__, str(exc)++ # What the product would do: settings/endpoints.py wraps provider+ # validation in `except Exception` and returns 400, so anything raised+ # here — PermissionError included — blocks the save.+ would_save = outcome == "accepted"+ evidence = (+ f"listing HTTP {status if status is not None else 'unreachable'}; "+ f"health check -> {outcome}; provider saveable={would_save}"+ )++ if status is None and _looks_like_connectivity_failure(body):+ matrix.record(number, name, BLOCKER, "fail", f"endpoint unreachable: {body[:140]}")+ return++ if scenario == "valid":+ if would_save:+ matrix.record(number, name, BLOCKER, "pass", evidence)+ elif status == 403:+ matrix.record(+ number,+ name,+ BLOCKER,+ "fail",+ "INCONCLUSIVE, and blocking by design: the credential was accepted "+ "for listing purposes but denied. `lightweight_health_check` raises "+ "PermissionError, and settings/endpoints.py turns that into HTTP 400, "+ "so the provider cannot be saved. Decide whether listing permission "+ "is required or whether this specific denial should be non-blocking, "+ f"using the response below. {evidence}. body={body[:160]!r}",+ )+ else:+ matrix.record(number, name, BLOCKER, "fail", f"{evidence}. body={body[:160]!r}")+ return++ if scenario == "invalid":+ if would_save:+ matrix.record(+ number, name, BLOCKER, "fail", f"an invalid key passed the check. {evidence}"+ )+ return+ combined = f"{detail} {body}"+ if _looks_like_connectivity_failure(combined):+ matrix.record(+ number,+ name,+ BLOCKER,+ "fail",+ f"rejected for a transport reason, which proves nothing: {evidence}",+ )+ elif status is not None and 500 <= status < 600:+ matrix.record(+ number,+ name,+ BLOCKER,+ "fail",+ f"upstream error rather than a credential rejection: {evidence}",+ )+ elif _looks_like_credential_failure(combined) or status == 401:+ matrix.record(number, name, BLOCKER, "pass", evidence)+ else:+ matrix.record(+ number,+ name,+ BLOCKER,+ "fail",+ "rejected, but not identifiably as a bad credential — so a bad key "+ f"is indistinguishable from other failures. {evidence}. "+ f"body={body[:160]!r}",+ )+ return++ # listing_denied: pure evidence gathering, never a pass/fail verdict.+ matrix.record(+ number,+ name,+ ADVISORY,+ "pass",+ "EVIDENCE for the 403 decision — compare with the invalid-key response "+ f"above; they must be reliably distinguishable. {evidence}. "+ f"body={body[:200]!r}",+ )+++def _sample_png(size: int = 128) -> bytes:+ """A small PNG with enough structure for a vision model to describe.++ Written here rather than committed as a fixture or a base64 blob so it+ stays readable, and generated at a realistic size: a 1x1 pixel is a valid+ PNG but some vision deployments reject images below a minimum dimension,+ which would show up as an integration failure that is really a fixture+ problem.++ Four coloured quadrants, a dark border, and a white disc in the middle —+ unambiguous enough that a description can be sanity-checked by eye.+ """+ quadrants = ((220, 60, 60), (60, 120, 220), (240, 200, 60), (60, 180, 110))+ centre = size / 2+ radius = size / 5+ rows = []+ for y in range(size):+ row = bytearray()+ for x in range(size):+ if x < 2 or y < 2 or x >= size - 2 or y >= size - 2:+ pixel = (30, 30, 30)+ elif (x - centre) ** 2 + (y - centre) ** 2 <= radius**2:+ pixel = (250, 250, 250)+ else:+ pixel = quadrants[(1 if x >= centre else 0) + (2 if y >= centre else 0)]+ row.extend(pixel)+ rows.append(bytes(row))++ def _chunk(tag: bytes, data: bytes) -> bytes:+ body = tag + data+ return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body))++ raw = b"".join(b"\x00" + row for row in rows)+ return (+ b"\x89PNG\r\n\x1a\n"+ + _chunk(b"IHDR", struct.pack(">IIBBBBB", size, size, 8, 2, 0, 0, 0))+ + _chunk(b"IDAT", zlib.compress(raw, 9))+ + _chunk(b"IEND", b"")+ )+++async def _check_docling_vlm(matrix: Matrix, api_base: str, settings: dict[str, str]) -> None:+ """Build the block OpenRAG hands docling-serve, then issue that request.++ docling-serve is not involved: the point is that the URL, headers and+ params OpenRAG produces are accepted by Foundry as given.+ """+ import httpx++ from services.docling_service import DoclingService++ deployment = settings.get("vlm_deployment")+ if not deployment:+ matrix.record(+ 10,+ "Docling VLM picture description",+ BLOCKER,+ "skipped",+ "set AZURE_AI_VLM_DEPLOYMENT to run this — it is a blocker for the flip",+ )+ return++ knowledge = SimpleNamespace(+ ocr=False,+ ocr_languages=[],+ table_structure=False,+ picture_descriptions=True,+ vlm_enabled=True,+ vlm_provider="azure_ai",+ vlm_model=f"azure_ai:{deployment}",+ vlm_prompt="Describe this image in one short sentence.",+ vlm_max_tokens=64,+ vlm_watsonx_api_version="2024-01-01",+ )+ config = SimpleNamespace(+ knowledge=knowledge,+ providers=SimpleNamespace(+ credential_values=lambda _key, **_kw: {+ "api_base": api_base,+ "api_key": settings["api_key"],+ }+ ),+ )++ import services.docling_service as docling_module++ original = docling_module.get_openrag_config+ docling_module.get_openrag_config = lambda: config+ try:+ options = await DoclingService()._build_docling_options_async()+ except Exception as exc:+ matrix.record(+ 10,+ "Docling VLM picture description",+ BLOCKER,+ "fail",+ f"could not build the request: {_describe_exception(exc)}",+ )+ return+ finally:+ docling_module.get_openrag_config = original++ api = options.get("picture_description_api")+ if not api:+ matrix.record(+ 10, "Docling VLM picture description", BLOCKER, "fail", "no request was built"+ )+ return++ # The model name must have shed its provider tag before it goes upstream.+ if api["params"].get("model") != deployment:+ matrix.record(+ 10,+ "Docling VLM picture description",+ BLOCKER,+ "fail",+ f"model sent as {api['params'].get('model')!r}, expected {deployment!r}",+ )+ return++ image = base64.b64encode(_sample_png()).decode()+ payload = {+ **api["params"],+ "messages": [+ {+ "role": "user",+ "content": [+ {"type": "text", "text": api["prompt"]},+ {+ "type": "image_url",+ "image_url": {"url": f"data:image/png;base64,{image}"},+ },+ ],+ }+ ],+ }+ try:+ async with httpx.AsyncClient(timeout=60.0) as client:+ response = await client.post(api["url"], headers=api["headers"], json=payload)+ if response.status_code == 200:+ matrix.record(+ 10,+ "Docling VLM picture description",+ BLOCKER,+ "pass",+ "Foundry accepted the request OpenRAG builds for docling-serve",+ )+ else:+ matrix.record(+ 10,+ "Docling VLM picture description",+ BLOCKER,+ "fail",+ f"HTTP {response.status_code}: {response.text[:200]}",+ )+ except Exception as exc:+ matrix.record(+ 10, "Docling VLM picture description", BLOCKER, "fail", _describe_exception(exc)+ )+++async def _check_legacy_header_behaviour(+ matrix: Matrix, legacy_base: str | None, api_key: str+) -> None:+ """The question open since the routing design: does `/models` take Bearer?++ `foundry.health_headers` sends both styles because litellm itself is+ inconsistent — its `azure_ai` chat path sends `api-key` while its embedding+ path sends `Bearer` to the same endpoint. This measures what the endpoint+ actually accepts, one header at a time.++ Advisory: the shipped code sends both, so it works either way. The answer+ decides whether that belt-and-braces can be simplified later.+ """+ import httpx++ if not legacy_base:+ matrix.record(+ 12,+ "legacy /models header behaviour",+ ADVISORY,+ "skipped",+ "set AZURE_AI_LEGACY_API_BASE to measure this",+ )+ return++ from enhancements.providers.azure import foundry++ url = foundry.models_url(legacy_base)+ results = []+ for style, headers in (+ ("api-key", {"api-key": api_key}),+ ("Bearer", {"Authorization": f"Bearer {api_key}"}),+ ):+ try:+ async with httpx.AsyncClient(timeout=30.0) as client:+ response = await client.get(url, headers=headers)+ results.append(f"{style}={response.status_code}")+ except Exception as exc:+ results.append(f"{style}=error({type(exc).__name__})")+ matrix.record(12, "legacy /models header behaviour", ADVISORY, "pass", ", ".join(results))+++def _check_secret_leak(matrix: Matrix, secrets: tuple[str, ...]) -> None:+ """Nothing this run produced may contain the key.++ Scans captured root logging, stdout, stderr, OpenRAG's structlog output+ and every recorded exception. On a hit the material is deliberately **not**+ printed — only the fact, how many fragments matched, and the first source+ kind — because printing it is the leak all over again, into a terminal and+ very likely into CI output.+ """+ if not secrets:+ matrix.record(+ 11, "no credential in errors or logs", BLOCKER, "fail", "no credential to scan for"+ )+ return++ scanned = sum(len(fragment) for fragment in matrix.captured_text if fragment)+ leaked = 0+ for index, secret in enumerate(secrets):+ hits = sum(1 for fragment in matrix.captured_text if fragment and secret in fragment)+ if hits:+ leaked += 1+ matrix.record(+ 11,+ "no credential in errors or logs",+ BLOCKER,+ "fail",+ # Neither the secret nor the surrounding material is named:+ # printing either is the leak again. The index identifies+ # which credential without disclosing it.+ f"credential #{index + 1} of {len(secrets)} appeared in {hits} captured "+ "fragment(s). Nothing is reproduced here — re-run with scratch "+ "credentials and inspect locally to find the source.",+ )+ if leaked:+ return+ matrix.record(+ 11,+ "no credential in errors or logs",+ BLOCKER,+ "pass",+ f"scanned {scanned} characters of captured logging, stdout, stderr and errors "+ f"for {len(secrets)} credential(s)",+ )+++def _check_cost_attribution(matrix: Matrix, api_base: str, chat_model: str) -> None:+ """Advisory by agreement: OpenRAG does not consume cost data today."""+ from enhancements.providers.azure import foundry++ route = foundry.litellm_route({"api_base": api_base})+ if route == foundry.LITELLM_PROVIDER:+ matrix.record(+ 13,+ "cost attribution",+ ADVISORY,+ "pass",+ "native route keeps litellm's azure_ai price rows",+ )+ return+ try:+ import litellm++ mapped = f"azure_ai/{chat_model}" in litellm.model_cost+ except Exception:+ mapped = False+ matrix.record(+ 13,+ "cost attribution",+ ADVISORY,+ "pass",+ (+ f"transport route {route!r} has no price row; "+ + (+ f"base_model='azure_ai/{chat_model}' would restore it"+ if mapped+ else "the deployment name is not in any price table either"+ )+ + " — not consumed by OpenRAG today"+ ),+ )+++# --------------------------------------------------------------------------+# Log capture+# --------------------------------------------------------------------------+++class _Recorder:+ """Collects every structlog call so check 11 can scan it."""++ def __init__(self, sink: list[str]) -> None:+ self._sink = sink++ def __getattr__(self, _name):+ def _log(message="", **fields):+ self._sink.append(f"{message} {fields}")++ return _log+++class _Tee:+ """A stream that passes writes through and keeps a copy.++ Progress stays visible while everything printed — including anything a+ library writes directly rather than through logging — is available to the+ secret scan. What reaches the real stream is redacted; what reaches the+ sink is not, because detection needs the original.+ """++ def __init__(self, stream, sink: list[str]) -> None:+ self._stream = stream+ self._sink = sink++ def write(self, text: str) -> int:+ # Raw into the capture so check 11 can still see a leak...+ self._sink.append(text)+ # ...redacted on the way to the terminal. Forwarding the raw text and+ # relying on the final report to redact would be too late: the secret+ # is already on screen, and in CI already in a retained build log.+ return self._stream.write(_redact(text))++ def flush(self) -> None:+ self._stream.flush()++ def isatty(self) -> bool:+ return False++ def __getattr__(self, name):+ return getattr(self._stream, name)+++class _SinkHandler(logging.Handler):+ """Root logging handler: catches litellm, httpx and anything else."""++ def __init__(self, sink: list[str]) -> None:+ super().__init__(level=logging.DEBUG)+ self._sink = sink++ def emit(self, record: logging.LogRecord) -> None:+ try:+ self._sink.append(self.format(record))+ except Exception:+ # A handler that raises would be worse than one that misses a+ # line, but a miss weakens the scan — so record that it happened.+ self._sink.append(f"<unformattable log record from {record.name}>")+++def _console_handlers() -> list[logging.StreamHandler]:+ """Every StreamHandler currently attached anywhere in the logging tree.++ Walked rather than taken from the root logger alone: the handlers that+ matter here are attached to named library loggers, not to root.+ """+ seen: list[logging.StreamHandler] = []+ loggers: list[Any] = [logging.getLogger()]+ loggers.extend(+ obj for obj in logging.root.manager.loggerDict.values() if isinstance(obj, logging.Logger)+ )+ for logger_obj in loggers:+ for candidate in getattr(logger_obj, "handlers", []):+ if isinstance(candidate, logging.StreamHandler) and candidate not in seen:+ seen.append(candidate)+ return seen+++def _capturing(sink: list[str]):+ """Capture root logging, stdout, stderr and OpenRAG's structlog loggers.++ Everything here feeds check 11. Anything outside it — a library writing to+ a file descriptor directly, or a subprocess — is not covered, which is why+ check 11 reports how much it scanned rather than claiming completeness.+ """+ from enhancements.providers.azure import foundry+ from services import docling_service, llm_gateway++ # Annotated `Any` so the swap below type-checks. These are module objects,+ # which mypy otherwise infers as `Module | Any` and then rejects assigning+ # to; `setattr` silences that but trips ruff's B010, so the type is widened+ # once here instead of suppressing a linter at each line.+ modules: tuple[Any, ...] = (foundry, llm_gateway, docling_service)+ original_loggers = [module.logger for module in modules]+ for module in modules:+ module.logger = _Recorder(sink)++ root = logging.getLogger()+ handler = _SinkHandler(sink)+ handler.setFormatter(logging.Formatter("%(name)s %(levelname)s %(message)s"))+ previous_level = root.level+ root.addHandler(handler)+ root.setLevel(logging.DEBUG)++ real_stdout, real_stderr = sys.stdout, sys.stderr+ tee_stdout = _Tee(real_stdout, sink)+ tee_stderr = _Tee(real_stderr, sink)+ sys.stdout = tee_stdout+ sys.stderr = tee_stderr++ # Swapping `sys.stderr` is not enough on its own. A StreamHandler attached+ # before this block holds the stream *object* it was given, so it keeps+ # writing to the real terminal no matter what `sys.stderr` now names —+ # litellm attaches three such handlers at import, and litellm is the+ # library most likely to log a request detail carrying a credential. Point+ # them at the redacting stream for the duration and put them back after.+ redirected: list[tuple[logging.StreamHandler, Any]] = []+ for existing in _console_handlers():+ if existing.stream is real_stdout:+ redirected.append((existing, existing.stream))+ existing.setStream(tee_stdout)+ elif existing.stream is real_stderr:+ redirected.append((existing, existing.stream))+ existing.setStream(tee_stderr)++ try:+ yield+ finally:+ for existing, original_stream in redirected:+ existing.setStream(original_stream)+ sys.stdout, sys.stderr = real_stdout, real_stderr+ root.removeHandler(handler)+ root.setLevel(previous_level)+ for module, logger in zip(modules, original_loggers, strict=True):+ module.logger = logger+++# --------------------------------------------------------------------------+# Entry point+# --------------------------------------------------------------------------+++def _settings() -> dict[str, str]:+ required = {+ "api_base": "AZURE_AI_API_BASE",+ "api_key": "AZURE_AI_API_KEY",+ "chat_deployment": "AZURE_AI_CHAT_DEPLOYMENT",+ "embedding_deployment": "AZURE_AI_EMBEDDING_DEPLOYMENT",+ }+ optional = {+ "project_api_base": "AZURE_AI_PROJECT_API_BASE",+ "legacy_api_base": "AZURE_AI_LEGACY_API_BASE",+ "vlm_deployment": "AZURE_AI_VLM_DEPLOYMENT",+ "listing_denied_api_key": "AZURE_AI_LISTING_DENIED_API_KEY",+ }+ values = {+ name: os.environ.get(env, "").strip() for name, env in {**required, **optional}.items()+ }+ missing = [env for name, env in required.items() if not values[name]]+ if missing:+ print("Missing required environment variables:", ", ".join(missing), file=sys.stderr)+ print(__doc__ or "", file=sys.stderr)+ raise SystemExit(2)+ return values+++async def _main() -> int:+ settings = _settings()+ # Before anything can print: every real credential the run was given,+ # so `_redact` can keep all of them off the terminal.+ _set_real_secrets(settings["api_key"], settings.get("listing_denied_api_key", ""))+ matrix = Matrix()++ # The profile each endpoint variable is *supposed* to name. Checked rather+ # than assumed: the three are set by hand, and pasting the wrong surface+ # into one of them would otherwise be reported as a pass for the surface+ # it was not.+ endpoints: list[tuple[str, str, int, str]] = [+ ("resource v1", settings["api_base"], 8, "resource_openai_v1")+ ]+ if settings.get("project_api_base"):+ endpoints.append(("project v1", settings["project_api_base"], 9, "project_openai_v1"))+ if settings.get("legacy_api_base"):+ endpoints.append(("legacy /models", settings["legacy_api_base"], 14, "legacy_models"))++ with _capturing(matrix.captured_text):+ await _run_all_checks(matrix, settings, endpoints)++ _print_report(matrix)+ return _exit_code(matrix)+++async def _run_all_checks(matrix: Matrix, settings: dict[str, str], endpoints) -> None:+ for label, api_base, number, expected_profile in endpoints:+ print(f"\n--- {label}: {_safe_endpoint(api_base)} ---", flush=True)+ await _run_endpoint_matrix(+ matrix,+ label,+ api_base,+ settings,+ base_number=number,+ expected_profile=expected_profile,+ )++ if not settings.get("project_api_base"):+ matrix.record(+ 9,+ "project /openai/v1",+ ADVISORY,+ "skipped",+ "set AZURE_AI_PROJECT_API_BASE if this resource has a project endpoint",+ )++ await _probe_credential(+ matrix,+ number=2,+ name="invalid API key is rejected at save time",+ api_base=settings["api_base"],+ api_key="openrag-live-validation-invalid-key",+ scenario="invalid",+ )+ if settings.get("listing_denied_api_key"):+ await _probe_credential(+ matrix,+ number=2,+ name="listing-denied credential: what Foundry actually returns",+ api_base=settings["api_base"],+ api_key=settings["listing_denied_api_key"],+ scenario="listing_denied",+ )+ else:+ matrix.record(+ 2,+ "listing-denied credential: what Foundry actually returns",+ ADVISORY,+ "skipped",+ "set AZURE_AI_LISTING_DENIED_API_KEY to settle the 403 question; "+ "without it, a 403 on a valid key stays inconclusive and blocking",+ )+ await _check_docling_vlm(matrix, settings["api_base"], settings)+ await _check_legacy_header_behaviour(+ matrix, settings.get("legacy_api_base"), settings["api_key"]+ )+ _check_cost_attribution(matrix, settings["api_base"], settings["chat_deployment"])+ # Last: it scans everything the run produced.+ _check_secret_leak(matrix, _REAL_SECRETS)+++def _print_report(matrix: Matrix) -> None:+ print("\n" + "=" * 78)+ print("Azure AI Foundry live validation")+ print("=" * 78)+ symbols = {"pass": "PASS", "fail": "FAIL", "skipped": "SKIP"}+ for check in sorted(matrix.checks, key=lambda c: (c.number, c.name)):+ tag = "blocker " if check.severity == BLOCKER else "advisory"+ print(f" [{symbols[check.status]}] ({tag}) {check.number:>2}. {check.name}")+ if check.detail:+ print(f" {_redact(check.detail)}")+ print("-" * 78)+++def _exit_code(matrix: Matrix) -> int:+ blocking_failures = [c for c in matrix.checks if c.severity == BLOCKER and c.status == "fail"]+ blocking_skips = [c for c in matrix.checks if c.severity == BLOCKER and c.status == "skipped"]+ if blocking_failures:+ print(f"{len(blocking_failures)} blocker(s) failed. Do not enable the provider.")+ return 1+ if blocking_skips:+ print(+ f"{len(blocking_skips)} blocker(s) were not exercised. "+ "The matrix is incomplete; do not enable the provider yet."+ )+ return 1+ print("All blockers passed. The visibility flip can proceed.")+ return 0+++if __name__ == "__main__":+ parser = argparse.ArgumentParser(description=__doc__)+ parser.parse_args()+ try:+ raise SystemExit(asyncio.run(_main()))+ except KeyboardInterrupt:+ raise SystemExit(130) from None+ except SystemExit:+ raise+ except Exception: # pragma: no cover - operator-facing script+ traceback.print_exc()+ raise SystemExit(1) from Nonediff --git a/tests/unit/test_validate_azure_foundry_live.py b/tests/unit/test_validate_azure_foundry_live.pynew file mode 100644index 000000000..766d2ddad--- /dev/null+++ b/tests/unit/test_validate_azure_foundry_live.py@@ -0,0 +1,418 @@+"""The live-validation harness's own safety properties.++`scripts/validate_azure_foundry_live.py` is run by hand against a real Foundry+resource, so its failure modes are not otherwise exercised by CI. Two of them+are worth pinning: it must never put a real credential on a terminal, and the+checks that pass by observing a rejection must not pass on an unrelated one.++The leak tests matter because detection alone is not enough. Check 11 finds a+leak *after* the run, by which point anything written to stdout or stderr is+already on screen and, in CI, already in a retained build log.+"""++from __future__ import annotations++import importlib.util+import io+import sys+from pathlib import Path++import pytest++ROOT = Path(__file__).resolve().parent.parent.parent+SCRIPT = ROOT / "scripts" / "validate_azure_foundry_live.py"++REAL_KEY = "primary-credential-must-never-be-printed"+LISTING_DENIED_KEY = "listing-denied-credential-also-secret"+++def harness():+ """The script as an importable module, with its secret list reset."""+ spec = importlib.util.spec_from_file_location("azure_foundry_live_validation", SCRIPT)+ assert spec and spec.loader+ module = importlib.util.module_from_spec(spec)+ sys.modules[spec.name] = module+ spec.loader.exec_module(module)+ try:+ yield module+ finally:+ module._set_real_secrets()+ sys.modules.pop(spec.name, None)+++class TestRedaction:+ def test_every_supplied_credential_is_redacted(self, harness) -> None:+ harness._set_real_secrets(REAL_KEY, LISTING_DENIED_KEY)+ text = f"key={REAL_KEY} other={LISTING_DENIED_KEY}"++ cleaned = harness._redact(text)++ assert REAL_KEY not in cleaned+ assert LISTING_DENIED_KEY not in cleaned+ assert cleaned.count("<redacted>") == 2++ def test_a_short_value_is_not_treated_as_a_secret(self, harness) -> None:+ """Blanking a two-character value would destroy the output instead."""+ harness._set_real_secrets("ab")+ assert harness._redact("a table of absolute values") == "a table of absolute values"++ def test_blank_credentials_are_ignored(self, harness) -> None:+ harness._set_real_secrets("", None or "")+ assert harness._REAL_SECRETS == ()+++class TestTeeDoesNotLeakToTheTerminal:+ """The stream forwards redacted text and keeps the original for detection."""++ @pytest.mark.parametrize("secret", [REAL_KEY, LISTING_DENIED_KEY])+ def test_the_real_stream_never_receives_the_secret(self, harness, secret) -> None:+ harness._set_real_secrets(REAL_KEY, LISTING_DENIED_KEY)+ terminal = io.StringIO()+ sink: list[str] = []+ tee = harness._Tee(terminal, sink)++ tee.write(f"litellm debug: authorization=Bearer {secret}\n")++ assert secret not in terminal.getvalue()+ assert "<redacted>" in terminal.getvalue()+ # ...but the capture keeps the original, or check 11 could not detect it.+ assert any(secret in fragment for fragment in sink)+++class TestCheckElevenDetectsEitherCredential:+ @pytest.mark.parametrize(+ "leaked, index",+ [(REAL_KEY, 1), (LISTING_DENIED_KEY, 2)],+ )+ def test_a_forced_leak_of_either_credential_fails_the_check(+ self, harness, leaked, index+ ) -> None:+ matrix = harness.Matrix()+ matrix.capture(f"some library wrote {leaked} to stderr")++ harness._check_secret_leak(matrix, (REAL_KEY, LISTING_DENIED_KEY))++ check = matrix.checks[-1]+ assert check.status == "fail"+ assert check.severity == harness.BLOCKER+ assert f"credential #{index}" in check.detail++ def test_the_failure_detail_never_contains_the_secret(self, harness) -> None:+ """Reporting a leak must not be the leak."""+ matrix = harness.Matrix()+ matrix.capture(f"leaked {REAL_KEY} here")++ harness._check_secret_leak(matrix, (REAL_KEY,))++ assert REAL_KEY not in matrix.checks[-1].detail++ def test_a_clean_run_passes_and_says_how_much_it_scanned(self, harness) -> None:+ matrix = harness.Matrix()+ matrix.capture("nothing sensitive at all")++ harness._check_secret_leak(matrix, (REAL_KEY, LISTING_DENIED_KEY))++ check = matrix.checks[-1]+ assert check.status == "pass"+ assert "2 credential(s)" in check.detail++ def test_no_credentials_to_scan_is_a_failure_not_a_pass(self, harness) -> None:+ """An empty secret list would otherwise pass vacuously."""+ matrix = harness.Matrix()++ harness._check_secret_leak(matrix, ())++ assert matrix.checks[-1].status == "fail"+++class TestTheReportCannotPrintASecret:+ @pytest.mark.parametrize("secret", [REAL_KEY, LISTING_DENIED_KEY])+ def test_a_secret_echoed_into_a_check_detail_is_redacted_on_print(+ self, harness, secret, capsys+ ) -> None:+ """A provider can echo the credential back in an error body.++ That body ends up in a check's detail, so the report has to redact on+ the way out even though check 11 has already flagged it.+ """+ harness._set_real_secrets(REAL_KEY, LISTING_DENIED_KEY)+ matrix = harness.Matrix()+ matrix.record(+ 2,+ "invalid API key is rejected at save time",+ harness.BLOCKER,+ "fail",+ f'upstream echoed the credential: body={{"key": "{secret}"}}',+ )++ harness._print_report(matrix)++ assert secret not in capsys.readouterr().out+++class TestNotFoundClassification:+ """A bare 404 is not evidence: those digits appear in ids and counters."""++ @pytest.mark.parametrize(+ "text",+ [+ "DeploymentNotFound",+ "The API deployment for this resource does not exist",+ "model_not_found",+ "HTTP 404",+ "HTTP/1.1 404",+ "status_code: 404",+ "status=404",+ "404 Not Found",+ ],+ )+ def test_a_real_not_found_is_recognised(self, harness, text) -> None:+ assert harness._looks_like_not_found(text)++ @pytest.mark.parametrize(+ "text",+ [+ "request id 8a404b2c",+ "used 404 prompt tokens",+ "completed in 404ms",+ "x-ms-request-id: 404aa1",+ "rate limit exceeded",+ ],+ )+ def test_an_incidental_404_is_not_mistaken_for_one(self, harness, text) -> None:+ assert not harness._looks_like_not_found(text)+++class TestFailureCategories:+ """Checks that pass on a rejection must not pass on an unrelated failure."""++ @pytest.mark.parametrize(+ "text",+ [+ "ConnectError: nodename nor servname provided, or not known",+ "Cannot connect to host contoso.services.ai.azure.com:443",+ "ReadTimeout",+ "certificate verify failed",+ ],+ )+ def test_transport_failures_are_identified_as_such(self, harness, text) -> None:+ assert harness._looks_like_connectivity_failure(text)+ assert not harness._looks_like_credential_failure(text)++ @pytest.mark.parametrize(+ "text",+ [+ "Access denied due to invalid subscription key",+ "401 Unauthorized",+ "Incorrect API key provided",+ "HTTP 401",+ "status_code: 401",+ ],+ )+ def test_credential_rejections_are_identified_as_such(self, harness, text) -> None:+ assert harness._looks_like_credential_failure(text)+ assert not harness._looks_like_connectivity_failure(text)++ @pytest.mark.parametrize(+ "text",+ [+ "request id 8a401b2c",+ "completed in 401 ms",+ "prompt_tokens: 401",+ ],+ )+ def test_an_incidental_401_is_not_mistaken_for_a_rejection(self, harness, text) -> None:+ """Same reasoning as the 404 case above.++ Check 2 runs this classifier over 300 characters of response body, so+ a request id carrying those three digits would otherwise let a+ non-credential rejection pass as a credential one.+ """+ assert not harness._looks_like_credential_failure(text)+++class TestTheRoutingCheckCanFail:+ """A blocker that always passes is not a check.++ Both paths below return before any network call, which is what makes them+ testable without a live resource.+ """++ @pytest.fixture+ def settings(self):+ return {+ "api_key": REAL_KEY,+ "chat_deployment": "chat-dep",+ "embedding_deployment": "embed-dep",+ }++ @pytest.mark.asyncio+ async def test_the_wrong_surface_is_reported_rather_than_passed(+ self, harness, settings, monkeypatch+ ) -> None:+ """A legacy `/models` URL in AZURE_AI_API_BASE used to satisfy the+ `/openai/v1` blocker, which is the one thing the matrix exists to+ prove."""+ from enhancements.providers.azure import foundry++ monkeypatch.setattr(foundry, "endpoint_profile", lambda _b: "legacy_models")+ monkeypatch.setattr(foundry, "litellm_route", lambda _s: "azure_ai")++ matrix = harness.Matrix()+ await harness._run_endpoint_matrix(+ matrix,+ "resource v1",+ "https://contoso.services.ai.azure.com/models",+ settings,+ base_number=8,+ expected_profile="resource_openai_v1",+ )++ assert [c.status for c in matrix.checks] == ["fail"]+ assert "expected resource_openai_v1" in matrix.checks[0].detail++ @pytest.mark.asyncio+ async def test_an_unclassifiable_endpoint_does_not_abort_the_run(+ self, harness, settings, monkeypatch+ ) -> None:+ """`litellm_route` refuses what it cannot classify, by design.++ Uncaught, that exception escaped `_run_all_checks` and the report was+ never printed — a traceback instead of the matrix, which is the least+ useful thing a release gate can hand an operator.+ """+ from enhancements.providers.azure import foundry++ def _refuse(_stored):+ raise foundry.UnsupportedEndpointError(foundry.UNSUPPORTED_ENDPOINT_MESSAGE)++ monkeypatch.setattr(foundry, "endpoint_profile", lambda _b: "unknown")+ monkeypatch.setattr(foundry, "litellm_route", _refuse)++ matrix = harness.Matrix()+ await harness._run_endpoint_matrix(+ matrix,+ "resource v1",+ "https://contoso.services.ai.azure.com/anthropic/v1",+ settings,+ base_number=8,+ expected_profile="resource_openai_v1",+ )++ assert [c.status for c in matrix.checks] == ["fail"]+ assert "UnsupportedEndpointError" in matrix.checks[0].detail+++class TestBlockersRejectUnusableSuccesses:+ """A 200 is not the same as a working deployment.++ Both of these recorded a pass on a successful-but-empty response, which+ made the release matrix green without usable inference — the one outcome+ a blocker here has to rule out.+ """++ @pytest.mark.parametrize("text", ["", " ", "\n"])+ def test_an_empty_completion_is_not_a_pass(self, harness, text) -> None:+ assert harness._unusable_chat_reply(text)++ def test_a_real_completion_passes(self, harness) -> None:+ assert harness._unusable_chat_reply("OK") == ""++ @pytest.mark.parametrize(+ "vector",+ [+ [],+ None,+ "not-a-vector",+ [0.1, float("nan")],+ [float("inf"), 0.2],+ [0.1, "0.2"],+ [0.1, True],+ ],+ )+ def test_an_unusable_vector_is_not_a_pass(self, harness, vector) -> None:+ """Length alone is not enough: a vector of NaN has the right shape and+ is useless for retrieval, and OpenSearch rejects it at index time+ rather than here."""+ assert harness._unusable_embedding(vector)++ def test_a_real_vector_passes(self, harness) -> None:+ assert harness._unusable_embedding([0.1, -0.2, 3]) == ""+++class TestTheEndpointIsNeverPrintedRaw:+ """`_redact` only knows the credentials this run was handed.++ A secret carried *inside* the endpoint is invisible to it, and the+ provider module strips or refuses one only after this line is on screen.+ """++ def test_userinfo_and_query_are_removed(self, harness) -> None:+ shown = harness._safe_endpoint(+ "https://operator:[email protected]/openai/v1?sig=abc123"+ )+ assert "s3cr3t" not in shown+ assert "operator" not in shown+ assert "abc123" not in shown+ assert "sig" not in shown+ assert shown.startswith("https://contoso.services.ai.azure.com/openai/v1")+ assert "userinfo" in shown and "query" in shown++ def test_an_ordinary_endpoint_is_left_alone(self, harness) -> None:+ plain = "https://contoso.services.ai.azure.com/openai/v1"+ assert harness._safe_endpoint(plain) == plain++ def test_a_non_url_is_passed_through(self, harness) -> None:+ assert harness._safe_endpoint("not-a-url") == "not-a-url"+++class TestPreboundLogHandlersCannotLeak:+ """Replacing `sys.stderr` does not retarget a handler that already holds it.++ litellm attaches three such handlers at import, and it is the library most+ likely to log a request detail carrying a credential — so without this the+ harness's central promise, that nothing reaches a terminal unredacted,+ does not hold for the most likely source.+ """++ def test_a_handler_bound_before_the_block_is_redirected(self, harness, monkeypatch) -> None:+ import logging++ # The handler must hold the *same object* `sys.stderr` names when+ # `_capturing` is entered — that is the situation litellm creates at+ # import, and the one swapping `sys.stderr` fails to cover. A handler+ # pointed somewhere else entirely is a log file, not a terminal, and+ # is deliberately left alone.+ terminal = io.StringIO()+ monkeypatch.setattr(sys, "stderr", terminal)+ logger = logging.getLogger("test.prebound.handler")+ logger.propagate = False+ handler = logging.StreamHandler(terminal)+ logger.addHandler(handler)+ logger.setLevel(logging.INFO)++ harness._set_real_secrets(REAL_KEY)+ sink: list[str] = []+ try:+ with harness._capturing(sink):+ logger.info("leaking %s now", REAL_KEY)+ finally:+ logger.removeHandler(handler)++ assert REAL_KEY not in terminal.getvalue(), "the secret reached the real stream"+ assert handler.stream is terminal, "the handler's stream was not restored"+ assert any(REAL_KEY in entry for entry in sink), "detection still needs the original"+++class TestImageFixture:+ def test_the_probe_image_is_large_enough_to_be_accepted(self, harness) -> None:+ """A 1x1 PNG is valid but some vision deployments reject it on size,+ which would read as an integration failure rather than a fixture one."""+ import struct++ png = harness._sample_png()++ assert png[:8] == b"\x89PNG\r\n\x1a\n"+ width, height = struct.unpack(">II", png[16:24])+ assert width == height == 128