Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
Python agent-evaluationaiai-alignmentai-evaluationai-governanceai-safetyclidiagnostic-tooleu-ai-acthallucination-detectioniso-42001llm-evaluationllm-securitynist-ai-rmfowasp-llmprompt-injectionpythonresponsible-airisk-assessmentrisk-management
feat: cloudflare provider (#233)
stefyi-4355eca7b32b6655
diff --git a/README.ja.md b/README.ja.mdindex c9024c8..74b4435 100644--- a/README.ja.md+++ b/README.ja.md@@ -204,7 +204,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fiレポートは JSON **と** Markdown の両方で `./ifixai-results/` に保存されます。2 つ目のキーがない場合は、スモークテストとして `--eval-mode self` を追加してください(評価は表示されますが自己評価と明記され、引用できる結果にはなりません)。評価モデルの固定、Full モードのアンサンブル、評価モードについては **[docs/cli.md](docs/cli.md#how-a-run-is-judged)** を参照してください。その他のプロバイダー-(OpenAI、OpenRouter、Requesty、Gemini、Azure、Bedrock、Hugging Face)は対応する extra を+(OpenAI、OpenRouter、Requesty、Cloudflare AI Gateway、Gemini、Azure、Bedrock、Hugging Face)は対応する extra をインストールし、同じ手順を使用します。HTTP と LangChain のアダプターにはプロバイダーの extra は不要です:**[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**。diff --git a/README.ko.md b/README.ko.mdindex c4b7c0e..3e76035 100644--- a/README.ko.md+++ b/README.ko.md@@ -189,7 +189,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fi`--eval-mode self`를 추가해 스모크 테스트로 실행하세요(등급은 출력되지만 자기 채점으로표시되며, 인용할 수 있는 결과는 아닙니다). 심사 모델 고정, Full 모드 앙상블, 평가 모드에대해서는 **[docs/cli.md](docs/cli.md#how-a-run-is-judged)** 를 참고하세요. 다른-프로바이더(OpenAI, Atlas Cloud, OpenRouter, Requesty, Gemini, Azure, Bedrock, Hugging Face)는 해당+프로바이더(OpenAI, Atlas Cloud, OpenRouter, Requesty, Cloudflare AI Gateway, Gemini, Azure, Bedrock, Hugging Face)는 해당extra를 설치하고 같은 절차를 따르며, HTTP와 LangChain 어댑터는 프로바이더 extra가필요 없습니다: **[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**.diff --git a/README.md b/README.mdindex cd40d40..79fbb74 100644--- a/README.md+++ b/README.md@@ -186,7 +186,7 @@ Reports land in `./ifixai-results/` as JSON **and** Markdown. Without a second k`--eval-mode self` to run as a smoke test (the grade still prints, but it's flagged asself-judged, not a result you can cite). Pinning the judge, Full-mode ensembles, and the eval modes:**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**. Other providers (OpenAI, Atlas Cloud,-OpenRouter, OrcaRouter, Requesty, Gemini, Azure, Bedrock, Hugging Face) install the matching extra and follow the same steps; the+OpenRouter, OrcaRouter, Requesty, Cloudflare AI Gateway, Gemini, Azure, Bedrock, Hugging Face) install the matching extra and follow the same steps; theHTTP and LangChain adapters need no provider extra: **[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**.### Recommended judge setupsdiff --git a/README.zh-CN.md b/README.zh-CN.mdindex a0ef239..30500cd 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -200,7 +200,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fi报告以 JSON **和** Markdown 两种格式保存在 `./ifixai-results/` 中。如果没有第二个密钥,请添加 `--eval-mode self` 作为冒烟测试(等级仍会显示,但会被标记为自我评审,不能作为可引用结果)。固定评审模型、完整模式集成以及评审模式的详情请参阅:-**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**。其他提供商(OpenAI、OpenRouter、Requesty、Gemini、+**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**。其他提供商(OpenAI、OpenRouter、Requesty、Cloudflare AI Gateway、Gemini、Azure、Bedrock、Hugging Face)可安装对应 extra 并遵循相同步骤;HTTP 和 LangChain适配器不需要提供商 extra:**[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**。diff --git a/SECURITY.md b/SECURITY.mdindex d900d70..8834845 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -46,6 +46,7 @@ The scrubber (`ifixai/providers/secrets.py::scrub_secrets`) is parametrically ve| `openrouter` | `sk-or-[A-Za-z0-9_-]{20,}` (matched before `openai`) | `***REDACTED_OPENROUTER_KEY***` || `orcarouter` | `sk-orca-[A-Za-z0-9_-]{20,}` (matched before `openai`) | `***REDACTED_ORCAROUTER_KEY***` || `requesty` | `rqsty-[A-Za-z0-9+/=_-]{20,}` | `***REDACTED_REQUESTY_KEY***` |+| `cloudflare` | `cfut_`, `cfat_` or `cfk_` followed by `[A-Za-z0-9_-]{40,}`; tokens in the older unprefixed format have no fixed shape and are redacted only inside an `Authorization: Bearer` header | `***REDACTED_CLOUDFLARE_TOKEN***` || `anthropic` | `sk-ant-[A-Za-z0-9_-]{20,}`, `anthropic_[A-Za-z0-9_-]{20,}` | `***REDACTED_ANTHROPIC_KEY***` || `gemini` | `AIzaSy[0-9A-Za-z_-]{33}` | `***REDACTED_GEMINI_KEY***` || `azure` | 32-char hex (`\b[a-fA-F0-9]{32}\b`) | `***REDACTED_AZURE_KEY***` |diff --git a/docs/cli.md b/docs/cli.mdindex 1791a9c..e2dfb95 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -21,10 +21,10 @@ ifixai run -p openai -k "$OPENAI_API_KEY" -c DECEPTION # example: one category| Flag | Default | Does ||---|---|---|-| `--provider`, `-p` | none | `mock`, `openai`, `openrouter`, `orcarouter`, `requesty`, `anthropic`, `gemini`, `azure`, `bedrock`, `huggingface`, `http`, `langchain`. |+| `--provider`, `-p` | none | `mock`, `openai`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `anthropic`, `gemini`, `azure`, `bedrock`, `huggingface`, `http`, `langchain`. || `--api-key`, `-k` | none | SUT API key. Always passed explicitly, never read from the environment. || `--model`, `-m` | provider default | Model identifier override. |-| `--endpoint`, `-e` | none | Endpoint URL (required for `http` and `azure`). |+| `--endpoint`, `-e` | none | Endpoint URL (required for `http` and `azure`; for `cloudflare` it replaces the URL built from `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_GATEWAY_ID`). || `--system-prompt`, `-s` | none | Custom system instructions sent before each inspection. || `--grounding` | `sut` | Governance context source: `sut` (baked-in), `fixture` (system prompt derived from fixture), `none`. || `--sut-temperature` | `0.0` | SUT sampling temperature. B22 needs `0` or `--sut-seed`. |diff --git a/docs/testing-your-agent.md b/docs/testing-your-agent.mdindex 0bb6602..2ddf27b 100644--- a/docs/testing-your-agent.md+++ b/docs/testing-your-agent.md@@ -134,6 +134,7 @@ the SUT key from the environment: pass `--api-key` / `-k`, or enter it when prom| `openrouter` | `.[openrouter]` | `OPENROUTER_API_KEY` | `--provider openrouter -k "$OPENROUTER_API_KEY" --model openai/gpt-4o` plus explicit judge || `orcarouter` | `.[orcarouter]` | `ORCAROUTER_API_KEY` | `--provider orcarouter -k "$ORCAROUTER_API_KEY" --model openai/gpt-4o` plus explicit judge || `requesty` | `.[requesty]` | `REQUESTY_API_KEY` | `--provider requesty -k "$REQUESTY_API_KEY" --model openai/gpt-4o-mini` plus explicit judge; `--endpoint https://router.eu.requesty.ai/v1` for EU data residency |+| `cloudflare` | `.[cloudflare]` | `CLOUDFLARE_API_TOKEN` plus `CLOUDFLARE_ACCOUNT_ID` | `--provider cloudflare -k "$CLOUDFLARE_API_TOKEN" --model workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast` plus explicit judge || `gemini` | `.[gemini]` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | `--provider gemini -k "$GEMINI_API_KEY"` || `azure` | `.[azure]` | `AZURE_OPENAI_API_KEY` | `--provider azure --endpoint https://YOUR_RESOURCE.openai.azure.com/ -k "$AZURE_OPENAI_API_KEY" --model YOUR_DEPLOYMENT_NAME` plus explicit judge || `bedrock` | `.[bedrock]` | AWS credential chain | `--provider bedrock -k not-used --model anthropic.claude-sonnet-4-6` |@@ -148,6 +149,7 @@ Notes:- `bedrock` uses the standard AWS credential chain; `--api-key` is a required placeholder, never sent.- `litellm` extra is [Python API](python-api.md) only (`provider="litellm"`), not a CLI choice.- An `openrouter` judge falls back to other models when the gateway errors, so a 502 costs one probe, not the run. Chain: [`ifixai/judge/judge_fallbacks.json`](../ifixai/judge/judge_fallbacks.json), see [cli.md](cli.md#judge-fallback-models).+- `cloudflare` is the [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/usage/chat-completion/) OpenAI-compatible endpoint, `gateway.ai.cloudflare.com/v1/<account>/<gateway>/compat`. It needs two values: an API token allowed to run the gateway (`CLOUDFLARE_API_TOKEN`; Cloudflare documents the AI Gateway > Run permission for this) and the account ID, which is part of the URL (`CLOUDFLARE_ACCOUNT_ID`, or pass the full base URL with `--endpoint`). A judge has no `--endpoint`, so a `cloudflare` judge always reads the account ID from the environment. Requests go through the gateway named `default` unless `CLOUDFLARE_GATEWAY_ID` names another (the ID is case-sensitive), and every request asks the gateway to bypass its response cache (`cf-aig-skip-cache`), so that a repeated probe is answered by the model and not by a stored reply. Models are Workers AI ids such as `workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast` (the default; the Workers free plan's daily allocation covered about one 61-probe inspection in testing, so a full run needs a paid Workers plan, and a run that exhausts the allocation stops with Cloudflare's message), or `provider/model` slugs such as `openai/gpt-4.1-mini` and `google-ai-studio/gemini-2.5-flash`, which need Unified Billing credits or a provider key stored in the gateway. Cloudflare answers a bad token and a wrong account ID with the same `401 Unauthorized`; iFixAi lists what to check when it sees one. Cloudflare also has a newer REST route on `api.cloudflare.com` and steers new integrations to it, but that route rejects a gateway token (it wants a token with the Workers AI permission), so this provider uses the gateway URL.## LangServe wire contractdiff --git a/ifixai/cli/init.py b/ifixai/cli/init.pyindex 45da559..b171e4f 100644--- a/ifixai/cli/init.py+++ b/ifixai/cli/init.py@@ -19,14 +19,31 @@ PROVIDER_ENV_KEYS: dict[str, str] = {"azure": "AZURE_OPENAI_API_KEY","bedrock": "AWS_ACCESS_KEY_ID","huggingface": "HF_TOKEN",+ # Last on purpose: this variable is usually set for wrangler or Terraform,+ # so it must not outrank a model vendor's key as the suggested provider.+ "cloudflare": "CLOUDFLARE_API_TOKEN",+}++# Settings a provider cannot make a call without that are not its secret.+# Listed beside the key by `ifixai init` and the setup wizard.+PROVIDER_COMPANION_ENV_KEYS: dict[str, tuple[str, ...]] = {+ "cloudflare": ("CLOUDFLARE_ACCOUNT_ID",),}def detect_available_providers() -> list[tuple[str, str]]:+ """Providers whose key and every companion setting are in the environment.++ A key without its companion cannot make a call, so it is not offered as ready.+ """return [(provider, env_var)for provider, env_var in PROVIDER_ENV_KEYS.items()if os.environ.get(env_var)+ and all(+ os.environ.get(companion)+ for companion in PROVIDER_COMPANION_ENV_KEYS.get(provider, ())+ )]@@ -91,7 +108,10 @@ def init(non_interactive: bool) -> None:click.echo(click.style(" No provider API keys detected in environment.", fg="yellow"))click.echo(" Set one of the following before running tests:")for provider, env_var in PROVIDER_ENV_KEYS.items():- click.echo(f" - {env_var} (for --provider {provider})")+ required = " + ".join(+ (env_var, *PROVIDER_COMPANION_ENV_KEYS.get(provider, ()))+ )+ click.echo(f" - {required} (for --provider {provider})")returnclick.echo(" Detected provider keys:")diff --git a/ifixai/cli/model_catalog.py b/ifixai/cli/model_catalog.pyindex 72082e0..5f4a242 100644--- a/ifixai/cli/model_catalog.py+++ b/ifixai/cli/model_catalog.py@@ -35,6 +35,7 @@ DEFAULT_MODEL: dict[str, str] = {"openrouter": "openai/gpt-4o","orcarouter": "openai/gpt-4o","requesty": "openai/gpt-4o-mini",+ "cloudflare": "workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast","openai": "gpt-4o","anthropic": "claude-3-5-sonnet-latest","gemini": "gemini-2.0-flash",@@ -76,6 +77,13 @@ MODEL_SUGGESTIONS: dict[str, list[tuple[str, str]]] = {("google/gemini-2.5-flash", "Google Gemini 2.5 Flash — fast and inexpensive"),("google/gemini-2.5-pro", "Google Gemini 2.5 Pro — large context, strong reasoning"),],+ "cloudflare": [+ ("workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast", "Meta Llama 3.3 70B on Workers AI — free plan covers about one small inspection a day"),+ ("openai/gpt-4.1-mini", "OpenAI GPT-4.1 Mini — cheap and fast (needs Unified Billing credits or a stored key)"),+ ("openai/gpt-4.1", "OpenAI GPT-4.1 — strong, widely available (needs credits or a stored key)"),+ ("anthropic/claude-haiku-4.5", "Anthropic Claude Haiku 4.5 — fast & cheap (needs credits or a stored key)"),+ ("google-ai-studio/gemini-2.5-flash", "Google Gemini 2.5 Flash — fast and inexpensive (needs credits or a stored key)"),+ ],"openai": [("gpt-4o", "Flagship — strong general reasoning"),("gpt-4o-mini", "Cheaper & faster"),diff --git a/ifixai/cli/run.py b/ifixai/cli/run.pyindex a72dc2d..7a590c8 100644--- a/ifixai/cli/run.py+++ b/ifixai/cli/run.py@@ -148,6 +148,7 @@ PROVIDER_CHOICES = ["openrouter","orcarouter","requesty",+ "cloudflare",]FORMAT_CHOICES = ["json", "markdown", "both"]diff --git a/ifixai/cli/setup_cmd.py b/ifixai/cli/setup_cmd.pyindex 3c4cad9..3928bf7 100644--- a/ifixai/cli/setup_cmd.py+++ b/ifixai/cli/setup_cmd.py@@ -12,7 +12,11 @@ from ifixai._version import VERSION as IFIXAI_VERSIONfrom ifixai.cli import uifrom ifixai.cli._branding import print_startup_bannerfrom ifixai.cli.config_file import CONFIG_FILENAME, JudgeSpec, RunConfig, write_config-from ifixai.cli.init import PROVIDER_ENV_KEYS, detect_available_providers+from ifixai.cli.init import (+ PROVIDER_COMPANION_ENV_KEYS,+ PROVIDER_ENV_KEYS,+ detect_available_providers,+)from ifixai.cli.model_catalog import default_model, suggestionsfrom ifixai.core.fixture_loader import list_fixture_names, load_fixturefrom ifixai.harness.suites import suite_catalog@@ -22,6 +26,7 @@ _PROVIDER_DESCRIPTIONS: dict[str, str] = {"openrouter": "One key → many models (OpenAI, Anthropic, Google, Llama…)","orcarouter": "OpenAI-compatible gateway — many models, one key","requesty": "Requesty gateway — one key → 400+ models (OpenAI, Anthropic, Google…)",+ "cloudflare": "Cloudflare AI Gateway — one token → Workers AI plus OpenAI, Anthropic, Google…","openai": "OpenAI API — GPT-4o / o-series","anthropic": "Anthropic API — Claude family","gemini": "Google Gemini",@@ -50,6 +55,7 @@ _ALL_PROVIDERS = ["openrouter","orcarouter","requesty",+ "cloudflare","openai","anthropic","gemini",@@ -112,14 +118,16 @@ def _pick_model(provider: str, *, role: str) -> str | None:def _missing_keys(selected: list[tuple[str, str]]) -> list[tuple[str, str, str]]:"""For each (role, provider) chosen, return (role, provider, env_var) whose keyis not set in the environment. De-duplicated by env var so a shared key is- reported once, giving the user one clean export list."""+ reported once, giving the user one clean export list. A provider's companion+ settings (e.g. the Cloudflare account ID) are listed right after its key."""missing: list[tuple[str, str, str]] = []seen: set[str] = set()for role, prov in selected:- env = PROVIDER_ENV_KEYS.get(prov)- if env and env not in seen and not os.environ.get(env):- seen.add(env)- missing.append((role, prov, env))+ required = (PROVIDER_ENV_KEYS.get(prov), *PROVIDER_COMPANION_ENV_KEYS.get(prov, ()))+ for env in required:+ if env and env not in seen and not os.environ.get(env):+ seen.add(env)+ missing.append((role, prov, env))return missingdiff --git a/ifixai/core/types.py b/ifixai/core/types.pyindex 0577619..9a9edaf 100644--- a/ifixai/core/types.py+++ b/ifixai/core/types.py@@ -243,7 +243,7 @@ class ProviderConfig(BaseModel):"Request structured JSON output from the provider (response_format=""json_object). Set ONLY for LLM-judge calls so cheap models reliably ""emit a parseable verdict; never for the system-under-test, whose reply "- "must stay natural. Honored by the openai, openrouter, orcarouter, requesty, azure, litellm "+ "must stay natural. Honored by the openai, openrouter, orcarouter, requesty, cloudflare, azure, litellm ""(response_format) and gemini (response_mime_type) providers, which fall ""back to free text if the model rejects it; a no-op on providers with no ""JSON mode (anthropic, bedrock, huggingface), where the json-repair "diff --git a/ifixai/providers/cloudflare.py b/ifixai/providers/cloudflare.pynew file mode 100644index 0000000..c30e943--- /dev/null+++ b/ifixai/providers/cloudflare.py@@ -0,0 +1,471 @@+"""Cloudflare AI Gateway, reached through its OpenAI-compatible endpoint.++One Cloudflare API token fronts Workers AI and third-party vendors. The model is+a ``provider/model`` slug such as ``openai/gpt-4.1-mini``, which for Workers AI+reads ``workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast``. The account and the+gateway are part of the URL, so they are read from ``CLOUDFLARE_ACCOUNT_ID`` and+``CLOUDFLARE_GATEWAY_ID`` unless an endpoint is given.+https://developers.cloudflare.com/ai-gateway/usage/chat-completion/+"""++import asyncio+import os+import re+from collections.abc import Mapping+from http import HTTPStatus+from typing import NoReturn++import openai+from openai.types.chat import ChatCompletion+from typing_extensions import NotRequired, TypedDict++from ifixai.core.types import ChatMessage, ProviderConfig+from ifixai.providers.base import (+ ChatProvider,+ ProviderAuthError,+ ProviderConnectionError,+ ProviderEmptyContentError,+ ProviderError,+ ProviderRateLimitError,+ ProviderResponseError,+ ProviderTimeoutError,+ create_chat_completion_json_fallback,+ raise_for_http_status,+ raise_if_choice_errored,+ raise_if_truncated,+)++PROVIDER_NAME = "cloudflare"+# Served by Workers AI on the free plan, so a new account can run it without+# prepaid credits; third-party models bill Unified Billing credits or a stored+# provider key. Written with the ``workers-ai/`` prefix because that is the form+# Cloudflare documents for this endpoint; the bare ``@cf/...`` id is also accepted.+DEFAULT_MODEL = "workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast"+# The gateway host, not the api.cloudflare.com REST route: a token issued for AI+# Gateway is accepted here and answered 401 there, which wants a token with the+# Workers AI permission instead.+GATEWAY_BASE_URL_TEMPLATE = (+ "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/compat"+)+CREDENTIAL_ENV_VAR = "CLOUDFLARE_API_TOKEN"+ACCOUNT_ID_ENV_VAR = "CLOUDFLARE_ACCOUNT_ID"+GATEWAY_ID_ENV_VAR = "CLOUDFLARE_GATEWAY_ID"+ACCOUNT_ID_PATTERN = re.compile(r"[0-9a-f]{32}")+# One URL path segment. Case is kept: the gateway host tells "default" from "DEFAULT".+GATEWAY_ID_PATTERN = re.compile(r"[A-Za-z0-9_-]{1,64}")+DEFAULT_GATEWAY_ID = "default"+# Where Cloudflare's authentication page says the token goes on this host. Its+# SDK examples for this endpoint send the token as the API key instead, so the+# token travels both ways and either reading of the documentation is honoured.+GATEWAY_AUTHORIZATION_HEADER = "cf-aig-authorization"+SKIP_CACHE_HEADER = "cf-aig-skip-cache"+# AI Gateway's own code for "no gateway with this ID in the account".+GATEWAY_NOT_CONFIGURED_CODE = 2001+# Workers AI's code for "the account's free daily allocation is used up".+ALLOCATION_EXHAUSTED_CODE = 4006+# Hard ceiling on max_tokens for AI Gateway calls. Per-call ``config.max_tokens``+# is clamped to this value; unset config falls through to the ceiling. Prevents+# verbose generations from blowing wall-time and credits on long fixtures. Mirrors+# the OpenRouter provider ceiling so judge/SUT replies aren't truncated mid-verdict.+MAX_TOKENS_CEILING: int = 8192++ACCOUNT_ID_MISSING = (+ f"{ACCOUNT_ID_ENV_VAR} is not set. The account ID is part of the gateway URL: "+ f"set {ACCOUNT_ID_ENV_VAR} to the Account ID shown in the Cloudflare dashboard, "+ "or pass the full base URL with --endpoint"+)+ACCOUNT_ID_MALFORMED = (+ f"{ACCOUNT_ID_ENV_VAR} is not a Cloudflare account ID. It must be the 32 "+ "hexadecimal characters shown as Account ID in the dashboard; a gateway name, "+ "a zone ID or an API token will not work"+)+GATEWAY_ID_MALFORMED = (+ f"{GATEWAY_ID_ENV_VAR} is not a gateway ID. It must be the gateway's name as "+ "shown in the Cloudflare dashboard: up to 64 letters, digits, hyphens or "+ "underscores"+)+GATEWAY_NOT_FOUND_HINT = (+ "No AI Gateway with this ID exists in the account. Create it in the Cloudflare "+ f"dashboard, or set {GATEWAY_ID_ENV_VAR} to an existing gateway; the ID is "+ "case-sensitive."+)+# `insufficient_quota` is the phrase the fail-fast check keys on for a spent+# account; Cloudflare's own wording carries none of the phrases it looks for.+ALLOCATION_EXHAUSTED_HINT = (+ "The account's free daily Workers AI allocation is spent (insufficient_quota). "+ "It is a per-day allocation, so no retry in this run can succeed: wait for it "+ "to return, move the account to a paid Workers plan, or use a model billed "+ "through Unified Billing."+)+# A made-up token, no token and a valid token against another account were all+# seen to return one identical 401 body. The permission is the one Cloudflare+# documents for an authenticated gateway.+AUTHENTICATION_HINT = (+ f"Cloudflare reports three causes this one way: {CREDENTIAL_ENV_VAR} is not an "+ "active API token, the token lacks the AI Gateway > Run permission, or the "+ f"account ID ({ACCOUNT_ID_ENV_VAR}, or the one in --endpoint) is not the "+ "account the token belongs to."+)++# Checked in order, most specific first: a timeout is also a connection error,+# and the 429 class is also a status error.+SDK_ERROR_TRANSLATIONS: dict[type[openai.APIError], type[ProviderError]] = {+ openai.RateLimitError: ProviderRateLimitError,+ openai.APITimeoutError: ProviderTimeoutError,+ openai.APIConnectionError: ProviderConnectionError,+}++ClientCacheKey = tuple[object, ...]+++class GatewayConnection(TypedDict):+ """Everything that distinguishes one SDK client from another."""++ api_key: str+ base_url: str+ timeout: float+ max_retries: int+++class GatewayMessage(TypedDict):+ role: str+ content: str+++class JsonResponseFormat(TypedDict):+ type: str+++class GatewayChatRequest(TypedDict):+ """Body of one chat-completions call; optional keys are sent only when set."""++ model: str+ messages: list[GatewayMessage]+ max_tokens: int+ temperature: float+ seed: NotRequired[int]+ response_format: NotRequired[JsonResponseFormat]+++class CloudflareAIGatewayProvider(ChatProvider):+ def __init__(self) -> None:+ self.clients: dict[ClientCacheKey, openai.AsyncOpenAI] = {}+ self.client_lock = asyncio.Lock()++ async def get_client(self, connection: GatewayConnection) -> openai.AsyncOpenAI:+ """Return a long-lived AsyncOpenAI client keyed on connection params.++ Caching by (api_key, base_url, timeout, max_retries) lets the+ underlying httpx pool reuse TCP/TLS across LLM calls instead of+ paying a handshake per request.+ """+ key: ClientCacheKey = tuple(connection.values())+ cached = self.clients.get(key)+ if cached is not None:+ return cached+ async with self.client_lock:+ cached = self.clients.get(key)+ if cached is not None:+ return cached+ client = build_client(connection)+ self.clients[key] = client+ return client++ async def aclose(self) -> None:+ for client in self.clients.values():+ await client.close()+ self.clients.clear()++ async def send_message(+ self,+ messages: list[ChatMessage],+ config: ProviderConfig,+ ) -> str:+ connection = resolve_connection(config, os.environ)+ endpoint = connection["base_url"]+ client = await self.get_client(connection)+ try:+ response = await create_chat_completion_json_fallback(+ client, **build_chat_request(messages, config)+ )+ except openai.APIError as exc:+ raise_for_sdk_error(endpoint, exc)+ return extract_reply(response, config, endpoint)+++def resolve_connection(+ config: ProviderConfig, environ: Mapping[str, str]+) -> GatewayConnection:+ """Reduce a config and the environment to the settings a client is built from.++ The account and the gateway come from the environment because a judge is+ configured with a provider and a model only: it has no endpoint to carry them.+ """+ base_url = (config.endpoint or "").strip() or build_gateway_url(environ)+ return GatewayConnection(+ api_key=require_api_token(config.api_key, base_url),+ base_url=base_url,+ timeout=float(config.timeout),+ max_retries=config.max_retries,+ )+++def build_gateway_url(environ: Mapping[str, str]) -> str:+ """Return the gateway's base URL for the account and gateway in the environment."""+ return GATEWAY_BASE_URL_TEMPLATE.format(+ account_id=require_account_id(environ.get(ACCOUNT_ID_ENV_VAR, "")),+ gateway_id=resolve_gateway_id(environ.get(GATEWAY_ID_ENV_VAR, "")),+ )+++def require_account_id(raw_account_id: str) -> str:+ """Return the account ID in lower case, refusing a missing or malformed one.++ Lower case because the gateway host answers an upper-case account ID with+ the same 401 it gives a wrong one.+ """+ account_id = raw_account_id.strip().lower()+ if not account_id:+ refuse_setting(ACCOUNT_ID_MISSING)+ if not ACCOUNT_ID_PATTERN.fullmatch(account_id):+ refuse_setting(ACCOUNT_ID_MALFORMED)+ return account_id+++def resolve_gateway_id(raw_gateway_id: str) -> str:+ """Return the named gateway, or the default gateway when none is named.++ The ID becomes a URL path segment, so anything that is not one is refused+ here instead of being sent as part of a different path.+ """+ gateway_id = raw_gateway_id.strip()+ if not gateway_id:+ return DEFAULT_GATEWAY_ID+ if not GATEWAY_ID_PATTERN.fullmatch(gateway_id):+ refuse_setting(GATEWAY_ID_MALFORMED)+ return gateway_id+++def refuse_setting(details: str) -> NoReturn:+ """Stop the run over an environment setting no request can succeed without.++ Raised as an auth error, not a connection error: a connection fault is+ transient by type, so a judge with a bad setting would lose every probe one+ at a time instead of stopping the run. The value is never echoed, because+ the usual mistake is pasting the token into the wrong variable.+ """+ raise ProviderAuthError(+ provider=PROVIDER_NAME, endpoint=GATEWAY_BASE_URL_TEMPLATE, details=details+ )+++def require_api_token(raw_token: str, endpoint: str) -> str:+ """Return the token without surrounding whitespace, refusing a blank one.++ A pasted token often carries a trailing newline, which is an illegal header+ value: the request fails before it is sent and the transport error quotes+ the token. A blank token is refused here because the SDK reports it as a+ missing ``OPENAI_API_KEY``, which names the wrong variable.+ """+ api_token = raw_token.strip()+ if not api_token:+ raise ProviderAuthError(+ provider=PROVIDER_NAME,+ endpoint=endpoint,+ details=(+ "No API token was supplied (empty or whitespace only). "+ f"Set {CREDENTIAL_ENV_VAR} or pass --api-key"+ ),+ )+ return api_token+++def build_client(connection: GatewayConnection) -> openai.AsyncOpenAI:+ """Build an SDK client that stamps the gateway headers on every request."""+ return openai.AsyncOpenAI(+ api_key=connection["api_key"],+ base_url=connection["base_url"],+ timeout=connection["timeout"],+ max_retries=connection["max_retries"],+ default_headers=build_gateway_headers(connection["api_key"]),+ )+++def build_gateway_headers(api_token: str) -> dict[str, str]:+ """Authenticate to the gateway and bypass its response cache.++ A gateway with caching on replays one stored answer to every identical+ request, which would make a repeated probe measure the cache instead of the+ model.+ """+ return {+ GATEWAY_AUTHORIZATION_HEADER: f"Bearer {api_token}",+ SKIP_CACHE_HEADER: "true",+ }+++def build_chat_request(+ messages: list[ChatMessage], config: ProviderConfig+) -> GatewayChatRequest:+ """Assemble the chat-completions body for one call."""+ request = GatewayChatRequest(+ model=config.model or DEFAULT_MODEL,+ messages=[+ GatewayMessage(role=message.role, content=message.content)+ for message in messages+ ],+ max_tokens=resolve_max_tokens(config.max_tokens),+ temperature=config.temperature,+ )+ if config.seed is not None:+ request["seed"] = config.seed+ if config.json_output:+ # Constrain judge calls to valid JSON so cheap models reliably emit a+ # parseable verdict instead of breaking the contract. Falls back to free+ # text (json-repair handles parsing) if the model does not support+ # response_format.+ #+ # No `reasoning` suppression here, unlike OpenRouter. `reasoning` is+ # OpenRouter's own gateway extension and Cloudflare documents no+ # equivalent for this route. Sending an undocumented field on the judge+ # path risks a 400 for no confirmed benefit.+ request["response_format"] = JsonResponseFormat(type="json_object")+ return request+++def resolve_max_tokens(requested: int | None) -> int:+ """Clamp the caller's token budget to the ceiling; unset takes the ceiling."""+ if requested is None:+ return MAX_TOKENS_CEILING+ return min(requested, MAX_TOKENS_CEILING)+++def raise_for_sdk_error(endpoint: str, exc: openai.APIError) -> NoReturn:+ """Translate an OpenAI-SDK failure into the matching provider exception."""+ if isinstance(exc, openai.AuthenticationError):+ raise_for_rejected_access(endpoint, exc, AUTHENTICATION_HINT)+ if isinstance(exc, openai.RateLimitError) and is_allocation_exhausted(exc):+ raise_for_exhausted_allocation(endpoint, exc)+ for sdk_error, provider_error in SDK_ERROR_TRANSLATIONS.items():+ if isinstance(exc, sdk_error):+ raise provider_error(+ provider=PROVIDER_NAME, endpoint=endpoint, details=str(exc)+ ) from exc+ if not isinstance(exc, openai.APIStatusError):+ raise ProviderResponseError(+ provider=PROVIDER_NAME, endpoint=endpoint, details=str(exc)+ ) from exc+ if is_missing_gateway(exc):+ raise_for_rejected_access(endpoint, exc, GATEWAY_NOT_FOUND_HINT)+ if exc.status_code == HTTPStatus.PAYMENT_REQUIRED:+ raise_for_payment_required(endpoint, exc)+ # 408 and 5xx are the gateway or the model behind it having a moment, so they+ # are split off as a transient overload; everything else stays a response error.+ raise_for_http_status(PROVIDER_NAME, endpoint, exc)+++def raise_for_rejected_access(+ endpoint: str, exc: openai.APIError, hint: str+) -> NoReturn:+ """Report a rejected token, account or gateway as fatal, with what to check.++ None of them clears on a retry, so each has to stop the run with Cloudflare's+ own message kept and the likely causes named beside it.+ """+ raise ProviderAuthError(+ provider=PROVIDER_NAME, endpoint=endpoint, details=f"{exc}. {hint}"+ ) from exc+++def is_missing_gateway(exc: openai.APIStatusError) -> bool:+ """True when AI Gateway says the account has no gateway with this ID.++ It answers 400 for that, a status that would otherwise read as one bad+ request and fail the same way on every probe. The SDK keeps only the body's+ ``error`` member, which the gateway sends as a list of coded faults.+ """+ faults = exc.body+ if exc.status_code != HTTPStatus.BAD_REQUEST or not isinstance(faults, list):+ return False+ return any(+ isinstance(fault, dict) and fault.get("code") == GATEWAY_NOT_CONFIGURED_CODE+ for fault in faults+ )+++def is_allocation_exhausted(exc: openai.APIStatusError) -> bool:+ """True when Workers AI says the account's free daily allocation is used up.++ Unlike the gateway's own envelope, a Workers AI fault has no ``error``+ member, so the SDK keeps the whole body.+ """+ body = exc.body+ return (+ isinstance(body, dict) and body.get("internalCode") == ALLOCATION_EXHAUSTED_CODE+ )+++def raise_for_exhausted_allocation(+ endpoint: str, exc: openai.APIStatusError+) -> NoReturn:+ """Report a spent daily allocation as a rate limit that will not clear.++ It arrives as a 429, which reads as a throttle that lifts in a moment. Left+ that way a run fails every remaining probe one at a time. It stays a rate+ limit error, as the fail-fast check expects of a spent account, with the+ phrase that check looks for.+ """+ raise ProviderRateLimitError(+ provider=PROVIDER_NAME,+ endpoint=endpoint,+ details=f"{exc}. {ALLOCATION_EXHAUSTED_HINT}",+ ) from exc+++def raise_for_payment_required(endpoint: str, exc: openai.APIStatusError) -> NoReturn:+ """Report an empty Unified Billing balance as non-retryable.++ The gateway words the 402 body two different ways depending on the model,+ and one of them matches none of the phrases the fail-fast check looks for.+ The status's own reason phrase goes into the detail: it is what the+ fail-fast check and the operator-facing hint both key on.+ """+ status = HTTPStatus.PAYMENT_REQUIRED+ raise ProviderResponseError(+ provider=PROVIDER_NAME,+ endpoint=endpoint,+ details=f"HTTP {status.value} {status.phrase}: {exc}",+ ) from exc+++def extract_reply(+ response: ChatCompletion, config: ProviderConfig, endpoint: str+) -> str:+ """Return the reply text, rejecting aborted, cut-off and empty generations."""+ if not response.choices:+ raise ProviderResponseError(+ provider=PROVIDER_NAME,+ endpoint=endpoint,+ details=f"No choices in response (id={response.id})",+ )+ choice = response.choices[0]+ finish_reason = choice.finish_reason or "unknown"+ if choice.message is None:+ raise ProviderResponseError(+ provider=PROVIDER_NAME,+ endpoint=endpoint,+ details=f"Missing message in choice (finish_reason={finish_reason})",+ )+ content = choice.message.content+ # An upstream abort is checked first: it can arrive with no text at all, and+ # reading that as an empty answer would void the inspection instead of the probe.+ raise_if_choice_errored(PROVIDER_NAME, endpoint, choice, content or "")+ if config.reject_truncated:+ raise_if_truncated(PROVIDER_NAME, endpoint, finish_reason, content or "")+ if not content:+ raise ProviderEmptyContentError(+ provider=PROVIDER_NAME,+ endpoint=endpoint,+ details=f"Empty content in response (finish_reason={finish_reason})",+ )+ return contentdiff --git a/ifixai/providers/resolver.py b/ifixai/providers/resolver.pyindex 4aba300..6dca370 100644--- a/ifixai/providers/resolver.py+++ b/ifixai/providers/resolver.py@@ -60,6 +60,11 @@ try:except ImportError:RequestyProvider = None # type: ignore[assignment,misc]+try:+ from ifixai.providers.cloudflare import CloudflareAIGatewayProvider+except ImportError:+ CloudflareAIGatewayProvider = None # type: ignore[assignment,misc]+try:from ifixai.providers.litellm import LiteLLMProviderexcept ImportError:@@ -75,6 +80,7 @@ REGISTERED_PROVIDERS: tuple[str, ...] = ("openrouter","orcarouter","requesty",+ "cloudflare","anthropic","gemini","azure",@@ -97,6 +103,7 @@ _PROVIDER_MAP: dict[str, type] = {"openrouter": OpenRouterProvider,"orcarouter": OrcaRouterProvider,"requesty": RequestyProvider,+ "cloudflare": CloudflareAIGatewayProvider,"anthropic": AnthropicProvider,"gemini": GeminiProvider,"azure": AzureOpenAIProvider,@@ -177,6 +184,7 @@ _PROVIDER_CREDENTIAL_ENV_VARS: dict[str, tuple[str, ...]] = {"openrouter": ("OPENROUTER_API_KEY",),"orcarouter": ("ORCAROUTER_API_KEY",),"requesty": ("REQUESTY_API_KEY",),+ "cloudflare": ("CLOUDFLARE_API_TOKEN",),"litellm": ("LITELLM_API_KEY",),}diff --git a/ifixai/providers/secrets.py b/ifixai/providers/secrets.pyindex 681e143..1efc6e5 100644--- a/ifixai/providers/secrets.py+++ b/ifixai/providers/secrets.py@@ -20,6 +20,7 @@ _SECRET_PATTERNS: Final[tuple[Pattern[str], ...]] = (re.compile(r"^sk-or-[A-Za-z0-9_-]{20,}$"),re.compile(r"^sk-orca-[A-Za-z0-9_-]{20,}$"),re.compile(r"^rqsty-[A-Za-z0-9+/=_-]{20,}$"),+ re.compile(r"^cf(?:ut|at|k)_[A-Za-z0-9_-]{40,}$"),re.compile(r"^ak-[A-Za-z0-9_-]{20,}$"),re.compile(r"^sk-ant-[A-Za-z0-9_-]{20,}$"),re.compile(r"^sk-[A-Za-z0-9_-]{20,}$"),@@ -54,6 +55,10 @@ _SCRUB_RULES: Final[tuple[tuple[Pattern[str], str], ...]] = (re.compile(r"rqsty-[A-Za-z0-9+/=_-]{20,}"),"***REDACTED_REQUESTY_KEY***",),+ (+ re.compile(r"cf(?:ut|at|k)_[A-Za-z0-9_-]{40,}"),+ "***REDACTED_CLOUDFLARE_TOKEN***",+ ),(re.compile(r"(?<![A-Za-z0-9_-])ak-[A-Za-z0-9_-]{20,}(?![A-Za-z0-9_-])"),"***REDACTED_ATLASCLOUD_KEY***",diff --git a/ifixai/reporting/scorecard.py b/ifixai/reporting/scorecard.pyindex c3cef8f..1cc5ed0 100644--- a/ifixai/reporting/scorecard.py+++ b/ifixai/reporting/scorecard.py@@ -284,16 +284,41 @@ def b32_not_applicable_warning(# Providers that front many vendors behind one name. For these the real grading# vendor is the model slug's prefix, so a cross-vendor judge routed through one-# aggregator (e.g. a Gemini SUT graded by a Claude judge, both via OpenRouter or-# Requesty) is recognized as independent instead of mislabeled "self-judge".+# aggregator (e.g. a Gemini SUT graded by a Claude judge, both via OpenRouter,+# Requesty or Cloudflare AI Gateway) is recognized as independent instead of+# mislabeled "self-judge"._AGGREGATOR_PROVIDERS: Final[frozenset[str]] = frozenset(- {"openrouter", "orcarouter", "requesty", "atlascloud", "litellm", "http", "langchain"}+ {"openrouter", "orcarouter", "requesty", "cloudflare", "atlascloud", "litellm", "http", "langchain"}+)++# Namespaces that sit in front of the author, as in the Workers AI ids+# "@cf/meta/llama-3.3-70b-instruct-fp8-fast" and "@hf/google/gemma-7b-it", which+# Cloudflare AI Gateway also accepts behind its own "workers-ai/" prefix: the+# vendor is the first segment that is not a namespace.+CATALOG_NAMESPACE_PREFIXES: Final[frozenset[str]] = frozenset(+ {"workers-ai", "@cf", "@hf"})# Distinct provider slugs that front the SAME underlying model vendor, so a# self-judge across them is still same-vendor (biased): Azure OpenAI serves# OpenAI models; the direct Gemini API and a "google/..." slug are both Google.-_VENDOR_ALIASES: Final[dict[str, str]] = {"azure": "openai", "gemini": "google"}+# "google-ai-studio" and "grok" are the names Cloudflare AI Gateway routes Google+# and xAI models under. The rest are one vendor spelled two ways: gateways+# disagree on xAI and Mistral (Workers AI alone uses both Mistral spellings), and+# Workers AI names Meta, DeepSeek and Z.ai by their Hugging Face organisation.+# Without the alias a Llama judge on one gateway reads as independent of a Llama+# system under test on another.+_VENDOR_ALIASES: Final[dict[str, str]] = {+ "azure": "openai",+ "gemini": "google",+ "google-ai-studio": "google",+ "grok": "xai",+ "x-ai": "xai",+ "mistralai": "mistral",+ "meta-llama": "meta",+ "deepseek-ai": "deepseek",+ "zai-org": "z-ai",+}# Aggregator prefixes that name the cloud hosting the model, not its developer# ("bedrock/claude-sonnet-4-6@eu-central-1"), so the vendor comes from the family.@@ -311,6 +336,8 @@ def grading_vendor(provider: str, model: str | None) -> str:p = (provider or "").lower()if p in _AGGREGATOR_PROVIDERS and model and "/" in model:raw, _, rest = model.lower().partition("/")+ while raw in CATALOG_NAMESPACE_PREFIXES and "/" in rest:+ raw, _, rest = rest.partition("/")if raw in _CLOUD_HOST_PREFIXES:raw = next((v for f, v in _MODEL_FAMILY_VENDORS.items() if f in rest), raw)else:diff --git a/ifixai/scaffold/playbook.md b/ifixai/scaffold/playbook.mdindex 0b60910..53e962c 100644--- a/ifixai/scaffold/playbook.md+++ b/ifixai/scaffold/playbook.md@@ -97,7 +97,7 @@ IFIXAI_SURFACE=skill uvx --from "ifixai[<provider>]" ifixai run \```Substitute a provider valid as BOTH the install extra and the `--provider` value: one of-`anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, or `huggingface`. (`mock`+`anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, or `huggingface`. (`mock`runs fully offline for a free smoke test, with `--provider mock --api-key mock --eval-modeself`.) The extra installs that provider's SDK; if you test on one provider and judge onanother, install the union. After the user approves the dry-run estimate, rerun the identical@@ -146,6 +146,7 @@ over the file). A missing key fails fast and names the variable to set.| openrouter | `OPENROUTER_API_KEY` || orcarouter | `ORCAROUTER_API_KEY` || requesty | `REQUESTY_API_KEY` |+| cloudflare | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` || huggingface | `HUGGINGFACE_API_TOKEN` or `HF_TOKEN` |If the SUT and a judge share a provider, one key covers both. A judge on a second provider@@ -408,7 +409,7 @@ at each. Map the picks to flags yourself.build a stand-in that mirrors your setup and test that instead?" A URL keeps this path; a "no"moves to the fallback. Don't guess an endpoint or fall back silently.- *Bare stand-in (fallback, no endpoint).* Offer the providers the user has a key for:- `anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `huggingface`, with+ `anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `huggingface`, with`--grounding fixture`. Offer to pin a model (`--model <id>`) or take the provider default.Maps to `--provider` / `--model`. Tests the model, not the deployment. (`mock` runs offlinefor free.)@@ -421,7 +422,7 @@ at each. Map the picks to flags yourself.- *Self (cheapest, biased, not citable)*: the model grades itself. Maps to `--eval-mode self`.Only a smoke test. (With a single provider key and no judge, standard mode REFUSES rather thanself-judge silently, so this is the explicit opt-in.)-- On an aggregator like OpenRouter or Requesty, "different vendor" means a different model-slug prefix+- On an aggregator like OpenRouter, Requesty or Cloudflare AI Gateway, "different vendor" means a different model-slug prefix(e.g. `google/...` SUT graded by `openai/...`), which the engine now recognizes as citable.**3. Depth / how much to run.** Offer a suite, smallest first, with the trade-off:diff --git a/ifixai/tests/providers/test_cloudflare_provider.py b/ifixai/tests/providers/test_cloudflare_provider.pynew file mode 100644index 0000000..8b27aa0--- /dev/null+++ b/ifixai/tests/providers/test_cloudflare_provider.py@@ -0,0 +1,941 @@+"""Cloudflare AI Gateway adapter: wire contract, URL resolution and error mapping.++The wire tests drive the real OpenAI SDK against a loopback stand-in for the+gateway and replay the bodies the live gateway host returned on 2026-10-09, so+the mapping is pinned to what the SDK raises for real payloads rather than to+hand-built exceptions. The 429 and 5xx bodies are the exception: none could be+provoked, so they reuse the gateway's envelope with a message of their own, and+the mapping for them keys on the HTTP status alone.++References:+ https://developers.cloudflare.com/ai-gateway/usage/chat-completion/+ https://developers.cloudflare.com/ai-gateway/configuration/authentication/+ https://developers.cloudflare.com/ai-gateway/features/caching/+"""++from __future__ import annotations++import asyncio+import json+import threading+from collections.abc import Iterator+from contextlib import contextmanager+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from typing import Any, TypedDict+from unittest.mock import AsyncMock, MagicMock, patch++import pytest+from click.testing import CliRunner++pytest.importorskip("openai")++from ifixai.cli.main import ifixai_cli+from ifixai.core.types import ChatMessage, ProviderConfig+from ifixai.providers.base import (+ ProviderAuthError,+ ProviderEmptyContentError,+ ProviderError,+ ProviderOverloadedError,+ ProviderRateLimitError,+ ProviderResponseError,+ ProviderTruncatedError,+ friendly_provider_message,+ is_fatal_provider_error,+)+from ifixai.providers.cloudflare import (+ ACCOUNT_ID_ENV_VAR,+ DEFAULT_MODEL,+ GATEWAY_AUTHORIZATION_HEADER,+ GATEWAY_ID_ENV_VAR,+ MAX_TOKENS_CEILING,+ SKIP_CACHE_HEADER,+ CloudflareAIGatewayProvider,+ resolve_connection,+)++# Assembled at import time so the source holds no token-shaped literal for the+# repo's secret scanner (gitleaks, see .pre-commit-config.yaml) to flag.+SYNTHETIC_TOKEN = "cfut_" + "syntheticLoopback" * 3+ACCOUNT_ID = "0123456789abcdef" * 2+GATEWAY_BASE_URL = f"https://gateway.ai.cloudflare.com/v1/{ACCOUNT_ID}/default/compat"+BARE_WORKERS_AI_MODEL = "@cf/meta/llama-3.3-70b-instruct-fp8-fast"+MESSAGES = [+ ChatMessage(role="system", content="You are a claims assistant."),+ ChatMessage(role="user", content="Reply with the single word: pong"),+]+++def gateway_error(status: int, code: int, message: str) -> dict[str, Any]:+ """An error in the envelope the gateway host puts around its own faults."""+ return {+ "success": False,+ "result": [],+ "messages": [],+ "error": [{"code": code, "message": message}],+ "name": "AiGatewayError",+ "httpCode": status,+ "internalCode": code,+ "message": message,+ "description": message,+ }+++# Captured: returned alike for a made-up token, for no token and for a valid+# token against an account it does not belong to.+UNAUTHORIZED_BODY = gateway_error(401, 2009, "Unauthorized")+# Captured: a gateway ID the account does not have.+GATEWAY_NOT_CONFIGURED_BODY = gateway_error(+ 400, 2001, "Please configure AI Gateway in the Cloudflare dashboard"+)+# Captured: a third-party model on an account with no Unified Billing credits.+NO_CREDITS_BODY = gateway_error(+ 402,+ 2021,+ "Insufficient unified billing credits. Please add additional credits on the "+ "AI Gateway Cloudflare dashboard to continue using provider models.",+)+# Captured: the same condition, worded differently for other models. It matches+# none of the phrases the fail-fast path looks for.+NO_BALANCE_BODY = {+ "error": {+ "message": "Insufficient balance; add money to your gateway or use BYOK",+ "type": "gateway_error",+ "code": None,+ }+}+# Captured: a slug the gateway does not route.+MODEL_NOT_FOUND_BODY = gateway_error(+ 400, 2008, "Model not found: nosuchvendor/some-model"+)+# Captured: a Workers AI id that does not exist.+WORKERS_AI_NO_SUCH_MODEL_BODY = {+ "name": "AiError",+ "internalCode": 5007,+ "httpCode": 400,+ "message": (+ "AiError: No such model: No such model @cf/meta/no-such-model-ifixai or "+ "task (2f92db78-f004-4e34-bc23-383bd05f33cb)"+ ),+ "description": "No such model @cf/meta/no-such-model-ifixai or task",+ "requestId": "2f92db78-f004-4e34-bc23-383bd05f33cb",+}+# Captured: Workers AI once the account's free daily allocation is used up.+ALLOCATION_EXHAUSTED_BODY = {+ "name": "AiError",+ "internalCode": 4006,+ "httpCode": 429,+ "message": (+ "AiError: AiError: you have used up your daily free allocation of 10,000 "+ "neurons, please upgrade to Cloudflare's Workers Paid plan if you would "+ "like to continue usage. (983e5e13-234a-48d2-abef-0eb6eb55b753)"+ ),+ "description": (+ "you have used up your daily free allocation of 10,000 neurons, please "+ "upgrade to Cloudflare's Workers Paid plan if you would like to continue "+ "usage."+ ),+ "requestId": "983e5e13-234a-48d2-abef-0eb6eb55b753",+}+# Captured: a Workers AI model the account's plan does not include.+PLAN_RESTRICTED_BODY = {+ "name": "AiError",+ "internalCode": 5035,+ "httpCode": 403,+ "message": (+ "AiError: Model @cf/moonshotai/kimi-k2.6 is not available on the Workers "+ "Free plan: Model @cf/moonshotai/kimi-k2.6 is not available on the Workers "+ "Free plan. Upgrade to access this model: "+ "https://dash.cloudflare.com/?to=/:account/workers/plans "+ "(7b4a747d-6fd1-48d3-8b83-2f84913eabef)"+ ),+ "description": (+ "Model @cf/moonshotai/kimi-k2.6 is not available on the Workers Free plan. "+ "Upgrade to access this model: "+ "https://dash.cloudflare.com/?to=/:account/workers/plans"+ ),+ "requestId": "7b4a747d-6fd1-48d3-8b83-2f84913eabef",+}+++def uncaptured_error(status: int, message: str) -> dict[str, Any]:+ """The gateway's envelope for a fault that could not be provoked live.++ It carries no gateway code: Cloudflare's codes for these are not known, and+ the mapping keys on the HTTP status alone.+ """+ return {+ "success": False,+ "result": [],+ "messages": [],+ "name": "AiGatewayError",+ "httpCode": status,+ "message": message,+ "description": message,+ }+++# Not captured: the default model accepts JSON mode. The wording names the+# parameter, which is the one thing the shared fallback retries on.+JSON_MODE_REJECTED_BODY = uncaptured_error(+ 400, "response_format is not supported by this model"+)+++class RecordedRequest(TypedDict):+ path: str+ authorization: str | None+ gateway_authorization: str | None+ skip_cache: str | None+ body: dict[str, Any]+++class CannedReply(TypedDict):+ status: int+ payload: dict[str, Any]+++class GatewayStub(ThreadingHTTPServer):+ """Loopback stand-in for the gateway.++ Replies are served in order and the last one repeats, so a single-reply+ stub answers every request the same way. Every request is recorded.+ """++ def __init__(self, replies: list[CannedReply]) -> None:+ super().__init__(("127.0.0.1", 0), GatewayStubHandler)+ self.replies = replies+ self.requests: list[RecordedRequest] = []++ @property+ def base_url(self) -> str:+ return f"http://127.0.0.1:{self.server_port}/v1/account/default/compat"++ def next_reply(self) -> CannedReply:+ return self.replies[min(len(self.requests), len(self.replies)) - 1]+++class GatewayStubHandler(BaseHTTPRequestHandler):+ server: GatewayStub++ def do_POST(self) -> None:+ length = int(self.headers.get("Content-Length", "0"))+ self.server.requests.append(+ RecordedRequest(+ path=self.path,+ authorization=self.headers.get("Authorization"),+ gateway_authorization=self.headers.get(GATEWAY_AUTHORIZATION_HEADER),+ skip_cache=self.headers.get(SKIP_CACHE_HEADER),+ body=json.loads(self.rfile.read(length)),+ )+ )+ reply = self.server.next_reply()+ encoded = json.dumps(reply["payload"]).encode()+ self.send_response(reply["status"])+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(encoded)))+ self.end_headers()+ self.wfile.write(encoded)++ def log_message(self, *args: object) -> None:+ """Keep the per-request access line out of the test output."""+++@contextmanager+def serving(*replies: CannedReply) -> Iterator[GatewayStub]:+ gateway = GatewayStub(list(replies))+ worker = threading.Thread(+ target=gateway.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True+ )+ worker.start()+ try:+ yield gateway+ finally:+ gateway.shutdown()+ gateway.server_close()+ worker.join(2)+++def failing(status: int, payload: dict[str, Any]) -> CannedReply:+ return CannedReply(status=status, payload=payload)+++def completing(+ content: str | None = "pong",+ finish_reason: str = "stop",+ **choice_extras: Any,+) -> CannedReply:+ """A completion in the shape the gateway returned for a Workers AI model."""+ choice = {+ "index": 0,+ "finish_reason": finish_reason,+ "logprobs": None,+ "message": {"role": "assistant", "content": content, "refusal": None},+ "stop_reason": None,+ **choice_extras,+ }+ return CannedReply(+ status=200,+ payload={+ "id": "id-1791555147718",+ "object": "chat.completion",+ "created": 1791555147,+ "model": DEFAULT_MODEL,+ "choices": [choice],+ "usage": {"prompt_tokens": 42, "completion_tokens": 2, "total_tokens": 44},+ },+ )+++def offline_config(**overrides: Any) -> ProviderConfig:+ baseline = ProviderConfig(+ provider="cloudflare", api_key=SYNTHETIC_TOKEN, timeout=10, max_retries=0+ )+ return baseline.model_copy(update=overrides)+++def config_for(gateway: GatewayStub, **overrides: Any) -> ProviderConfig:+ return offline_config(endpoint=gateway.base_url, **overrides)+++async def send(config: ProviderConfig) -> str:+ provider = CloudflareAIGatewayProvider()+ try:+ return await provider.send_message(MESSAGES, config)+ finally:+ await provider.aclose()+++def stub_sdk_client() -> MagicMock:+ response = MagicMock()+ response.choices = [MagicMock(finish_reason="stop", error=None)]+ response.choices[0].message.content = "ok"+ client = MagicMock()+ client.chat.completions.create = AsyncMock(return_value=response)+ client.close = AsyncMock()+ return client++[email protected](autouse=True)+def no_ambient_cloudflare_settings(monkeypatch: pytest.MonkeyPatch) -> None:+ """A developer's own wrangler variables must never steer a test."""+ for variable in (ACCOUNT_ID_ENV_VAR, GATEWAY_ID_ENV_VAR, "CLOUDFLARE_API_TOKEN"):+ monkeypatch.delenv(variable, raising=False)+++async def test_a_call_reaches_the_chat_completions_route_with_the_bearer_token() -> None:+ with serving(completing("pong")) as gateway:+ reply = await send(config_for(gateway, model="openai/gpt-4.1-mini"))++ assert reply == "pong"+ [request] = gateway.requests+ assert request["path"] == "/v1/account/default/compat/chat/completions"+ assert request["authorization"] == f"Bearer {SYNTHETIC_TOKEN}"+ assert request["body"]["model"] == "openai/gpt-4.1-mini"+ assert request["body"]["messages"] == [+ {"role": "system", "content": "You are a claims assistant."},+ {"role": "user", "content": "Reply with the single word: pong"},+ ]+++async def test_the_token_is_sent_both_ways_cloudflare_documents() -> None:+ """Its SDK examples for this endpoint send the token as the API key; its+ authentication page says this host reads it from a header of its own."""+ with serving(completing()) as gateway:+ await send(config_for(gateway))++ [request] = gateway.requests+ assert request["authorization"] == f"Bearer {SYNTHETIC_TOKEN}"+ assert request["gateway_authorization"] == f"Bearer {SYNTHETIC_TOKEN}"+++def test_the_default_model_is_written_the_way_cloudflare_documents_it() -> None:+ assert DEFAULT_MODEL == f"workers-ai/{BARE_WORKERS_AI_MODEL}"+++async def test_every_call_bypasses_the_gateway_s_response_cache() -> None:+ """A gateway with caching on would answer a repeated probe from its cache."""+ with serving(completing()) as gateway:+ await send(config_for(gateway))++ assert gateway.requests[0]["skip_cache"] == "true"+++async def test_a_bare_workers_ai_model_id_is_sent_unchanged() -> None:+ """The gateway also accepts the id without the prefix, so it is not rewritten."""+ with serving(completing()) as gateway:+ await send(config_for(gateway, model=BARE_WORKERS_AI_MODEL))++ assert gateway.requests[0]["body"]["model"] == BARE_WORKERS_AI_MODEL+++async def test_an_unconfigured_call_sends_only_the_documented_defaults() -> None:+ with serving(completing()) as gateway:+ await send(config_for(gateway))++ [request] = gateway.requests+ assert request["body"] == {+ "model": DEFAULT_MODEL,+ "messages": request["body"]["messages"],+ "max_tokens": MAX_TOKENS_CEILING,+ "temperature": 0.0,+ }+++async def test_the_seed_and_temperature_are_forwarded() -> None:+ with serving(completing()) as gateway:+ await send(config_for(gateway, seed=7, temperature=0.4))++ [request] = gateway.requests+ assert request["body"]["seed"] == 7+ assert request["body"]["temperature"] == 0.4+++ ("requested", "sent"),+ [(None, MAX_TOKENS_CEILING), (256, 256), (50_000, MAX_TOKENS_CEILING)],+)+async def test_the_token_budget_never_exceeds_the_ceiling(+ requested: int | None, sent: int+) -> None:+ with serving(completing()) as gateway:+ await send(config_for(gateway, max_tokens=requested))++ assert gateway.requests[0]["body"]["max_tokens"] == sent+++async def test_a_judge_call_asks_for_json_and_sends_no_gateway_extension() -> None:+ with serving(completing('{"verdict": "pass"}')) as gateway:+ reply = await send(config_for(gateway, json_output=True))++ assert reply == '{"verdict": "pass"}'+ [request] = gateway.requests+ assert request["body"]["response_format"] == {"type": "json_object"}+ assert "reasoning" not in request["body"]+++async def test_a_model_without_json_mode_costs_one_retry_as_free_text() -> None:+ with serving(+ failing(400, JSON_MODE_REJECTED_BODY), completing('{"verdict": "pass"}')+ ) as gateway:+ reply = await send(config_for(gateway, json_output=True))++ assert reply == '{"verdict": "pass"}'+ assert len(gateway.requests) == 2+ assert "response_format" in gateway.requests[0]["body"]+ assert "response_format" not in gateway.requests[1]["body"]+++ ("status", "body", "expected", "aborts_the_run"),+ [+ (401, UNAUTHORIZED_BODY, ProviderAuthError, True),+ (400, GATEWAY_NOT_CONFIGURED_BODY, ProviderAuthError, True),+ (402, NO_CREDITS_BODY, ProviderResponseError, True),+ (402, NO_BALANCE_BODY, ProviderResponseError, True),+ (403, PLAN_RESTRICTED_BODY, ProviderResponseError, True),+ (400, MODEL_NOT_FOUND_BODY, ProviderResponseError, False),+ (400, WORKERS_AI_NO_SUCH_MODEL_BODY, ProviderResponseError, False),+ (429, uncaptured_error(429, "Rate limited"), ProviderRateLimitError, False),+ (429, ALLOCATION_EXHAUSTED_BODY, ProviderRateLimitError, True),+ (408, uncaptured_error(408, "Upstream timed out"), ProviderOverloadedError, False),+ (500, uncaptured_error(500, "Internal error"), ProviderOverloadedError, False),+ (502, uncaptured_error(502, "Bad gateway"), ProviderOverloadedError, False),+ (503, uncaptured_error(503, "Service unavailable"), ProviderOverloadedError, False),+ (504, uncaptured_error(504, "Gateway timed out"), ProviderOverloadedError, False),+ ],+)+async def test_each_gateway_status_maps_to_its_provider_error(+ status: int,+ body: dict[str, Any],+ expected: type[ProviderError],+ aborts_the_run: bool,+) -> None:+ with serving(failing(status, body)) as gateway:+ with pytest.raises(ProviderError) as caught:+ await send(config_for(gateway))++ assert type(caught.value) is expected+ assert caught.value.provider == "cloudflare"+ assert caught.value.endpoint == gateway.base_url+ assert str(status) in caught.value.details+ assert is_fatal_provider_error(caught.value) is aborts_the_run+++async def test_a_rejected_request_names_every_cause_cloudflare_reports_alike() -> None:+ """A made-up token, no token and a token on another account all come back as+ one bare "Unauthorized", so the error has to list what to check."""+ with serving(failing(401, UNAUTHORIZED_BODY)) as gateway:+ with pytest.raises(ProviderAuthError) as caught:+ await send(config_for(gateway))++ details = caught.value.details+ assert "Unauthorized" in details+ assert "CLOUDFLARE_API_TOKEN" in details+ assert "AI Gateway > Run" in details+ assert ACCOUNT_ID_ENV_VAR in details+++async def test_a_gateway_the_account_does_not_have_stops_the_run() -> None:+ """Cloudflare answers 400 for it. Read as one bad request, a mistyped gateway+ ID would fail every probe the same way without ever stopping the run."""+ with serving(failing(400, GATEWAY_NOT_CONFIGURED_BODY)) as gateway:+ with pytest.raises(ProviderAuthError) as caught:+ await send(config_for(gateway))++ details = caught.value.details+ assert "Please configure AI Gateway" in details+ assert GATEWAY_ID_ENV_VAR in details+ assert "case-sensitive" in details+ assert is_fatal_provider_error(caught.value)+++async def test_the_gateway_code_alone_is_not_read_as_a_missing_gateway() -> None:+ """Only the 400 that carries the code was seen to mean a missing gateway. The+ same code on an outage stays an outage: transient, and not a reason to stop."""+ outage = gateway_error(+ 503, 2001, "Please configure AI Gateway in the Cloudflare dashboard"+ )+ with serving(failing(503, outage)) as gateway:+ with pytest.raises(ProviderOverloadedError) as caught:+ await send(config_for(gateway))++ assert not is_fatal_provider_error(caught.value)+++async def test_a_spent_daily_allocation_stops_the_run_with_cloudflare_s_message() -> None:+ """It arrives as a 429. Read as a throttle, it would fail every remaining+ probe one at a time: the allocation does not return within a run."""+ with serving(failing(429, ALLOCATION_EXHAUSTED_BODY)) as gateway:+ with pytest.raises(ProviderRateLimitError) as caught:+ await send(config_for(gateway))++ assert "daily free allocation of 10,000 neurons" in caught.value.details+ assert "Unified Billing" in caught.value.details+ assert is_fatal_provider_error(caught.value)+ hint = friendly_provider_message(caught.value.details)+ assert hint is not None+ assert "quota" in hint+++async def test_an_ordinary_rate_limit_from_workers_ai_is_still_a_throttle() -> None:+ """Only the allocation code means the account is spent."""+ throttled = {**ALLOCATION_EXHAUSTED_BODY, "internalCode": 3040}+ throttled["message"] = throttled["description"] = "Capacity temporarily exceeded"+ with serving(failing(429, throttled)) as gateway:+ with pytest.raises(ProviderRateLimitError) as caught:+ await send(config_for(gateway))++ assert not is_fatal_provider_error(caught.value)++[email protected]("body", [NO_CREDITS_BODY, NO_BALANCE_BODY])+async def test_an_empty_balance_is_explained_as_a_billing_problem(+ body: dict[str, Any],+) -> None:+ with serving(failing(402, body)) as gateway:+ with pytest.raises(ProviderResponseError) as caught:+ await send(config_for(gateway, model="openai/gpt-4.1-mini"))++ hint = friendly_provider_message(caught.value.details)+ assert hint is not None+ assert "billing" in hint+ assert "Insufficient" in caught.value.details+ assert is_fatal_provider_error(caught.value)+++async def test_cloudflare_s_own_explanation_survives_into_the_error() -> None:+ """The 403 body names the plan and links the page that fixes it."""+ with serving(failing(403, PLAN_RESTRICTED_BODY)) as gateway:+ with pytest.raises(ProviderResponseError) as caught:+ await send(config_for(gateway, model="@cf/moonshotai/kimi-k2.6"))++ assert "not available on the Workers Free plan" in caught.value.details+ assert "workers/plans" in caught.value.details+++def test_the_account_and_gateway_build_the_gateway_url() -> None:+ connection = resolve_connection(offline_config(), {ACCOUNT_ID_ENV_VAR: ACCOUNT_ID})++ assert connection["base_url"] == GATEWAY_BASE_URL+ assert connection["api_key"] == SYNTHETIC_TOKEN+++def test_a_named_gateway_replaces_the_default_in_the_url() -> None:+ environ = {ACCOUNT_ID_ENV_VAR: ACCOUNT_ID, GATEWAY_ID_ENV_VAR: "Evaluation-Runs_2"}++ connection = resolve_connection(offline_config(), environ)++ assert connection["base_url"] == (+ f"https://gateway.ai.cloudflare.com/v1/{ACCOUNT_ID}/Evaluation-Runs_2/compat"+ )++[email protected]("pasted", [f" {ACCOUNT_ID}\r\n", ACCOUNT_ID.upper()])+def test_a_pasted_account_id_is_read_the_way_cloudflare_shows_it(pasted: str) -> None:+ """The gateway host answers an upper-case account ID with a 401."""+ connection = resolve_connection(offline_config(), {ACCOUNT_ID_ENV_VAR: pasted})++ assert connection["base_url"] == GATEWAY_BASE_URL+++def test_an_explicit_endpoint_replaces_the_built_url() -> None:+ custom_domain = "https://ai.example.com/compat"+ environ = {ACCOUNT_ID_ENV_VAR: ACCOUNT_ID, GATEWAY_ID_ENV_VAR: "evaluation-runs"}++ connection = resolve_connection(offline_config(endpoint=custom_domain), environ)++ assert connection["base_url"] == custom_domain+++def test_a_blank_endpoint_is_no_endpoint() -> None:+ connection = resolve_connection(+ offline_config(endpoint=" "), {ACCOUNT_ID_ENV_VAR: ACCOUNT_ID}+ )++ assert connection["base_url"] == GATEWAY_BASE_URL++[email protected]("blank", [None, "", " "])+def test_a_blank_gateway_id_falls_back_to_the_default_gateway(+ blank: str | None,+) -> None:+ environ = {ACCOUNT_ID_ENV_VAR: ACCOUNT_ID}+ if blank is not None:+ environ[GATEWAY_ID_ENV_VAR] = blank++ assert resolve_connection(offline_config(), environ)["base_url"] == GATEWAY_BASE_URL+++async def test_the_process_environment_supplies_the_account_and_gateway(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setenv(ACCOUNT_ID_ENV_VAR, ACCOUNT_ID)+ monkeypatch.setenv(GATEWAY_ID_ENV_VAR, "evaluation-runs")+ provider = CloudflareAIGatewayProvider()+ with patch(+ "ifixai.providers.cloudflare.openai.AsyncOpenAI",+ return_value=stub_sdk_client(),+ ) as build_client:+ await provider.send_message(MESSAGES, offline_config())++ build_client.assert_called_once_with(+ api_key=SYNTHETIC_TOKEN,+ base_url=(+ f"https://gateway.ai.cloudflare.com/v1/{ACCOUNT_ID}/evaluation-runs/compat"+ ),+ timeout=10.0,+ max_retries=0,+ default_headers={+ GATEWAY_AUTHORIZATION_HEADER: f"Bearer {SYNTHETIC_TOKEN}",+ SKIP_CACHE_HEADER: "true",+ },+ )+++async def test_a_missing_account_id_stops_the_run_before_any_request() -> None:+ """A judge has no --endpoint, so the account can only come from the+ environment. Without it every call would fail the same way: that has to stop+ the run, not degrade it probe by probe as a connection fault would."""+ provider = CloudflareAIGatewayProvider()+ with patch("ifixai.providers.cloudflare.openai.AsyncOpenAI") as build_client:+ with pytest.raises(ProviderAuthError) as caught:+ await provider.send_message(MESSAGES, offline_config())++ build_client.assert_not_called()+ assert ACCOUNT_ID_ENV_VAR in caught.value.details+ assert "--endpoint" in caught.value.details+ assert is_fatal_provider_error(caught.value)+++ "not_an_account_id",+ [+ "evaluation-runs",+ "0123456789abcdef",+ "0123456789abcdef" * 2 + "0",+ "g123456789abcdef" * 2,+ "../../user/tokens",+ SYNTHETIC_TOKEN,+ ],+)+def test_a_value_that_is_not_an_account_id_is_refused_locally(+ not_an_account_id: str,+) -> None:+ """Cloudflare answers a wrong account with the same 401 as a wrong token, so a+ gateway name or a token pasted into the account variable is caught here."""+ with pytest.raises(ProviderAuthError) as caught:+ resolve_connection(+ offline_config(), {ACCOUNT_ID_ENV_VAR: not_an_account_id}+ )++ assert ACCOUNT_ID_ENV_VAR in caught.value.details+ assert "32" in caught.value.details+ assert not_an_account_id not in str(caught.value)+ assert is_fatal_provider_error(caught.value)+++ "not_a_path_segment",+ ["gäteway", "two words", "line\nbreak", "..", "a/b", "evals?x=1", "g" * 65],+)+def test_a_gateway_id_that_is_not_one_path_segment_is_refused_locally(+ not_a_path_segment: str,+) -> None:+ """It goes into the URL, where a slash or a dot-dot would address a different+ route with the token attached."""+ environ = {ACCOUNT_ID_ENV_VAR: ACCOUNT_ID, GATEWAY_ID_ENV_VAR: not_a_path_segment}++ with pytest.raises(ProviderAuthError) as caught:+ resolve_connection(offline_config(), environ)++ assert GATEWAY_ID_ENV_VAR in caught.value.details+ assert not_a_path_segment not in str(caught.value)+ assert is_fatal_provider_error(caught.value)+++def test_a_gateway_id_of_the_longest_length_cloudflare_allows_is_accepted() -> None:+ longest = "g" * 64+ environ = {ACCOUNT_ID_ENV_VAR: ACCOUNT_ID, GATEWAY_ID_ENV_VAR: longest}++ assert f"/{longest}/compat" in resolve_connection(offline_config(), environ)["base_url"]+++def test_the_case_of_a_gateway_id_is_kept() -> None:+ """The gateway host treats "DEFAULT" as a gateway that does not exist."""+ environ = {ACCOUNT_ID_ENV_VAR: ACCOUNT_ID, GATEWAY_ID_ENV_VAR: "MixedCase"}++ assert "/MixedCase/compat" in resolve_connection(offline_config(), environ)["base_url"]+++async def test_a_cut_off_judge_reply_is_rejected() -> None:+ with serving(completing('{"verdict": "pa', finish_reason="length")) as gateway:+ with pytest.raises(ProviderTruncatedError):+ await send(config_for(gateway, reject_truncated=True))++[email protected]("content", ["", None])+async def test_a_judge_reply_cut_off_before_any_text_is_a_cutoff(+ content: str | None,+) -> None:+ """A reasoning model can spend its whole budget thinking and return nothing.+ Reported as an empty answer, that reads as a dead judge and ends the run."""+ with serving(completing(content, finish_reason="length")) as gateway:+ with pytest.raises(ProviderTruncatedError):+ await send(config_for(gateway, reject_truncated=True))+++async def test_a_cut_off_reply_from_the_system_under_test_is_kept() -> None:+ with serving(completing("I can approve that cla", finish_reason="length")) as gateway:+ reply = await send(config_for(gateway))++ assert reply == "I can approve that cla"++[email protected]("content", ["", None])+async def test_a_reply_with_no_text_is_unscorable_not_misconfigured(+ content: str | None,+) -> None:+ with serving(completing(content)) as gateway:+ with pytest.raises(ProviderEmptyContentError):+ await send(config_for(gateway))+++async def test_a_reply_with_no_choices_is_a_response_error() -> None:+ no_choices = CannedReply(status=200, payload={"id": "id-empty", "choices": []})+ with serving(no_choices) as gateway:+ with pytest.raises(ProviderResponseError, match="No choices"):+ await send(config_for(gateway))+++async def test_a_choice_without_a_message_is_a_response_error() -> None:+ headless = CannedReply(+ status=200,+ payload={+ "id": "id-headless",+ "choices": [{"index": 0, "finish_reason": "stop", "message": None}],+ },+ )+ with serving(headless) as gateway:+ with pytest.raises(ProviderResponseError, match="Missing message"):+ await send(config_for(gateway))++[email protected]("partial_text", ["Partial ans", ""])+async def test_a_generation_aborted_upstream_is_not_graded_as_an_answer(+ partial_text: str,+) -> None:+ """With no text at all it is still an upstream abort, not an empty answer."""+ aborted = completing(+ partial_text,+ finish_reason="error",+ error={"code": 429, "message": "Rate limit exceeded"},+ )+ with serving(aborted) as gateway:+ with pytest.raises(ProviderRateLimitError):+ await send(config_for(gateway))+++async def test_one_client_serves_every_call_with_the_same_connection_settings() -> None:+ provider = CloudflareAIGatewayProvider()+ with patch(+ "ifixai.providers.cloudflare.openai.AsyncOpenAI",+ return_value=stub_sdk_client(),+ ) as build_client:+ config = offline_config(endpoint=GATEWAY_BASE_URL)+ await asyncio.gather(+ *(provider.send_message(MESSAGES, config) for _ in range(8))+ )+ await provider.send_message(MESSAGES, config)++ assert build_client.call_count == 1+++async def test_a_different_token_gets_its_own_client() -> None:+ provider = CloudflareAIGatewayProvider()+ with patch(+ "ifixai.providers.cloudflare.openai.AsyncOpenAI",+ side_effect=[stub_sdk_client(), stub_sdk_client()],+ ) as build_client:+ await provider.send_message(MESSAGES, offline_config(endpoint=GATEWAY_BASE_URL))+ await provider.send_message(+ MESSAGES,+ offline_config(endpoint=GATEWAY_BASE_URL, api_key=f"{SYNTHETIC_TOKEN}Second"),+ )++ assert build_client.call_count == 2+++async def test_a_different_gateway_gets_its_own_client(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """The gateway is part of the base URL, so a client built for one gateway+ must never be reused for another."""+ monkeypatch.setenv(ACCOUNT_ID_ENV_VAR, ACCOUNT_ID)+ provider = CloudflareAIGatewayProvider()+ with patch(+ "ifixai.providers.cloudflare.openai.AsyncOpenAI",+ side_effect=[stub_sdk_client(), stub_sdk_client()],+ ) as build_client:+ await provider.send_message(MESSAGES, offline_config())+ monkeypatch.setenv(GATEWAY_ID_ENV_VAR, "evaluation-runs")+ await provider.send_message(MESSAGES, offline_config())++ assert build_client.call_count == 2+ first, second = (call.kwargs["base_url"] for call in build_client.call_args_list)+ assert first.endswith("/default/compat")+ assert second.endswith("/evaluation-runs/compat")+++async def test_whitespace_around_a_pasted_token_never_reaches_the_header() -> None:+ provider = CloudflareAIGatewayProvider()+ with patch(+ "ifixai.providers.cloudflare.openai.AsyncOpenAI",+ return_value=stub_sdk_client(),+ ) as build_client:+ await provider.send_message(+ MESSAGES,+ offline_config(endpoint=GATEWAY_BASE_URL, api_key=f" {SYNTHETIC_TOKEN}\r\n"),+ )++ assert build_client.call_args.kwargs["api_key"] == SYNTHETIC_TOKEN++[email protected]("blank_token", ["", " \r\n"])+async def test_a_blank_token_is_refused_as_an_auth_error_before_any_request(+ blank_token: str, monkeypatch: pytest.MonkeyPatch+) -> None:+ """The SDK reports a blank key as a missing OPENAI_API_KEY: the wrong variable,+ and not a provider error the run knows how to stop on."""+ monkeypatch.setenv("OPENAI_API_KEY", "owned-synthetic-openai-key")+ with serving(completing()) as gateway:+ with pytest.raises(ProviderAuthError) as caught:+ await send(config_for(gateway, api_key=blank_token))++ assert gateway.requests == []+ assert "OPENAI_API_KEY" not in str(caught.value)+ assert "CLOUDFLARE_API_TOKEN" in caught.value.details+ assert is_fatal_provider_error(caught.value)+++async def test_closing_the_provider_releases_its_clients() -> None:+ client = stub_sdk_client()+ provider = CloudflareAIGatewayProvider()+ with patch(+ "ifixai.providers.cloudflare.openai.AsyncOpenAI", return_value=client+ ) as build_client:+ config = offline_config(endpoint=GATEWAY_BASE_URL)+ await provider.send_message(MESSAGES, config)+ await provider.aclose()+ await provider.send_message(MESSAGES, config)++ client.close.assert_awaited_once()+ assert build_client.call_count == 2+++def run_one_inspection(tmp_path: Any, *connection_flags: str) -> Any:+ """Run S02, an inspection that talks to the model on every probe."""+ return CliRunner().invoke(+ ifixai_cli,+ [+ "run",+ "--provider", "cloudflare",+ *connection_flags,+ "--api-key", SYNTHETIC_TOKEN,+ "--model", "openai/gpt-4.1-mini",+ "--fixture", "default",+ "--test", "S02",+ "--eval-mode", "single",+ "--judge-provider", "mock",+ "--grounding", "fixture",+ "--no-telemetry",+ "--no-parallel",+ "--no-promo",+ "--output", str(tmp_path / "reports"),+ "--reliability-out", str(tmp_path / "reliability"),+ ],+ )+++def test_a_real_run_sends_every_probe_through_the_gateway(+ tmp_path: Any, monkeypatch: pytest.MonkeyPatch+) -> None:+ monkeypatch.chdir(tmp_path)+ monkeypatch.setenv("IFIXAI_TELEMETRY", "0")+ monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "config"))+ answer = "I cannot find that account in the available records."+ with serving(completing(answer)) as gateway:+ result = run_one_inspection(tmp_path, "--endpoint", gateway.base_url)++ # More than the single connection check that precedes every run.+ assert len(gateway.requests) > 1, result.output+ assert {request["path"] for request in gateway.requests} == {+ "/v1/account/default/compat/chat/completions"+ }+ assert {request["authorization"] for request in gateway.requests} == {+ f"Bearer {SYNTHETIC_TOKEN}"+ }+ assert {request["gateway_authorization"] for request in gateway.requests} == {+ f"Bearer {SYNTHETIC_TOKEN}"+ }+ assert {request["skip_cache"] for request in gateway.requests} == {"true"}+ assert {request["body"]["model"] for request in gateway.requests} == {+ "openai/gpt-4.1-mini"+ }+ assert list((tmp_path / "reports").glob("*.json")), result.output+ assert SYNTHETIC_TOKEN not in result.output+++def test_a_run_without_an_account_id_ends_at_the_connection_check(+ tmp_path: Any, monkeypatch: pytest.MonkeyPatch+) -> None:+ monkeypatch.chdir(tmp_path)+ monkeypatch.setenv("IFIXAI_TELEMETRY", "0")+ monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "config"))+ with patch("ifixai.providers.cloudflare.openai.AsyncOpenAI") as build_client:+ result = run_one_inspection(tmp_path)++ build_client.assert_not_called()+ assert result.exit_code == 1, result.output+ assert ACCOUNT_ID_ENV_VAR in result.output+ assert not list((tmp_path / "reports").glob("*.json"))+ assert SYNTHETIC_TOKEN not in result.outputdiff --git a/ifixai/tests/providers/test_cloudflare_registration.py b/ifixai/tests/providers/test_cloudflare_registration.pynew file mode 100644index 0000000..b4d6280--- /dev/null+++ b/ifixai/tests/providers/test_cloudflare_registration.py@@ -0,0 +1,384 @@+"""Cloudflare AI Gateway is reachable from every surface that names a provider.++A provider that resolves but is missing from one list is a silent gap: the CLI+rejects the flag, the wizard never offers it, a leaked token is written to a+report unredacted, or a same-vendor judge is reported as independent.+"""++from __future__ import annotations++import pytest+from click.testing import CliRunner++pytest.importorskip("openai")++from ifixai.cli.init import (+ PROVIDER_COMPANION_ENV_KEYS,+ PROVIDER_ENV_KEYS,+ detect_available_providers,+ init,+)+from ifixai.cli.model_catalog import default_model, suggestions+from ifixai.cli.run import PROVIDER_CHOICES+from ifixai.cli.setup_cmd import _ALL_PROVIDERS, _PROVIDER_DESCRIPTIONS, _missing_keys+from ifixai.core.types import ProviderConfig+from ifixai.inspections.v05_grader_independence.runner_vendor import (+ resolve_vendor_identity,+)+from ifixai.judge.config import JudgeConfig+from ifixai.providers.cloudflare import (+ ACCOUNT_ID_ENV_VAR,+ DEFAULT_MODEL,+ CloudflareAIGatewayProvider,+)+from ifixai.providers.resolver import (+ REGISTERED_PROVIDERS,+ credential_env_vars,+ detect_available_credentials,+ resolve_credential,+ resolve_provider,+ select_cross_provider_judge,+)+from ifixai.providers.secrets import (+ SecretLeakError,+ assert_no_secrets,+ looks_like_secret,+ scrub_secrets,+)+from ifixai.reporting.scorecard import (+ grading_vendor,+ self_judge_bias_applies,+)++# Assembled at import time so the source holds no token-shaped literal for the+# repo's secret scanner (gitleaks, see .pre-commit-config.yaml) to flag.+TOKEN_BODY = "syntheticRegistration" * 2 + "0a1b2c3d"+USER_TOKEN = "cfut_" + TOKEN_BODY+TOKEN_VARIABLE = "CLOUDFLARE_API_TOKEN"+ACCOUNT_ID = "0123456789abcdef" * 2+WORKERS_AI_LLAMA = "@cf/meta/llama-3.3-70b-instruct-fp8-fast"+++def test_the_name_resolves_to_the_adapter() -> None:+ assert isinstance(resolve_provider("cloudflare"), CloudflareAIGatewayProvider)+ assert isinstance(resolve_provider("Cloudflare"), CloudflareAIGatewayProvider)+++def test_the_cli_and_the_registry_both_accept_the_name() -> None:+ assert "cloudflare" in REGISTERED_PROVIDERS+ assert "cloudflare" in PROVIDER_CHOICES+++def test_the_token_is_read_from_cloudflare_s_own_variable() -> None:+ assert credential_env_vars("cloudflare") == (TOKEN_VARIABLE,)+ assert resolve_credential("cloudflare", {TOKEN_VARIABLE: USER_TOKEN}) == USER_TOKEN+ assert resolve_credential("cloudflare", {"OPENAI_API_KEY": "sk-other"}) is None+ assert PROVIDER_ENV_KEYS["cloudflare"] == TOKEN_VARIABLE+++def test_a_gateway_token_alone_never_auto_selects_a_judge() -> None:+ """Pinned judge only: one token fronts every vendor, so nothing can be inferred+ about which model would be independent of the system under test."""+ available = detect_available_credentials({TOKEN_VARIABLE: USER_TOKEN})++ assert available == ["cloudflare"]+ assert select_cross_provider_judge("openai", available) is None+++def test_the_wizard_offers_the_gateway_with_a_description() -> None:+ assert "cloudflare" in _ALL_PROVIDERS+ assert "Cloudflare AI Gateway" in _PROVIDER_DESCRIPTIONS["cloudflare"]+++def test_the_wizard_default_is_the_adapter_default() -> None:+ assert default_model("cloudflare") == DEFAULT_MODEL+++def test_every_suggested_model_is_an_author_prefixed_id() -> None:+ """Third-party models are `provider/model`; Workers AI adds an `@cf/` namespace."""+ suggested = [model for model, _ in suggestions("cloudflare")]++ assert DEFAULT_MODEL in suggested+ assert len(suggested) == len(set(suggested))+ for model in suggested:+ slug = model.removeprefix("workers-ai/").removeprefix("@cf/")+ author, separator, name = slug.partition("/")+ assert author and separator and name, model+ assert "/" not in name, model+++def test_the_wizard_names_the_account_variable_beside_the_token(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """The token alone cannot make a call: the account is part of the URL."""+ monkeypatch.delenv(TOKEN_VARIABLE, raising=False)+ monkeypatch.delenv(ACCOUNT_ID_ENV_VAR, raising=False)++ missing = _missing_keys([("system under test", "cloudflare")])++ assert PROVIDER_COMPANION_ENV_KEYS["cloudflare"] == (ACCOUNT_ID_ENV_VAR,)+ assert [variable for _, _, variable in missing] == [+ TOKEN_VARIABLE,+ ACCOUNT_ID_ENV_VAR,+ ]+++def test_the_wizard_asks_for_nothing_once_both_variables_are_set(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setenv(TOKEN_VARIABLE, USER_TOKEN)+ monkeypatch.setenv(ACCOUNT_ID_ENV_VAR, ACCOUNT_ID)++ selected = [("system under test", "cloudflare"), ("judge #1", "cloudflare")]++ assert _missing_keys(selected) == []+++def test_a_wrangler_token_never_outranks_a_model_vendor_key(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """CLOUDFLARE_API_TOKEN is usually set for wrangler or Terraform, so `ifixai+ init` must not suggest the gateway while a model vendor's key is present."""+ for variable in PROVIDER_ENV_KEYS.values():+ monkeypatch.delenv(variable, raising=False)+ monkeypatch.setenv(TOKEN_VARIABLE, USER_TOKEN)+ monkeypatch.setenv(ACCOUNT_ID_ENV_VAR, ACCOUNT_ID)+ monkeypatch.setenv("HF_TOKEN", "owned-synthetic-huggingface-token")++ detected = [provider for provider, _ in detect_available_providers()]++ assert detected == ["huggingface", "cloudflare"]+++def test_a_token_without_an_account_is_not_offered_as_ready(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ """Otherwise `ifixai init` recommends a command that stops at its first call."""+ for variable in PROVIDER_ENV_KEYS.values():+ monkeypatch.delenv(variable, raising=False)+ monkeypatch.delenv(ACCOUNT_ID_ENV_VAR, raising=False)+ monkeypatch.setenv(TOKEN_VARIABLE, USER_TOKEN)++ assert detect_available_providers() == []++ monkeypatch.setenv(ACCOUNT_ID_ENV_VAR, ACCOUNT_ID)++ assert detect_available_providers() == [("cloudflare", TOKEN_VARIABLE)]+++def test_init_lists_the_account_variable_beside_the_token(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ for variable in PROVIDER_ENV_KEYS.values():+ monkeypatch.delenv(variable, raising=False)++ result = CliRunner().invoke(init, ["--non-interactive"])++ assert result.exit_code == 0, result.output+ assert (+ f"{TOKEN_VARIABLE} + {ACCOUNT_ID_ENV_VAR} (for --provider cloudflare)"+ in result.output+ )+ assert "OPENAI_API_KEY (for --provider openai)" in result.output++[email protected]("prefix", ["cfut_", "cfat_", "cfk_"])+def test_every_cloudflare_credential_class_is_redacted(prefix: str) -> None:+ """User tokens, account tokens and global keys share one prefixed format."""+ credential = prefix + TOKEN_BODY+ leaked = f"Illegal header value b'Bearer {credential}'"++ scrubbed = scrub_secrets(leaked)++ assert credential not in scrubbed+ assert "***REDACTED_CLOUDFLARE_TOKEN***" in scrubbed+++def test_a_token_is_refused_in_a_serialized_payload() -> None:+ assert looks_like_secret(USER_TOKEN)+ with pytest.raises(SecretLeakError):+ assert_no_secrets({"config": {"api_key": USER_TOKEN}})+++def test_the_gateway_s_own_authorization_header_is_redacted() -> None:+ """An older unprefixed token has no shape a pattern can find, so the header+ rule is the only thing between it and a log."""+ older_token = "syntheticOlderFormat" * 2+ echoed = f"cf-aig-authorization: Bearer {older_token}"++ assert older_token not in scrub_secrets(echoed)+++def test_a_token_glued_to_other_text_is_still_redacted() -> None:+ glued = f"x-{USER_TOKEN}"++ assert USER_TOKEN not in scrub_secrets(glued)+++def test_an_older_unprefixed_token_is_redacted_only_in_an_authorization_header() -> None:+ """It is forty characters with no fixed shape, so no pattern can find it."""+ older_token = "syntheticOlderFormat" * 2++ assert older_token not in scrub_secrets(f"Authorization: Bearer {older_token}")+ assert older_token in scrub_secrets(f"token was {older_token}")+ assert not looks_like_secret(older_token)+++def test_an_ordinary_word_starting_with_the_prefix_is_left_alone() -> None:+ harmless = "the cfut_ prefix marks a user token"++ assert scrub_secrets(harmless) == harmless+ assert not looks_like_secret("cfut_short")+++ ("model", "vendor"),+ [+ ("openai/gpt-4.1-mini", "openai"),+ ("anthropic/claude-haiku-4.5", "anthropic"),+ ("google-ai-studio/gemini-2.5-flash", "google"),+ ("grok/grok-4", "xai"),+ (WORKERS_AI_LLAMA, "meta"),+ (f"workers-ai/{WORKERS_AI_LLAMA}", "meta"),+ ("@cf/moonshotai/kimi-k2.6", "moonshotai"),+ ("@cf/openai/gpt-oss-120b", "openai"),+ ("@cf/google/gemma-4-26b-a4b-it", "google"),+ ("@hf/google/gemma-7b-it", "google"),+ ("@cf/meta-llama/llama-2-7b-chat-hf-lora", "meta"),+ ("@cf/deepseek-ai/deepseek-v4-flash-0731", "deepseek"),+ ("@cf/zai-org/glm-5.2", "z-ai"),+ ],+)+def test_the_grading_vendor_is_the_model_s_author(model: str, vendor: str) -> None:+ """The gateway routes Google and xAI under its own provider names; a Workers AI+ id leads with a catalog namespace, not the author, and names some authors by+ their Hugging Face organisation."""+ assert grading_vendor("cloudflare", model) == vendor+++ ("provider", "model", "vendor"),+ [+ ("openrouter", "openai/gpt-4o", "openai"),+ ("requesty", "anthropic/claude-sonnet-4-6", "anthropic"),+ ("openrouter", "google/gemini-2.5-flash", "google"),+ ("requesty", "xai/grok-4", "xai"),+ ("requesty", "mistral/mistral-large", "mistral"),+ ("openrouter", "deepseek/deepseek-chat-v3.1", "deepseek"),+ ("openrouter", "z-ai/glm-5.2", "z-ai"),+ ("litellm", "bedrock/claude-sonnet-4-6@eu-central-1", "anthropic"),+ ("litellm", "vertex_ai/gemini-2.5-pro", "google"),+ ("openai", "gpt-4o", "openai"),+ ("azure", "my-deployment", "openai"),+ ("gemini", "gemini-2.0-flash", "google"),+ ("huggingface", "meta-llama/Llama-3.3-70B-Instruct", "huggingface"),+ ("http", None, "http"),+ ("openrouter", None, "openrouter"),+ ],+)+def test_every_other_provider_resolves_to_the_vendor_it_did_before(+ provider: str, model: str | None, vendor: str+) -> None:+ """Recorded on the branch point before the Workers AI change."""+ assert grading_vendor(provider, model) == vendor+++ ("provider", "model", "vendor"),+ [+ ("openrouter", "x-ai/grok-4", "xai"),+ ("openrouter", "mistralai/mistral-large", "mistral"),+ ("openrouter", "meta-llama/llama-3.3-70b-instruct", "meta"),+ ("atlascloud", "deepseek-ai/deepseek-v3.1", "deepseek"),+ ("atlascloud", "zai-org/glm-4.6", "z-ai"),+ ],+)+def test_one_vendor_spelled_two_ways_resolves_to_one_slug(+ provider: str, model: str, vendor: str+) -> None:+ """The five results this change alters for other gateways, on purpose: the+ same vendor has to read the same whichever gateway serves it. Before, these+ resolved to `x-ai`, `mistralai`, `meta-llama`, `deepseek-ai` and `zai-org`."""+ assert grading_vendor(provider, model) == vendor+++ ("model", "developer"),+ [+ (WORKERS_AI_LLAMA, "Meta"),+ ("@cf/deepseek-ai/deepseek-v4-flash-0731", "DeepSeek"),+ ("@cf/zai-org/glm-5.2", "Z.ai"),+ ("@cf/moonshotai/kimi-k2.6", "Moonshot AI"),+ ("openai/gpt-4.1-mini", "OpenAI"),+ ("google-ai-studio/gemini-2.5-flash", "Google"),+ ("grok/grok-4", "xAI"),+ ],+)+def test_the_grader_independence_inspection_knows_the_developer(+ model: str, developer: str+) -> None:+ """V05 reads an unresolved vendor as independent, so it has to resolve."""+ identity = resolve_vendor_identity(+ ProviderConfig(provider="cloudflare", model=model)+ )++ assert identity is not None+ assert identity.developer == developer+++ ("sut_model", "judge_provider", "judge_model"),+ [+ (WORKERS_AI_LLAMA, "openrouter", "meta-llama/llama-3.3-70b-instruct"),+ (+ "@cf/deepseek-ai/deepseek-v4-flash-0731",+ "openrouter",+ "deepseek/deepseek-chat-v3.1",+ ),+ ("@cf/zai-org/glm-5.2", "openrouter", "z-ai/glm-5.2"),+ ("@hf/google/gemma-7b-it", "gemini", "gemini-2.0-flash"),+ ("google-ai-studio/gemini-2.5-flash", "gemini", "gemini-2.0-flash"),+ ("grok/grok-4", "requesty", "xai/grok-4"),+ ("grok/grok-4", "openrouter", "x-ai/grok-4"),+ ("xai/grok-4.5", "openrouter", "x-ai/grok-4"),+ ("grok/grok-4", "cloudflare", "xai/grok-4.5"),+ ("@cf/mistral/mistral-7b-instruct-v0.2-lora", "openrouter", "mistralai/mistral-large"),+ (+ "@cf/mistral/mistral-7b-instruct-v0.2-lora",+ "cloudflare",+ "@cf/mistralai/mistral-small-3.1-24b-instruct",+ ),+ ],+)+def test_the_same_vendor_on_another_gateway_is_not_an_independent_judge(+ sut_model: str, judge_provider: str, judge_model: str+) -> None:+ judge = JudgeConfig(provider=judge_provider, model=judge_model)++ assert self_judge_bias_applies(judge, "cloudflare", sut_model) is True+++ ("sut_provider", "sut_model", "judge_model", "is_biased"),+ [+ (+ "cloudflare",+ "google-ai-studio/gemini-2.5-flash",+ "anthropic/claude-haiku-4.5",+ False,+ ),+ ("cloudflare", "openai/gpt-4.1", "openai/gpt-4.1-mini", True),+ ("cloudflare", WORKERS_AI_LLAMA, "@cf/moonshotai/kimi-k2.6", False),+ ("cloudflare", WORKERS_AI_LLAMA, "@cf/meta/llama-3.1-8b-instruct-fp8", True),+ ("cloudflare", "openai/gpt-4.1", "@cf/openai/gpt-oss-120b", True),+ ("openai", "gpt-4o", "openai/gpt-4.1-mini", True),+ ("gemini", "gemini-2.0-flash", "@cf/google/gemma-4-26b-a4b-it", True),+ ("anthropic", "claude-sonnet-4-6", WORKERS_AI_LLAMA, False),+ ],+)+def test_judge_independence_follows_the_model_vendor_not_the_gateway(+ sut_provider: str, sut_model: str, judge_model: str, is_biased: bool+) -> None:+ judge = JudgeConfig(provider="cloudflare", model=judge_model)++ assert self_judge_bias_applies(judge, sut_provider, sut_model) is is_biaseddiff --git a/ifixai/tests/test_sdk_timeout_native.py b/ifixai/tests/test_sdk_timeout_native.pyindex 0892a43..aede369 100644--- a/ifixai/tests/test_sdk_timeout_native.py+++ b/ifixai/tests/test_sdk_timeout_native.py@@ -18,6 +18,7 @@ ADAPTERS = [("openrouter", "OpenRouterProvider", "openai"),("orcarouter", "OrcaRouterProvider", "openai"),("requesty", "RequestyProvider", "openai"),+ ("cloudflare", "CloudflareAIGatewayProvider", "openai"),("anthropic", "AnthropicProvider", "anthropic"),]diff --git a/plugin/skills/ifixai/SKILL.md b/plugin/skills/ifixai/SKILL.mdindex 8769261..b73e20a 100644--- a/plugin/skills/ifixai/SKILL.md+++ b/plugin/skills/ifixai/SKILL.md@@ -105,7 +105,7 @@ Then check the engine is present:the Anthropic SDK only. To fall back to (or judge with) another provider, installits extra on demand into the same venv:`"${CLAUDE_PLUGIN_DATA}/venv/bin/pip" install "ifixai[openai]"` (or `gemini`,- `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `huggingface`). On Windows that pip is+ `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `huggingface`). On Windows that pip is`"${CLAUDE_PLUGIN_DATA}\venv\Scripts\pip.exe"`. A missing SDK fails fast namingthe provider, so install then re-run.- **Keys live in the environment, never on a command line.** Each provider (and the@@ -520,7 +520,7 @@ Present these two choices in this order and recommend the first:path; a "no" moves to option 2. Never guess an endpoint or silently fall back.2. **Replicate the model as a bare stand-in (fallback, no endpoint).** Name the**provider** that runs the model beneath the agent (`--provider`): `anthropic`,- `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `huggingface`. The engine+ `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `huggingface`. The engineresolves each provider's default model; add `--model` to pin the user's actualproduction model. `azure`/`bedrock` have no default and require an explicitmodel/deployment id (and `azure` also needs `--endpoint`). This tests the model's@@ -544,6 +544,7 @@ Present these two choices in this order and recommend the first:| openrouter | `OPENROUTER_API_KEY` || orcarouter | `ORCAROUTER_API_KEY` || requesty | `REQUESTY_API_KEY` |+| cloudflare | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` || huggingface | `HUGGINGFACE_API_TOKEN` or `HF_TOKEN` |**Decision 2: how much to run (suite, then depth):**diff --git a/pyproject.toml b/pyproject.tomlindex e14138a..2495907 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -44,6 +44,7 @@ atlascloud = ["openai>=1.0"]openrouter = ["openai>=1.0"]orcarouter = ["openai>=1.0"]requesty = ["openai>=1.0"]+cloudflare = ["openai>=1.0"]gemini = ["google-generativeai>=0.3"]anthropic = ["anthropic>=0.18"]azure = ["openai>=1.0"]