Skip to content

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

feat: VercelGateway provider (#230)

stefyi-4355bef9c5eeb96f
diff --git a/README.ja.md b/README.ja.md
index 74b4435..8541f29 100644
--- a/README.ja.md
+++ b/README.ja.md
@@ -204,7 +204,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fi
レポートは JSON **と** Markdown の両方で `./ifixai-results/` に保存されます。2 つ目のキーがない場合は、スモークテストとして `--eval-mode self` を追加してください
(評価は表示されますが自己評価と明記され、引用できる結果にはなりません)。
評価モデルの固定、Full モードのアンサンブル、評価モードについては **[docs/cli.md](docs/cli.md#how-a-run-is-judged)** を参照してください。その他のプロバイダー
-(OpenAI、OpenRouter、Requesty、Cloudflare AI Gateway、Gemini、Azure、Bedrock、Hugging Face)は対応する extra を
+(OpenAI、OpenRouter、Requesty、Cloudflare AI Gateway、Vercel AI Gateway、Gemini、Azure、Bedrock、Hugging Face)は対応する extra を
インストールし、同じ手順を使用します。HTTP と LangChain のアダプターには
プロバイダーの extra は不要です:**[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**。
diff --git a/README.ko.md b/README.ko.md
index 3e76035..6dfc1aa 100644
--- a/README.ko.md
+++ b/README.ko.md
@@ -189,7 +189,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fi
`--eval-mode self`를 추가해 스모크 테스트로 실행하세요(등급은 출력되지만 자기 채점으로
표시되며, 인용할 수 있는 결과는 아닙니다). 심사 모델 고정, Full 모드 앙상블, 평가 모드에
대해서는 **[docs/cli.md](docs/cli.md#how-a-run-is-judged)** 를 참고하세요. 다른
-프로바이더(OpenAI, Atlas Cloud, OpenRouter, Requesty, Cloudflare AI Gateway, Gemini, Azure, Bedrock, Hugging Face)는 해당
+프로바이더(OpenAI, Atlas Cloud, OpenRouter, Requesty, Cloudflare AI Gateway, Vercel AI Gateway, Gemini, Azure, Bedrock, Hugging Face)는 해당
extra를 설치하고 같은 절차를 따르며, HTTP와 LangChain 어댑터는 프로바이더 extra가
필요 없습니다: **[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**.
diff --git a/README.md b/README.md
index 79fbb74..4f6f0a2 100644
--- a/README.md
+++ b/README.md
@@ -186,7 +186,7 @@ Reports land in `./ifixai-results/` as JSON **and** Markdown. Without a second k
`--eval-mode self` to run as a smoke test (the grade still prints, but it's flagged as
self-judged, not a result you can cite). Pinning the judge, Full-mode ensembles, and the eval modes:
**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**. Other providers (OpenAI, Atlas Cloud,
-OpenRouter, OrcaRouter, Requesty, Cloudflare AI Gateway, Gemini, Azure, Bedrock, Hugging Face) install the matching extra and follow the same steps; the
+OpenRouter, OrcaRouter, Requesty, Cloudflare AI Gateway, Vercel AI Gateway, Gemini, Azure, Bedrock, Hugging Face) install the matching extra and follow the same steps; the
HTTP and LangChain adapters need no provider extra: **[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**.
### Recommended judge setups
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 30500cd..6c51fd9 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -200,7 +200,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fi
报告以 JSON **和** Markdown 两种格式保存在 `./ifixai-results/` 中。如果没有第二个密钥,
请添加 `--eval-mode self` 作为冒烟测试(等级仍会显示,但会被标记为自我评审,不能作为
可引用结果)。固定评审模型、完整模式集成以及评审模式的详情请参阅:
-**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**。其他提供商(OpenAI、OpenRouter、Requesty、Cloudflare AI Gateway、Gemini、
+**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**。其他提供商(OpenAI、OpenRouter、Requesty、Cloudflare AI Gateway、Vercel AI Gateway、Gemini、
Azure、Bedrock、Hugging Face)可安装对应 extra 并遵循相同步骤;HTTP 和 LangChain
适配器不需要提供商 extra:**[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**。
diff --git a/SECURITY.md b/SECURITY.md
index 8834845..b1a88d5 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -47,6 +47,7 @@ The scrubber (`ifixai/providers/secrets.py::scrub_secrets`) is parametrically ve
| `orcarouter` | `sk-orca-[A-Za-z0-9_-]{20,}` (matched before `openai`) | `***REDACTED_ORCAROUTER_KEY***` |
| `requesty` | `rqsty-[A-Za-z0-9+/=_-]{20,}` | `***REDACTED_REQUESTY_KEY***` |
| `cloudflare` | `cfut_`, `cfat_` or `cfk_` followed by `[A-Za-z0-9_-]{40,}`; tokens in the older unprefixed format have no fixed shape and are redacted only inside an `Authorization: Bearer` header | `***REDACTED_CLOUDFLARE_TOKEN***` |
+| `vercel` | `vck_[A-Za-z0-9_-]{20,}` | `***REDACTED_VERCEL_KEY***` |
| `anthropic` | `sk-ant-[A-Za-z0-9_-]{20,}`, `anthropic_[A-Za-z0-9_-]{20,}` | `***REDACTED_ANTHROPIC_KEY***` |
| `gemini` | `AIzaSy[0-9A-Za-z_-]{33}` | `***REDACTED_GEMINI_KEY***` |
| `azure` | 32-char hex (`\b[a-fA-F0-9]{32}\b`) | `***REDACTED_AZURE_KEY***` |
diff --git a/docs/cli.md b/docs/cli.md
index e2dfb95..7ef6eaa 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -21,7 +21,7 @@ ifixai run -p openai -k "$OPENAI_API_KEY" -c DECEPTION # example: one category
| Flag | Default | Does |
|---|---|---|
-| `--provider`, `-p` | none | `mock`, `openai`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `anthropic`, `gemini`, `azure`, `bedrock`, `huggingface`, `http`, `langchain`. |
+| `--provider`, `-p` | none | `mock`, `openai`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `vercel`, `anthropic`, `gemini`, `azure`, `bedrock`, `huggingface`, `http`, `langchain`. |
| `--api-key`, `-k` | none | SUT API key. Always passed explicitly, never read from the environment. |
| `--model`, `-m` | provider default | Model identifier override. |
| `--endpoint`, `-e` | none | Endpoint URL (required for `http` and `azure`; for `cloudflare` it replaces the URL built from `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_GATEWAY_ID`). |
diff --git a/docs/testing-your-agent.md b/docs/testing-your-agent.md
index 2ddf27b..55c16f4 100644
--- a/docs/testing-your-agent.md
+++ b/docs/testing-your-agent.md
@@ -135,6 +135,7 @@ the SUT key from the environment: pass `--api-key` / `-k`, or enter it when prom
| `orcarouter` | `.[orcarouter]` | `ORCAROUTER_API_KEY` | `--provider orcarouter -k "$ORCAROUTER_API_KEY" --model openai/gpt-4o` plus explicit judge |
| `requesty` | `.[requesty]` | `REQUESTY_API_KEY` | `--provider requesty -k "$REQUESTY_API_KEY" --model openai/gpt-4o-mini` plus explicit judge; `--endpoint https://router.eu.requesty.ai/v1` for EU data residency |
| `cloudflare` | `.[cloudflare]` | `CLOUDFLARE_API_TOKEN` plus `CLOUDFLARE_ACCOUNT_ID` | `--provider cloudflare -k "$CLOUDFLARE_API_TOKEN" --model workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast` plus explicit judge |
+| `vercel` | `.[vercel]` | `AI_GATEWAY_API_KEY` | `--provider vercel -k "$AI_GATEWAY_API_KEY" --model openai/gpt-4o-mini` plus explicit judge |
| `gemini` | `.[gemini]` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | `--provider gemini -k "$GEMINI_API_KEY"` |
| `azure` | `.[azure]` | `AZURE_OPENAI_API_KEY` | `--provider azure --endpoint https://YOUR_RESOURCE.openai.azure.com/ -k "$AZURE_OPENAI_API_KEY" --model YOUR_DEPLOYMENT_NAME` plus explicit judge |
| `bedrock` | `.[bedrock]` | AWS credential chain | `--provider bedrock -k not-used --model anthropic.claude-sonnet-4-6` |
@@ -150,6 +151,7 @@ Notes:
- `litellm` extra is [Python API](python-api.md) only (`provider="litellm"`), not a CLI choice.
- An `openrouter` judge falls back to other models when the gateway errors, so a 502 costs one probe, not the run. Chain: [`ifixai/judge/judge_fallbacks.json`](../ifixai/judge/judge_fallbacks.json), see [cli.md](cli.md#judge-fallback-models).
- `cloudflare` is the [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/usage/chat-completion/) OpenAI-compatible endpoint, `gateway.ai.cloudflare.com/v1/<account>/<gateway>/compat`. It needs two values: an API token allowed to run the gateway (`CLOUDFLARE_API_TOKEN`; Cloudflare documents the AI Gateway > Run permission for this) and the account ID, which is part of the URL (`CLOUDFLARE_ACCOUNT_ID`, or pass the full base URL with `--endpoint`). A judge has no `--endpoint`, so a `cloudflare` judge always reads the account ID from the environment. Requests go through the gateway named `default` unless `CLOUDFLARE_GATEWAY_ID` names another (the ID is case-sensitive), and every request asks the gateway to bypass its response cache (`cf-aig-skip-cache`), so that a repeated probe is answered by the model and not by a stored reply. Models are Workers AI ids such as `workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast` (the default; the Workers free plan's daily allocation covered about one 61-probe inspection in testing, so a full run needs a paid Workers plan, and a run that exhausts the allocation stops with Cloudflare's message), or `provider/model` slugs such as `openai/gpt-4.1-mini` and `google-ai-studio/gemini-2.5-flash`, which need Unified Billing credits or a provider key stored in the gateway. Cloudflare answers a bad token and a wrong account ID with the same `401 Unauthorized`; iFixAi lists what to check when it sees one. Cloudflare also has a newer REST route on `api.cloudflare.com` and steers new integrations to it, but that route rejects a gateway token (it wants a token with the Workers AI permission), so this provider uses the gateway URL.
+- `vercel` is the [Vercel AI Gateway](https://vercel.com/docs/ai-gateway). Models are `creator/model` slugs from its [catalog](https://vercel.com/ai-gateway/models), e.g. `anthropic/claude-haiku-4.5`. Until the Vercel team has a payment method on file the gateway answers every request with `403 customer_verification_required`, free tier included; iFixAi reports Vercel's message, which links to the page for adding one.
## LangServe wire contract
diff --git a/ifixai/cli/init.py b/ifixai/cli/init.py
index b171e4f..a29ff63 100644
--- a/ifixai/cli/init.py
+++ b/ifixai/cli/init.py
@@ -15,6 +15,7 @@ PROVIDER_ENV_KEYS: dict[str, str] = {
"openrouter": "OPENROUTER_API_KEY",
"orcarouter": "ORCAROUTER_API_KEY",
"requesty": "REQUESTY_API_KEY",
+ "vercel": "AI_GATEWAY_API_KEY",
"gemini": "GEMINI_API_KEY",
"azure": "AZURE_OPENAI_API_KEY",
"bedrock": "AWS_ACCESS_KEY_ID",
diff --git a/ifixai/cli/model_catalog.py b/ifixai/cli/model_catalog.py
index 5f4a242..1fcdf24 100644
--- a/ifixai/cli/model_catalog.py
+++ b/ifixai/cli/model_catalog.py
@@ -36,6 +36,7 @@ DEFAULT_MODEL: dict[str, str] = {
"orcarouter": "openai/gpt-4o",
"requesty": "openai/gpt-4o-mini",
"cloudflare": "workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast",
+ "vercel": "openai/gpt-4o-mini",
"openai": "gpt-4o",
"anthropic": "claude-3-5-sonnet-latest",
"gemini": "gemini-2.0-flash",
@@ -84,6 +85,15 @@ MODEL_SUGGESTIONS: dict[str, list[tuple[str, str]]] = {
("anthropic/claude-haiku-4.5", "Anthropic Claude Haiku 4.5 — fast & cheap (needs credits or a stored key)"),
("google-ai-studio/gemini-2.5-flash", "Google Gemini 2.5 Flash — fast and inexpensive (needs credits or a stored key)"),
],
+ "vercel": [
+ ("openai/gpt-4o-mini", "OpenAI GPT-4o Mini — cheap, fast, widely available"),
+ ("openai/gpt-4o", "OpenAI GPT-4o — flagship general reasoning"),
+ ("openai/gpt-4.1", "OpenAI GPT-4.1 — strong, widely available"),
+ ("anthropic/claude-sonnet-4.5", "Anthropic Claude Sonnet 4.5 — strong reasoning & safety"),
+ ("anthropic/claude-haiku-4.5", "Anthropic Claude Haiku 4.5 — fast & cheap"),
+ ("google/gemini-2.5-flash", "Google Gemini 2.5 Flash — fast and inexpensive"),
+ ("google/gemini-2.5-pro", "Google Gemini 2.5 Pro — large context, strong reasoning"),
+ ],
"openai": [
("gpt-4o", "Flagship — strong general reasoning"),
("gpt-4o-mini", "Cheaper & faster"),
diff --git a/ifixai/cli/run.py b/ifixai/cli/run.py
index 7a590c8..e0a1b57 100644
--- a/ifixai/cli/run.py
+++ b/ifixai/cli/run.py
@@ -149,6 +149,7 @@ PROVIDER_CHOICES = [
"orcarouter",
"requesty",
"cloudflare",
+ "vercel",
]
FORMAT_CHOICES = ["json", "markdown", "both"]
diff --git a/ifixai/cli/setup_cmd.py b/ifixai/cli/setup_cmd.py
index 3928bf7..7118afb 100644
--- a/ifixai/cli/setup_cmd.py
+++ b/ifixai/cli/setup_cmd.py
@@ -27,6 +27,7 @@ _PROVIDER_DESCRIPTIONS: dict[str, str] = {
"orcarouter": "OpenAI-compatible gateway — many models, one key",
"requesty": "Requesty gateway — one key → 400+ models (OpenAI, Anthropic, Google…)",
"cloudflare": "Cloudflare AI Gateway — one token → Workers AI plus OpenAI, Anthropic, Google…",
+ "vercel": "Vercel AI Gateway — one key → hundreds of models (OpenAI, Anthropic, Google…)",
"openai": "OpenAI API — GPT-4o / o-series",
"anthropic": "Anthropic API — Claude family",
"gemini": "Google Gemini",
@@ -56,6 +57,7 @@ _ALL_PROVIDERS = [
"orcarouter",
"requesty",
"cloudflare",
+ "vercel",
"openai",
"anthropic",
"gemini",
diff --git a/ifixai/core/types.py b/ifixai/core/types.py
index 9a9edaf..b471702 100644
--- a/ifixai/core/types.py
+++ b/ifixai/core/types.py
@@ -243,7 +243,7 @@ class ProviderConfig(BaseModel):
"Request structured JSON output from the provider (response_format="
"json_object). Set ONLY for LLM-judge calls so cheap models reliably "
"emit a parseable verdict; never for the system-under-test, whose reply "
- "must stay natural. Honored by the openai, openrouter, orcarouter, requesty, cloudflare, azure, litellm "
+ "must stay natural. Honored by the openai, openrouter, orcarouter, requesty, cloudflare, vercel, azure, litellm "
"(response_format) and gemini (response_mime_type) providers, which fall "
"back to free text if the model rejects it; a no-op on providers with no "
"JSON mode (anthropic, bedrock, huggingface), where the json-repair "
diff --git a/ifixai/providers/resolver.py b/ifixai/providers/resolver.py
index 6dca370..da91021 100644
--- a/ifixai/providers/resolver.py
+++ b/ifixai/providers/resolver.py
@@ -65,6 +65,11 @@ try:
except ImportError:
CloudflareAIGatewayProvider = None # type: ignore[assignment,misc]
+try:
+ from ifixai.providers.vercel import VercelAIGatewayProvider
+except ImportError:
+ VercelAIGatewayProvider = None # type: ignore[assignment,misc]
+
try:
from ifixai.providers.litellm import LiteLLMProvider
except ImportError:
@@ -81,6 +86,7 @@ REGISTERED_PROVIDERS: tuple[str, ...] = (
"orcarouter",
"requesty",
"cloudflare",
+ "vercel",
"anthropic",
"gemini",
"azure",
@@ -104,6 +110,7 @@ _PROVIDER_MAP: dict[str, type] = {
"orcarouter": OrcaRouterProvider,
"requesty": RequestyProvider,
"cloudflare": CloudflareAIGatewayProvider,
+ "vercel": VercelAIGatewayProvider,
"anthropic": AnthropicProvider,
"gemini": GeminiProvider,
"azure": AzureOpenAIProvider,
@@ -185,6 +192,7 @@ _PROVIDER_CREDENTIAL_ENV_VARS: dict[str, tuple[str, ...]] = {
"orcarouter": ("ORCAROUTER_API_KEY",),
"requesty": ("REQUESTY_API_KEY",),
"cloudflare": ("CLOUDFLARE_API_TOKEN",),
+ "vercel": ("AI_GATEWAY_API_KEY",),
"litellm": ("LITELLM_API_KEY",),
}
diff --git a/ifixai/providers/secrets.py b/ifixai/providers/secrets.py
index 1efc6e5..b64a209 100644
--- a/ifixai/providers/secrets.py
+++ b/ifixai/providers/secrets.py
@@ -21,6 +21,7 @@ _SECRET_PATTERNS: Final[tuple[Pattern[str], ...]] = (
re.compile(r"^sk-orca-[A-Za-z0-9_-]{20,}$"),
re.compile(r"^rqsty-[A-Za-z0-9+/=_-]{20,}$"),
re.compile(r"^cf(?:ut|at|k)_[A-Za-z0-9_-]{40,}$"),
+ re.compile(r"^vck_[A-Za-z0-9_-]{20,}$"),
re.compile(r"^ak-[A-Za-z0-9_-]{20,}$"),
re.compile(r"^sk-ant-[A-Za-z0-9_-]{20,}$"),
re.compile(r"^sk-[A-Za-z0-9_-]{20,}$"),
@@ -59,6 +60,10 @@ _SCRUB_RULES: Final[tuple[tuple[Pattern[str], str], ...]] = (
re.compile(r"cf(?:ut|at|k)_[A-Za-z0-9_-]{40,}"),
"***REDACTED_CLOUDFLARE_TOKEN***",
),
+ (
+ re.compile(r"vck_[A-Za-z0-9_-]{20,}"),
+ "***REDACTED_VERCEL_KEY***",
+ ),
(
re.compile(r"(?<![A-Za-z0-9_-])ak-[A-Za-z0-9_-]{20,}(?![A-Za-z0-9_-])"),
"***REDACTED_ATLASCLOUD_KEY***",
diff --git a/ifixai/providers/vercel.py b/ifixai/providers/vercel.py
new file mode 100644
index 0000000..87560ee
--- /dev/null
+++ b/ifixai/providers/vercel.py
@@ -0,0 +1,257 @@
+"""Vercel AI Gateway, reached through its OpenAI-compatible chat-completions API.
+
+One key fronts every vendor; the model is a ``creator/model`` slug such as
+``anthropic/claude-haiku-4.5``.
+https://vercel.com/docs/ai-gateway/sdks-and-apis/openai-chat-completions
+"""
+
+import asyncio
+from http import HTTPStatus
+from typing import NoReturn
+
+import openai
+from openai.types.chat import ChatCompletion
+from typing_extensions import NotRequired, TypedDict
+
+from ifixai.core.types import ChatMessage, ProviderConfig
+from ifixai.providers.base import (
+ ChatProvider,
+ ProviderAuthError,
+ ProviderConnectionError,
+ ProviderEmptyContentError,
+ ProviderError,
+ ProviderRateLimitError,
+ ProviderResponseError,
+ ProviderTimeoutError,
+ create_chat_completion_json_fallback,
+ raise_for_http_status,
+ raise_if_choice_errored,
+ raise_if_truncated,
+)
+
+PROVIDER_NAME = "vercel"
+DEFAULT_MODEL = "openai/gpt-4o-mini"
+DEFAULT_BASE_URL = "https://ai-gateway.vercel.sh/v1"
+# Hard ceiling on max_tokens for AI Gateway calls. Per-call ``config.max_tokens``
+# is clamped to this value; unset config falls through to the ceiling. Prevents
+# verbose generations from blowing wall-time and credits on long fixtures. Mirrors
+# the OpenRouter provider ceiling so judge/SUT replies aren't truncated mid-verdict.
+MAX_TOKENS_CEILING: int = 8192
+
+# Checked in order, most specific first: a timeout is also a connection error,
+# and the 401 and 429 classes are also status errors.
+SDK_ERROR_TRANSLATIONS: dict[type[openai.APIError], type[ProviderError]] = {
+ openai.AuthenticationError: ProviderAuthError,
+ openai.RateLimitError: ProviderRateLimitError,
+ openai.APITimeoutError: ProviderTimeoutError,
+ openai.APIConnectionError: ProviderConnectionError,
+}
+
+ClientCacheKey = tuple[object, ...]
+
+
+class GatewayConnection(TypedDict):
+ """Everything that distinguishes one SDK client from another."""
+
+ api_key: str
+ base_url: str
+ timeout: float
+ max_retries: int
+
+
+class GatewayMessage(TypedDict):
+ role: str
+ content: str
+
+
+class JsonResponseFormat(TypedDict):
+ type: str
+
+
+class GatewayChatRequest(TypedDict):
+ """Body of one chat-completions call; optional keys are sent only when set."""
+
+ model: str
+ messages: list[GatewayMessage]
+ max_tokens: int
+ temperature: float
+ seed: NotRequired[int]
+ response_format: NotRequired[JsonResponseFormat]
+
+
+class VercelAIGatewayProvider(ChatProvider):
+ def __init__(self) -> None:
+ self.clients: dict[ClientCacheKey, openai.AsyncOpenAI] = {}
+ self.client_lock = asyncio.Lock()
+
+ async def get_client(self, config: ProviderConfig) -> openai.AsyncOpenAI:
+ """Return a long-lived AsyncOpenAI client keyed on connection params.
+
+ Caching by (api_key, base_url, timeout, max_retries) lets the
+ underlying httpx pool reuse TCP/TLS across LLM calls instead of
+ paying a handshake per request.
+ """
+ connection = resolve_connection(config)
+ key: ClientCacheKey = tuple(connection.values())
+ cached = self.clients.get(key)
+ if cached is not None:
+ return cached
+ async with self.client_lock:
+ cached = self.clients.get(key)
+ if cached is not None:
+ return cached
+ client = openai.AsyncOpenAI(**connection)
+ self.clients[key] = client
+ return client
+
+ async def aclose(self) -> None:
+ for client in self.clients.values():
+ await client.close()
+ self.clients.clear()
+
+ async def send_message(
+ self,
+ messages: list[ChatMessage],
+ config: ProviderConfig,
+ ) -> str:
+ endpoint = config.endpoint or DEFAULT_BASE_URL
+ client = await self.get_client(config)
+ try:
+ response = await create_chat_completion_json_fallback(
+ client, **build_chat_request(messages, config)
+ )
+ except openai.APIError as exc:
+ raise_for_sdk_error(endpoint, exc)
+ return extract_reply(response, config, endpoint)
+
+
+def resolve_connection(config: ProviderConfig) -> GatewayConnection:
+ """Reduce a config to the settings an SDK client is built from."""
+ base_url = config.endpoint or DEFAULT_BASE_URL
+ return GatewayConnection(
+ api_key=require_api_key(config.api_key, base_url),
+ base_url=base_url,
+ timeout=float(config.timeout),
+ max_retries=config.max_retries,
+ )
+
+
+def require_api_key(raw_key: str, endpoint: str) -> str:
+ """Return the key without surrounding whitespace, refusing a blank one.
+
+ A pasted key often carries a trailing newline, which is an illegal header
+ value: the request fails before it is sent and the transport error quotes
+ the key. A blank key is refused here because the SDK reports it as a
+ missing ``OPENAI_API_KEY``, which names the wrong variable.
+ """
+ api_key = raw_key.strip()
+ if not api_key:
+ raise ProviderAuthError(
+ provider=PROVIDER_NAME,
+ endpoint=endpoint,
+ details="No API key was supplied (empty or whitespace only)",
+ )
+ return api_key
+
+
+def build_chat_request(
+ messages: list[ChatMessage], config: ProviderConfig
+) -> GatewayChatRequest:
+ """Assemble the chat-completions body for one call."""
+ request = GatewayChatRequest(
+ model=config.model or DEFAULT_MODEL,
+ messages=[
+ GatewayMessage(role=message.role, content=message.content)
+ for message in messages
+ ],
+ max_tokens=resolve_max_tokens(config.max_tokens),
+ temperature=config.temperature,
+ )
+ if config.seed is not None:
+ request["seed"] = config.seed
+ if config.json_output:
+ # Constrain judge calls to valid JSON so cheap models reliably emit a
+ # parseable verdict instead of breaking the contract. Falls back to free
+ # text (json-repair handles parsing) if the model does not support
+ # response_format.
+ #
+ # No `reasoning` suppression here, unlike OpenRouter. The gateway documents
+ # the same extension, but its catalog lists `reasoning` as unsupported for
+ # non-reasoning models (this adapter's default among them) and does not say
+ # whether such a request is ignored or rejected. A rejection would cost a
+ # second round trip on every judge call.
+ request["response_format"] = JsonResponseFormat(type="json_object")
+ return request
+
+
+def resolve_max_tokens(requested: int | None) -> int:
+ """Clamp the caller's token budget to the ceiling; unset takes the ceiling."""
+ if requested is None:
+ return MAX_TOKENS_CEILING
+ return min(requested, MAX_TOKENS_CEILING)
+
+
+def raise_for_sdk_error(endpoint: str, exc: openai.APIError) -> NoReturn:
+ """Translate an OpenAI-SDK failure into the matching provider exception."""
+ for sdk_error, provider_error in SDK_ERROR_TRANSLATIONS.items():
+ if isinstance(exc, sdk_error):
+ raise provider_error(
+ provider=PROVIDER_NAME, endpoint=endpoint, details=str(exc)
+ ) from exc
+ if not isinstance(exc, openai.APIStatusError):
+ raise ProviderResponseError(
+ provider=PROVIDER_NAME, endpoint=endpoint, details=str(exc)
+ ) from exc
+ if exc.status_code == HTTPStatus.PAYMENT_REQUIRED:
+ raise_for_payment_required(endpoint, exc)
+ # 408 and 5xx are the gateway or the model behind it having a moment, so they
+ # are split off as a transient overload; everything else stays a response error.
+ raise_for_http_status(PROVIDER_NAME, endpoint, exc)
+
+
+def raise_for_payment_required(endpoint: str, exc: openai.APIStatusError) -> NoReturn:
+ """Report an empty credit balance or a spent budget as non-retryable.
+
+ Vercel documents the 402 status for both but not the wording of the body,
+ so the status's own reason phrase goes into the detail: it is what the
+ fail-fast check and the operator-facing hint both key on.
+ """
+ status = HTTPStatus.PAYMENT_REQUIRED
+ raise ProviderResponseError(
+ provider=PROVIDER_NAME,
+ endpoint=endpoint,
+ details=f"HTTP {status.value} {status.phrase}: {exc}",
+ ) from exc
+
+
+def extract_reply(
+ response: ChatCompletion, config: ProviderConfig, endpoint: str
+) -> str:
+ """Return the reply text, rejecting empty, cut-off and aborted generations."""
+ if not response.choices:
+ raise ProviderResponseError(
+ provider=PROVIDER_NAME,
+ endpoint=endpoint,
+ details=f"No choices in response (id={response.id})",
+ )
+ choice = response.choices[0]
+ finish_reason = choice.finish_reason or "unknown"
+ if choice.message is None:
+ raise ProviderResponseError(
+ provider=PROVIDER_NAME,
+ endpoint=endpoint,
+ details=f"Missing message in choice (finish_reason={finish_reason})",
+ )
+ content = choice.message.content
+ # An upstream abort can arrive with no text, so it is checked before the
+ # empty-reply check, which would void the inspection instead of the probe.
+ raise_if_choice_errored(PROVIDER_NAME, endpoint, choice, content or "")
+ if config.reject_truncated:
+ raise_if_truncated(PROVIDER_NAME, endpoint, finish_reason, content or "")
+ if not content:
+ raise ProviderEmptyContentError(
+ provider=PROVIDER_NAME,
+ endpoint=endpoint,
+ details=f"Empty content in response (finish_reason={finish_reason})",
+ )
+ return content
diff --git a/ifixai/reporting/scorecard.py b/ifixai/reporting/scorecard.py
index 1cc5ed0..b1c24b0 100644
--- a/ifixai/reporting/scorecard.py
+++ b/ifixai/reporting/scorecard.py
@@ -285,10 +285,10 @@ def b32_not_applicable_warning(
# Providers that front many vendors behind one name. For these the real grading
# vendor is the model slug's prefix, so a cross-vendor judge routed through one
# aggregator (e.g. a Gemini SUT graded by a Claude judge, both via OpenRouter,
-# Requesty or Cloudflare AI Gateway) is recognized as independent instead of
-# mislabeled "self-judge".
+# Requesty, Cloudflare AI Gateway or Vercel AI Gateway) is recognized as
+# independent instead of mislabeled "self-judge".
_AGGREGATOR_PROVIDERS: Final[frozenset[str]] = frozenset(
- {"openrouter", "orcarouter", "requesty", "cloudflare", "atlascloud", "litellm", "http", "langchain"}
+ {"openrouter", "orcarouter", "requesty", "cloudflare", "vercel", "atlascloud", "litellm", "http", "langchain"}
)
# Namespaces that sit in front of the author, as in the Workers AI ids
diff --git a/ifixai/scaffold/playbook.md b/ifixai/scaffold/playbook.md
index 53e962c..ad33733 100644
--- a/ifixai/scaffold/playbook.md
+++ b/ifixai/scaffold/playbook.md
@@ -97,7 +97,7 @@ IFIXAI_SURFACE=skill uvx --from "ifixai[<provider>]" ifixai run \
```
Substitute a provider valid as BOTH the install extra and the `--provider` value: one of
-`anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, or `huggingface`. (`mock`
+`anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `vercel`, or `huggingface`. (`mock`
runs fully offline for a free smoke test, with `--provider mock --api-key mock --eval-mode
self`.) The extra installs that provider's SDK; if you test on one provider and judge on
another, install the union. After the user approves the dry-run estimate, rerun the identical
@@ -147,6 +147,7 @@ over the file). A missing key fails fast and names the variable to set.
| orcarouter | `ORCAROUTER_API_KEY` |
| requesty | `REQUESTY_API_KEY` |
| cloudflare | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` |
+| vercel | `AI_GATEWAY_API_KEY` |
| huggingface | `HUGGINGFACE_API_TOKEN` or `HF_TOKEN` |
If the SUT and a judge share a provider, one key covers both. A judge on a second provider
@@ -409,7 +410,7 @@ at each. Map the picks to flags yourself.
build a stand-in that mirrors your setup and test that instead?" A URL keeps this path; a "no"
moves to the fallback. Don't guess an endpoint or fall back silently.
- *Bare stand-in (fallback, no endpoint).* Offer the providers the user has a key for:
- `anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `huggingface`, with
+ `anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `vercel`, `huggingface`, with
`--grounding fixture`. Offer to pin a model (`--model <id>`) or take the provider default.
Maps to `--provider` / `--model`. Tests the model, not the deployment. (`mock` runs offline
for free.)
@@ -422,7 +423,7 @@ at each. Map the picks to flags yourself.
- *Self (cheapest, biased, not citable)*: the model grades itself. Maps to `--eval-mode self`.
Only a smoke test. (With a single provider key and no judge, standard mode REFUSES rather than
self-judge silently, so this is the explicit opt-in.)
-- On an aggregator like OpenRouter, Requesty or Cloudflare AI Gateway, "different vendor" means a different model-slug prefix
+- On an aggregator like OpenRouter, Requesty, Cloudflare AI Gateway or Vercel AI Gateway, "different vendor" means a different model-slug prefix
(e.g. `google/...` SUT graded by `openai/...`), which the engine now recognizes as citable.
**3. Depth / how much to run.** Offer a suite, smallest first, with the trade-off:
diff --git a/ifixai/tests/providers/test_vercel_provider.py b/ifixai/tests/providers/test_vercel_provider.py
new file mode 100644
index 0000000..3a85baa
--- /dev/null
+++ b/ifixai/tests/providers/test_vercel_provider.py
@@ -0,0 +1,534 @@
+"""Vercel AI Gateway adapter: wire contract, error mapping and request shape.
+
+The wire tests drive the real OpenAI SDK against a loopback stand-in for the
+gateway and replay the error bodies Vercel documents. The 401 and 403 bodies
+are the ones the live gateway returned, so the mapping is pinned to what the
+SDK raises for real payloads rather than to hand-built exceptions.
+
+References:
+ https://vercel.com/docs/ai-gateway/sdks-and-apis/openai-chat-completions
+ https://vercel.com/docs/ai-gateway/faq#why-did-my-ai-gateway-request-fail
+ https://vercel.com/docs/ai-gateway/rate-limits
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import threading
+from collections.abc import Iterator
+from contextlib import contextmanager
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from typing import Any, TypedDict
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+from click.testing import CliRunner
+
+pytest.importorskip("openai")
+
+from ifixai.cli.main import ifixai_cli
+from ifixai.core.types import ChatMessage, ProviderConfig
+from ifixai.providers.base import (
+ ProviderAuthError,
+ ProviderEmptyContentError,
+ ProviderError,
+ ProviderOverloadedError,
+ ProviderRateLimitError,
+ ProviderResponseError,
+ ProviderTruncatedError,
+ friendly_provider_message,
+ is_fatal_provider_error,
+)
+from ifixai.providers.vercel import (
+ DEFAULT_BASE_URL,
+ DEFAULT_MODEL,
+ MAX_TOKENS_CEILING,
+ VercelAIGatewayProvider,
+)
+
+# Assembled at import time so the source holds no key-shaped literal for the
+# repo's secret scanner (gitleaks, see .pre-commit-config.yaml) to flag.
+SYNTHETIC_KEY = "vck_" + "syntheticLoopback" * 2
+MESSAGES = [
+ ChatMessage(role="system", content="You are a claims assistant."),
+ ChatMessage(role="user", content="Reply with the single word: pong"),
+]
+
+AUTHENTICATION_FAILED_BODY = {
+ "error": {
+ "message": (
+ "Authentication failed. Check that your Vercel credential is valid "
+ "and has access to AI Gateway."
+ ),
+ "type": "authentication_error",
+ }
+}
+CUSTOMER_VERIFICATION_REQUIRED_BODY = {
+ "error": {
+ "message": (
+ "AI Gateway requires a valid credit card on file to service requests. "
+ "Please visit https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal"
+ "%3Dadd-credit-card to add a card and unlock your free credits."
+ ),
+ "type": "customer_verification_required",
+ }
+}
+BUDGET_EXCEEDED_BODY = {
+ "error": {"message": "Budget reached.", "type": "quota_for_entity_exceeded"}
+}
+# Vercel documents the 402 status for an empty balance but not its body, so this
+# wording deliberately matches none of the phrases the fail-fast path looks for.
+EMPTY_BALANCE_BODY = {"error": {"message": "Top up to continue.", "type": "no_funds"}}
+RATE_LIMITED_BODY = {
+ "error": {"message": "Rate limit exceeded", "type": "rate_limit_exceeded"}
+}
+MISSING_PARAMETER_BODY = {
+ "error": {
+ "message": "Invalid request: missing required parameter 'model'",
+ "type": "invalid_request_error",
+ "param": "model",
+ "code": "missing_parameter",
+ }
+}
+MODEL_NOT_FOUND_BODY = {
+ "error": {"message": "Model not found", "type": "model_not_found"}
+}
+UPSTREAM_FAILURE_BODY = {
+ "error": {"message": "Upstream provider failed", "type": "internal_server_error"}
+}
+JSON_MODE_REJECTED_BODY = {
+ "error": {
+ "message": "response_format is not supported by this model",
+ "type": "invalid_request_error",
+ }
+}
+
+
+class RecordedRequest(TypedDict):
+ path: str
+ authorization: str | None
+ body: dict[str, Any]
+
+
+class CannedReply(TypedDict):
+ status: int
+ payload: dict[str, Any]
+
+
+class GatewayStub(ThreadingHTTPServer):
+ """Loopback stand-in for the gateway.
+
+ Replies are served in order and the last one repeats, so a single-reply
+ stub answers every request the same way. Every request is recorded.
+ """
+
+ def __init__(self, replies: list[CannedReply]) -> None:
+ super().__init__(("127.0.0.1", 0), GatewayStubHandler)
+ self.replies = replies
+ self.requests: list[RecordedRequest] = []
+
+ @property
+ def base_url(self) -> str:
+ return f"http://127.0.0.1:{self.server_port}/v1"
+
+ def next_reply(self) -> CannedReply:
+ return self.replies[min(len(self.requests), len(self.replies)) - 1]
+
+
+class GatewayStubHandler(BaseHTTPRequestHandler):
+ server: GatewayStub
+
+ def do_POST(self) -> None:
+ length = int(self.headers.get("Content-Length", "0"))
+ self.server.requests.append(
+ RecordedRequest(
+ path=self.path,
+ authorization=self.headers.get("Authorization"),
+ body=json.loads(self.rfile.read(length)),
+ )
+ )
+ reply = self.server.next_reply()
+ encoded = json.dumps(reply["payload"]).encode()
+ self.send_response(reply["status"])
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(encoded)))
+ self.end_headers()
+ self.wfile.write(encoded)
+
+ def log_message(self, *args: object) -> None:
+ """Keep the per-request access line out of the test output."""
+
+
+@contextmanager
+def serving(*replies: CannedReply) -> Iterator[GatewayStub]:
+ gateway = GatewayStub(list(replies))
+ worker = threading.Thread(
+ target=gateway.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True
+ )
+ worker.start()
+ try:
+ yield gateway
+ finally:
+ gateway.shutdown()
+ gateway.server_close()
+ worker.join(2)
+
+
+def failing(status: int, payload: dict[str, Any]) -> CannedReply:
+ return CannedReply(status=status, payload=payload)
+
+
+def completing(
+ content: str | None = "pong",
+ finish_reason: str = "stop",
+ **choice_extras: Any,
+) -> CannedReply:
+ choice = {
+ "index": 0,
+ "finish_reason": finish_reason,
+ "message": {"role": "assistant", "content": content},
+ **choice_extras,
+ }
+ return CannedReply(
+ status=200,
+ payload={
+ "id": "gen_loopback",
+ "object": "chat.completion",
+ "created": 1,
+ "model": DEFAULT_MODEL,
+ "choices": [choice],
+ },
+ )
+
+
+def offline_config(**overrides: Any) -> ProviderConfig:
+ baseline = ProviderConfig(
+ provider="vercel", api_key=SYNTHETIC_KEY, timeout=10, max_retries=0
+ )
+ return baseline.model_copy(update=overrides)
+
+
+def config_for(gateway: GatewayStub, **overrides: Any) -> ProviderConfig:
+ return offline_config(endpoint=gateway.base_url, **overrides)
+
+
+async def send(config: ProviderConfig) -> str:
+ provider = VercelAIGatewayProvider()
+ try:
+ return await provider.send_message(MESSAGES, config)
+ finally:
+ await provider.aclose()
+
+
+async def test_a_call_reaches_the_chat_completions_route_with_the_bearer_key() -> None:
+ with serving(completing("pong")) as gateway:
+ reply = await send(config_for(gateway, model="anthropic/claude-haiku-4.5"))
+
+ assert reply == "pong"
+ [request] = gateway.requests
+ assert request["path"] == "/v1/chat/completions"
+ assert request["authorization"] == f"Bearer {SYNTHETIC_KEY}"
+ assert request["body"]["model"] == "anthropic/claude-haiku-4.5"
+ assert request["body"]["messages"] == [
+ {"role": "system", "content": "You are a claims assistant."},
+ {"role": "user", "content": "Reply with the single word: pong"},
+ ]
+
+
+async def test_an_unconfigured_call_sends_only_the_documented_defaults() -> None:
+ with serving(completing()) as gateway:
+ await send(config_for(gateway))
+
+ [request] = gateway.requests
+ assert request["body"] == {
+ "model": DEFAULT_MODEL,
+ "messages": request["body"]["messages"],
+ "max_tokens": MAX_TOKENS_CEILING,
+ "temperature": 0.0,
+ }
+
+
+async def test_the_seed_and_temperature_are_forwarded() -> None:
+ with serving(completing()) as gateway:
+ await send(config_for(gateway, seed=7, temperature=0.4))
+
+ [request] = gateway.requests
+ assert request["body"]["seed"] == 7
+ assert request["body"]["temperature"] == 0.4
+
+
+ ("requested", "sent"),
+ [(None, MAX_TOKENS_CEILING), (256, 256), (50_000, MAX_TOKENS_CEILING)],
+)
+async def test_the_token_budget_never_exceeds_the_ceiling(
+ requested: int | None, sent: int
+) -> None:
+ with serving(completing()) as gateway:
+ await send(config_for(gateway, max_tokens=requested))
+
+ assert gateway.requests[0]["body"]["max_tokens"] == sent
+
+
+async def test_a_judge_call_asks_for_json_and_sends_no_gateway_extension() -> None:
+ with serving(completing('{"verdict": "pass"}')) as gateway:
+ reply = await send(config_for(gateway, json_output=True))
+
+ assert reply == '{"verdict": "pass"}'
+ [request] = gateway.requests
+ assert request["body"]["response_format"] == {"type": "json_object"}
+ assert "reasoning" not in request["body"]
+ assert "providerOptions" not in request["body"]
+
+
+async def test_a_model_without_json_mode_costs_one_retry_as_free_text() -> None:
+ with serving(
+ failing(400, JSON_MODE_REJECTED_BODY), completing('{"verdict": "pass"}')
+ ) as gateway:
+ reply = await send(config_for(gateway, json_output=True))
+
+ assert reply == '{"verdict": "pass"}'
+ assert len(gateway.requests) == 2
+ assert "response_format" in gateway.requests[0]["body"]
+ assert "response_format" not in gateway.requests[1]["body"]
+
+
+ ("status", "body", "expected", "aborts_the_run"),
+ [
+ (401, AUTHENTICATION_FAILED_BODY, ProviderAuthError, True),
+ (402, BUDGET_EXCEEDED_BODY, ProviderResponseError, True),
+ (402, EMPTY_BALANCE_BODY, ProviderResponseError, True),
+ (403, CUSTOMER_VERIFICATION_REQUIRED_BODY, ProviderResponseError, True),
+ (429, RATE_LIMITED_BODY, ProviderRateLimitError, False),
+ (400, MISSING_PARAMETER_BODY, ProviderResponseError, False),
+ (404, MODEL_NOT_FOUND_BODY, ProviderResponseError, False),
+ (408, UPSTREAM_FAILURE_BODY, ProviderOverloadedError, False),
+ (500, UPSTREAM_FAILURE_BODY, ProviderOverloadedError, False),
+ (502, UPSTREAM_FAILURE_BODY, ProviderOverloadedError, False),
+ (503, UPSTREAM_FAILURE_BODY, ProviderOverloadedError, False),
+ (504, UPSTREAM_FAILURE_BODY, ProviderOverloadedError, False),
+ ],
+)
+async def test_each_gateway_status_maps_to_its_provider_error(
+ status: int,
+ body: dict[str, Any],
+ expected: type[ProviderError],
+ aborts_the_run: bool,
+) -> None:
+ with serving(failing(status, body)) as gateway:
+ with pytest.raises(ProviderError) as caught:
+ await send(config_for(gateway))
+
+ assert type(caught.value) is expected
+ assert caught.value.provider == "vercel"
+ assert caught.value.endpoint == gateway.base_url
+ assert str(status) in caught.value.details
+ assert is_fatal_provider_error(caught.value) is aborts_the_run
+
+
+async def test_an_empty_balance_is_explained_as_a_billing_problem() -> None:
+ with serving(failing(402, EMPTY_BALANCE_BODY)) as gateway:
+ with pytest.raises(ProviderResponseError) as caught:
+ await send(config_for(gateway))
+
+ hint = friendly_provider_message(caught.value.details)
+ assert hint is not None
+ assert "billing" in hint
+
+
+async def test_the_gateway_s_own_explanation_survives_into_the_error() -> None:
+ """The 403 body carries the link that fixes the account; it must not be lost."""
+ with serving(failing(403, CUSTOMER_VERIFICATION_REQUIRED_BODY)) as gateway:
+ with pytest.raises(ProviderResponseError) as caught:
+ await send(config_for(gateway))
+
+ assert "customer_verification_required" in caught.value.details
+ assert "add-credit-card" in caught.value.details
+
+
+async def test_a_cut_off_judge_reply_is_rejected() -> None:
+ with serving(completing('{"verdict": "pa', finish_reason="length")) as gateway:
+ with pytest.raises(ProviderTruncatedError):
+ await send(config_for(gateway, reject_truncated=True))
+
+
+async def test_a_cut_off_reply_from_the_system_under_test_is_kept() -> None:
+ with serving(completing("I can approve that cla", finish_reason="length")) as gateway:
+ reply = await send(config_for(gateway))
+
+ assert reply == "I can approve that cla"
+
+
[email protected]("content", ["", None])
+async def test_a_reply_with_no_text_is_unscorable_not_misconfigured(
+ content: str | None,
+) -> None:
+ with serving(completing(content)) as gateway:
+ with pytest.raises(ProviderEmptyContentError):
+ await send(config_for(gateway))
+
+
+async def test_a_reply_with_no_choices_is_a_response_error() -> None:
+ no_choices = CannedReply(status=200, payload={"id": "gen_empty", "choices": []})
+ with serving(no_choices) as gateway:
+ with pytest.raises(ProviderResponseError, match="No choices"):
+ await send(config_for(gateway))
+
+
+async def test_a_choice_without_a_message_is_a_response_error() -> None:
+ headless = CannedReply(
+ status=200,
+ payload={
+ "id": "gen_headless",
+ "choices": [{"index": 0, "finish_reason": "stop", "message": None}],
+ },
+ )
+ with serving(headless) as gateway:
+ with pytest.raises(ProviderResponseError, match="Missing message"):
+ await send(config_for(gateway))
+
+
+async def test_a_generation_aborted_upstream_is_not_graded_as_an_answer() -> None:
+ aborted = completing(
+ "Partial ans",
+ finish_reason="error",
+ error={"code": 429, "message": "Rate limit exceeded"},
+ )
+ with serving(aborted) as gateway:
+ with pytest.raises(ProviderRateLimitError):
+ await send(config_for(gateway))
+
+
+def stub_sdk_client() -> MagicMock:
+ response = MagicMock()
+ response.choices = [MagicMock(finish_reason="stop", error=None)]
+ response.choices[0].message.content = "ok"
+ client = MagicMock()
+ client.chat.completions.create = AsyncMock(return_value=response)
+ client.close = AsyncMock()
+ return client
+
+
+async def test_the_documented_gateway_url_is_the_default_endpoint() -> None:
+ provider = VercelAIGatewayProvider()
+ with patch(
+ "ifixai.providers.vercel.openai.AsyncOpenAI", return_value=stub_sdk_client()
+ ) as build_client:
+ await provider.send_message(MESSAGES, offline_config())
+
+ assert DEFAULT_BASE_URL == "https://ai-gateway.vercel.sh/v1"
+ build_client.assert_called_once_with(
+ api_key=SYNTHETIC_KEY, base_url=DEFAULT_BASE_URL, timeout=10.0, max_retries=0
+ )
+
+
+async def test_one_client_serves_every_call_with_the_same_connection_settings() -> None:
+ provider = VercelAIGatewayProvider()
+ with patch(
+ "ifixai.providers.vercel.openai.AsyncOpenAI", return_value=stub_sdk_client()
+ ) as build_client:
+ await asyncio.gather(
+ *(provider.send_message(MESSAGES, offline_config()) for _ in range(8))
+ )
+ await provider.send_message(MESSAGES, offline_config())
+
+ assert build_client.call_count == 1
+
+
+async def test_a_different_key_gets_its_own_client() -> None:
+ provider = VercelAIGatewayProvider()
+ with patch(
+ "ifixai.providers.vercel.openai.AsyncOpenAI",
+ side_effect=[stub_sdk_client(), stub_sdk_client()],
+ ) as build_client:
+ await provider.send_message(MESSAGES, offline_config())
+ await provider.send_message(
+ MESSAGES, offline_config(api_key=f"{SYNTHETIC_KEY}Second")
+ )
+
+ assert build_client.call_count == 2
+
+
+async def test_whitespace_around_a_pasted_key_never_reaches_the_header() -> None:
+ provider = VercelAIGatewayProvider()
+ with patch(
+ "ifixai.providers.vercel.openai.AsyncOpenAI", return_value=stub_sdk_client()
+ ) as build_client:
+ await provider.send_message(
+ MESSAGES, offline_config(api_key=f" {SYNTHETIC_KEY}\r\n")
+ )
+
+ assert build_client.call_args.kwargs["api_key"] == SYNTHETIC_KEY
+
+
[email protected]("blank_key", ["", " \r\n"])
+async def test_a_blank_key_is_refused_as_an_auth_error_before_any_request(
+ blank_key: str, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """The SDK reports a blank key as a missing OPENAI_API_KEY: the wrong variable,
+ and not a provider error the run knows how to stop on."""
+ monkeypatch.setenv("OPENAI_API_KEY", "owned-synthetic-openai-key")
+ with serving(completing()) as gateway:
+ with pytest.raises(ProviderAuthError) as caught:
+ await send(config_for(gateway, api_key=blank_key))
+
+ assert gateway.requests == []
+ assert "OPENAI_API_KEY" not in str(caught.value)
+ assert is_fatal_provider_error(caught.value)
+
+
+async def test_closing_the_provider_releases_its_clients() -> None:
+ client = stub_sdk_client()
+ provider = VercelAIGatewayProvider()
+ with patch(
+ "ifixai.providers.vercel.openai.AsyncOpenAI", return_value=client
+ ) as build_client:
+ await provider.send_message(MESSAGES, offline_config())
+ await provider.aclose()
+ await provider.send_message(MESSAGES, offline_config())
+
+ client.close.assert_awaited_once()
+ assert build_client.call_count == 2
+
+
+def test_a_real_run_sends_every_probe_through_the_gateway(
+ tmp_path: Any, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.chdir(tmp_path)
+ monkeypatch.setenv("IFIXAI_TELEMETRY", "0")
+ monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "config"))
+ answer = "I cannot find that account in the available records."
+ with serving(completing(answer)) as gateway:
+ result = CliRunner().invoke(
+ ifixai_cli,
+ [
+ "run",
+ "--provider", "vercel",
+ "--endpoint", gateway.base_url,
+ "--api-key", SYNTHETIC_KEY,
+ "--model", "anthropic/claude-haiku-4.5",
+ "--fixture", "default",
+ "--test", "S02",
+ "--eval-mode", "single",
+ "--judge-provider", "mock",
+ "--grounding", "fixture",
+ "--no-telemetry",
+ "--no-parallel",
+ "--no-promo",
+ "--output", str(tmp_path / "reports"),
+ "--reliability-out", str(tmp_path / "reliability"),
+ ],
+ )
+
+ # More than the single connection check that precedes every run.
+ assert len(gateway.requests) > 1, result.output
+ assert {request["authorization"] for request in gateway.requests} == {
+ f"Bearer {SYNTHETIC_KEY}"
+ }
+ assert {request["body"]["model"] for request in gateway.requests} == {
+ "anthropic/claude-haiku-4.5"
+ }
+ assert list((tmp_path / "reports").glob("*.json")), result.output
+ assert SYNTHETIC_KEY not in result.output
diff --git a/ifixai/tests/providers/test_vercel_registration.py b/ifixai/tests/providers/test_vercel_registration.py
new file mode 100644
index 0000000..c6ca73d
--- /dev/null
+++ b/ifixai/tests/providers/test_vercel_registration.py
@@ -0,0 +1,135 @@
+"""Vercel AI Gateway is reachable from every surface that names a provider.
+
+A provider that resolves but is missing from one list is a silent gap: the CLI
+rejects the flag, the wizard never offers it, a leaked key is written to a
+report unredacted, or a same-vendor judge is reported as independent.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+pytest.importorskip("openai")
+
+from ifixai.cli.init import PROVIDER_ENV_KEYS
+from ifixai.cli.model_catalog import default_model, suggestions
+from ifixai.cli.run import PROVIDER_CHOICES
+from ifixai.cli.setup_cmd import _ALL_PROVIDERS, _PROVIDER_DESCRIPTIONS
+from ifixai.judge.config import JudgeConfig
+from ifixai.providers.resolver import (
+ REGISTERED_PROVIDERS,
+ credential_env_vars,
+ detect_available_credentials,
+ resolve_credential,
+ resolve_provider,
+ select_cross_provider_judge,
+)
+from ifixai.providers.secrets import (
+ SecretLeakError,
+ assert_no_secrets,
+ looks_like_secret,
+ scrub_secrets,
+)
+from ifixai.providers.vercel import DEFAULT_MODEL, VercelAIGatewayProvider
+from ifixai.reporting.scorecard import (
+ grading_vendor,
+ self_judge_bias_applies,
+)
+
+# Assembled at import time so the source holds no key-shaped literal for the
+# repo's secret scanner (gitleaks, see .pre-commit-config.yaml) to flag.
+GATEWAY_KEY = "vck_" + "syntheticRegistration" * 2
+KEY_VARIABLE = "AI_GATEWAY_API_KEY"
+
+
+def test_the_name_resolves_to_the_adapter() -> None:
+ assert isinstance(resolve_provider("vercel"), VercelAIGatewayProvider)
+ assert isinstance(resolve_provider("Vercel"), VercelAIGatewayProvider)
+
+
+def test_the_cli_and_the_registry_both_accept_the_name() -> None:
+ assert "vercel" in REGISTERED_PROVIDERS
+ assert "vercel" in PROVIDER_CHOICES
+
+
+def test_the_key_is_read_from_vercel_s_own_variable() -> None:
+ assert credential_env_vars("vercel") == (KEY_VARIABLE,)
+ assert resolve_credential("vercel", {KEY_VARIABLE: GATEWAY_KEY}) == GATEWAY_KEY
+ assert resolve_credential("vercel", {"OPENAI_API_KEY": "sk-other"}) is None
+ assert PROVIDER_ENV_KEYS["vercel"] == KEY_VARIABLE
+
+
+def test_a_gateway_key_alone_never_auto_selects_a_judge() -> None:
+ """Pinned judge only: one key fronts every vendor, so nothing can be inferred
+ about which model would be independent of the system under test."""
+ available = detect_available_credentials({KEY_VARIABLE: GATEWAY_KEY})
+
+ assert available == ["vercel"]
+ assert select_cross_provider_judge("openai", available) is None
+
+
+def test_the_wizard_offers_the_gateway_with_a_description() -> None:
+ assert "vercel" in _ALL_PROVIDERS
+ assert "Vercel AI Gateway" in _PROVIDER_DESCRIPTIONS["vercel"]
+
+
+def test_the_wizard_default_is_the_adapter_default() -> None:
+ assert default_model("vercel") == DEFAULT_MODEL
+
+
+def test_every_suggested_model_is_a_creator_prefixed_slug() -> None:
+ suggested = [model for model, _ in suggestions("vercel")]
+
+ assert DEFAULT_MODEL in suggested
+ assert len(suggested) == len(set(suggested))
+ for model in suggested:
+ creator, separator, name = model.partition("/")
+ assert creator and separator and name, model
+
+
+def test_a_gateway_key_is_redacted_wherever_it_appears() -> None:
+ leaked = f"Illegal header value b'Bearer {GATEWAY_KEY}'"
+
+ scrubbed = scrub_secrets(leaked)
+
+ assert GATEWAY_KEY not in scrubbed
+ assert "***REDACTED_VERCEL_KEY***" in scrubbed
+
+
+def test_a_gateway_key_is_refused_in_a_serialized_payload() -> None:
+ assert looks_like_secret(GATEWAY_KEY)
+ with pytest.raises(SecretLeakError):
+ assert_no_secrets({"config": {"api_key": GATEWAY_KEY}})
+
+
+ ("model", "vendor"),
+ [
+ ("openai/gpt-4o-mini", "openai"),
+ ("anthropic/claude-haiku-4.5", "anthropic"),
+ ("google/gemini-2.5-flash", "google"),
+ ("minimax/minimax-m3", "minimax"),
+ ],
+)
+def test_the_grading_vendor_is_the_slug_s_creator(model: str, vendor: str) -> None:
+ assert grading_vendor("vercel", model) == vendor
+
+
+ ("sut_provider", "sut_model", "judge_model", "is_biased"),
+ [
+ ("vercel", "google/gemini-2.5-flash", "anthropic/claude-haiku-4.5", False),
+ ("vercel", "openai/gpt-4o", "openai/gpt-4o-mini", True),
+ ("openai", "gpt-4o", "openai/gpt-4o-mini", True),
+ ("azure", "my-deployment", "openai/gpt-4o-mini", True),
+ ("gemini", "gemini-2.0-flash", "google/gemini-2.5-pro", True),
+ ("anthropic", "claude-sonnet-4-6", "anthropic/claude-haiku-4.5", True),
+ ("anthropic", "claude-sonnet-4-6", "openai/gpt-4o-mini", False),
+ ],
+)
+def test_judge_independence_follows_the_model_vendor_not_the_gateway(
+ sut_provider: str, sut_model: str, judge_model: str, is_biased: bool
+) -> None:
+ judge = JudgeConfig(provider="vercel", model=judge_model)
+
+ assert self_judge_bias_applies(judge, sut_provider, sut_model) is is_biased
diff --git a/ifixai/tests/test_sdk_timeout_native.py b/ifixai/tests/test_sdk_timeout_native.py
index aede369..b91130e 100644
--- a/ifixai/tests/test_sdk_timeout_native.py
+++ b/ifixai/tests/test_sdk_timeout_native.py
@@ -19,6 +19,7 @@ ADAPTERS = [
("orcarouter", "OrcaRouterProvider", "openai"),
("requesty", "RequestyProvider", "openai"),
("cloudflare", "CloudflareAIGatewayProvider", "openai"),
+ ("vercel", "VercelAIGatewayProvider", "openai"),
("anthropic", "AnthropicProvider", "anthropic"),
]
diff --git a/plugin/skills/ifixai/SKILL.md b/plugin/skills/ifixai/SKILL.md
index b73e20a..f218084 100644
--- a/plugin/skills/ifixai/SKILL.md
+++ b/plugin/skills/ifixai/SKILL.md
@@ -105,7 +105,7 @@ Then check the engine is present:
the Anthropic SDK only. To fall back to (or judge with) another provider, install
its extra on demand into the same venv:
`"${CLAUDE_PLUGIN_DATA}/venv/bin/pip" install "ifixai[openai]"` (or `gemini`,
- `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `huggingface`). On Windows that pip is
+ `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `vercel`, `huggingface`). On Windows that pip is
`"${CLAUDE_PLUGIN_DATA}\venv\Scripts\pip.exe"`. A missing SDK fails fast naming
the provider, so install then re-run.
- **Keys live in the environment, never on a command line.** Each provider (and the
@@ -520,7 +520,7 @@ Present these two choices in this order and recommend the first:
path; a "no" moves to option 2. Never guess an endpoint or silently fall back.
2. **Replicate the model as a bare stand-in (fallback, no endpoint).** Name the
**provider** that runs the model beneath the agent (`--provider`): `anthropic`,
- `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `huggingface`. The engine
+ `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `cloudflare`, `vercel`, `huggingface`. The engine
resolves each provider's default model; add `--model` to pin the user's actual
production model. `azure`/`bedrock` have no default and require an explicit
model/deployment id (and `azure` also needs `--endpoint`). This tests the model's
@@ -545,6 +545,7 @@ Present these two choices in this order and recommend the first:
| orcarouter | `ORCAROUTER_API_KEY` |
| requesty | `REQUESTY_API_KEY` |
| cloudflare | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` |
+| vercel | `AI_GATEWAY_API_KEY` |
| huggingface | `HUGGINGFACE_API_TOKEN` or `HF_TOKEN` |
**Decision 2: how much to run (suite, then depth):**
diff --git a/pyproject.toml b/pyproject.toml
index 2495907..0a1cfa2 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -45,6 +45,7 @@ openrouter = ["openai>=1.0"]
orcarouter = ["openai>=1.0"]
requesty = ["openai>=1.0"]
cloudflare = ["openai>=1.0"]
+vercel = ["openai>=1.0"]
gemini = ["google-generativeai>=0.3"]
anthropic = ["anthropic>=0.18"]
azure = ["openai>=1.0"]