Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
Python agent-evaluationaiai-alignmentai-evaluationai-governanceai-safetyclidiagnostic-tooleu-ai-acthallucination-detectioniso-42001llm-evaluationllm-securitynist-ai-rmfowasp-llmprompt-injectionpythonresponsible-airisk-assessmentrisk-management
feat: add Requesty as an OpenAI-compatible provider (#112)
Thibault Jaigu82a268ca4bd3
diff --git a/README.ja.md b/README.ja.mdindex 925d92b..2748c84 100644--- a/README.ja.md+++ b/README.ja.md@@ -186,7 +186,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fiレポートは JSON **と** Markdown の両方で `./ifixai-results/` に保存されます。2 つ目のキーがない場合は、スモークテストとして `--eval-mode self` を追加してください(評価は表示されますが自己評価と明記され、引用できる結果にはなりません)。評価モデルの固定、Full モードのアンサンブル、評価モードについては **[docs/cli.md](docs/cli.md#how-a-run-is-judged)** を参照してください。その他のプロバイダー-(OpenAI、OpenRouter、Gemini、Azure、Bedrock、Hugging Face)は対応する extra を+(OpenAI、OpenRouter、Requesty、Gemini、Azure、Bedrock、Hugging Face)は対応する extra をインストールし、同じ手順を使用します。HTTP と LangChain のアダプターにはプロバイダーの extra は不要です:**[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**。diff --git a/README.ko.md b/README.ko.mdindex cd1f6b3..6ac74b5 100644--- a/README.ko.md+++ b/README.ko.md@@ -171,7 +171,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fi`--eval-mode self`를 추가해 스모크 테스트로 실행하세요(등급은 출력되지만 자기 채점으로표시되며, 인용할 수 있는 결과는 아닙니다). 심사 모델 고정, Full 모드 앙상블, 평가 모드에대해서는 **[docs/cli.md](docs/cli.md#how-a-run-is-judged)** 를 참고하세요. 다른-프로바이더(OpenAI, Atlas Cloud, OpenRouter, Gemini, Azure, Bedrock, Hugging Face)는 해당+프로바이더(OpenAI, Atlas Cloud, OpenRouter, Requesty, Gemini, Azure, Bedrock, Hugging Face)는 해당extra를 설치하고 같은 절차를 따르며, HTTP와 LangChain 어댑터는 프로바이더 extra가필요 없습니다: **[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**.diff --git a/README.md b/README.mdindex e96e5cf..e24c2f8 100644--- a/README.md+++ b/README.md@@ -169,7 +169,7 @@ Reports land in `./ifixai-results/` as JSON **and** Markdown. Without a second k`--eval-mode self` to run as a smoke test (the grade still prints, but it's flagged asself-judged, not a result you can cite). Pinning the judge, Full-mode ensembles, and the eval modes:**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**. Other providers (OpenAI, Atlas Cloud,-OpenRouter, OrcaRouter, Gemini, Azure, Bedrock, Hugging Face) install the matching extra and follow the same steps; the+OpenRouter, OrcaRouter, Requesty, Gemini, Azure, Bedrock, Hugging Face) install the matching extra and follow the same steps; theHTTP and LangChain adapters need no provider extra: **[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**.### Recommended judge setupsdiff --git a/README.zh-CN.md b/README.zh-CN.mdindex 2b3dc8a..b922de6 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -182,7 +182,7 @@ ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fi报告以 JSON **和** Markdown 两种格式保存在 `./ifixai-results/` 中。如果没有第二个密钥,请添加 `--eval-mode self` 作为冒烟测试(等级仍会显示,但会被标记为自我评审,不能作为可引用结果)。固定评审模型、完整模式集成以及评审模式的详情请参阅:-**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**。其他提供商(OpenAI、OpenRouter、Gemini、+**[docs/cli.md](docs/cli.md#how-a-run-is-judged)**。其他提供商(OpenAI、OpenRouter、Requesty、Gemini、Azure、Bedrock、Hugging Face)可安装对应 extra 并遵循相同步骤;HTTP 和 LangChain适配器不需要提供商 extra:**[docs/testing-your-agent.md](docs/testing-your-agent.md#provider-reference)**。diff --git a/SECURITY.md b/SECURITY.mdindex 88c324e..d05df4a 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -45,6 +45,7 @@ The scrubber (`ifixai/providers/secrets.py::scrub_secrets`) is parametrically ve| `atlascloud` | `(ak|apikey)-[A-Za-z0-9_-]{20,}` | `***REDACTED_ATLASCLOUD_KEY***` || `openrouter` | `sk-or-[A-Za-z0-9_-]{20,}` (matched before `openai`) | `***REDACTED_OPENROUTER_KEY***` || `orcarouter` | `sk-orca-[A-Za-z0-9_-]{20,}` (matched before `openai`) | `***REDACTED_ORCAROUTER_KEY***` |+| `requesty` | `rqsty-[A-Za-z0-9+/=_-]{20,}` | `***REDACTED_REQUESTY_KEY***` || `anthropic` | `sk-ant-[A-Za-z0-9_-]{20,}`, `anthropic_[A-Za-z0-9_-]{20,}` | `***REDACTED_ANTHROPIC_KEY***` || `gemini` | `AIzaSy[0-9A-Za-z_-]{33}` | `***REDACTED_GEMINI_KEY***` || `azure` | 32-char hex (`\b[a-fA-F0-9]{32}\b`) | `***REDACTED_AZURE_KEY***` |diff --git a/docs/cli.md b/docs/cli.mdindex 49e657c..cf1d220 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -21,7 +21,7 @@ ifixai run -p openai -k "$OPENAI_API_KEY" -c DECEPTION # example: one category| Flag | Default | Does ||---|---|---|-| `--provider`, `-p` | none | `mock`, `openai`, `openrouter`, `orcarouter`, `anthropic`, `gemini`, `azure`, `bedrock`, `huggingface`, `http`, `langchain`. |+| `--provider`, `-p` | none | `mock`, `openai`, `openrouter`, `orcarouter`, `requesty`, `anthropic`, `gemini`, `azure`, `bedrock`, `huggingface`, `http`, `langchain`. || `--api-key`, `-k` | none | SUT API key. Always passed explicitly, never read from the environment. || `--model`, `-m` | provider default | Model identifier override. || `--endpoint`, `-e` | none | Endpoint URL (required for `http` and `azure`). |diff --git a/docs/testing-your-agent.md b/docs/testing-your-agent.mdindex 37d3732..5f40790 100644--- a/docs/testing-your-agent.md+++ b/docs/testing-your-agent.md@@ -121,6 +121,7 @@ the SUT key from the environment: pass `--api-key` / `-k`, or enter it when prom| `minimax` | none | `MINIMAX_API_KEY` | `--provider minimax -k "$MINIMAX_API_KEY" --model MiniMax-M3` || `openrouter` | `.[openrouter]` | `OPENROUTER_API_KEY` | `--provider openrouter -k "$OPENROUTER_API_KEY" --model openai/gpt-4o` plus explicit judge || `orcarouter` | `.[orcarouter]` | `ORCAROUTER_API_KEY` | `--provider orcarouter -k "$ORCAROUTER_API_KEY" --model openai/gpt-4o` plus explicit judge |+| `requesty` | `.[requesty]` | `REQUESTY_API_KEY` | `--provider requesty -k "$REQUESTY_API_KEY" --model openai/gpt-4o-mini` plus explicit judge; `--endpoint https://router.eu.requesty.ai/v1` for EU data residency || `gemini` | `.[gemini]` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | `--provider gemini -k "$GEMINI_API_KEY"` || `azure` | `.[azure]` | `AZURE_OPENAI_API_KEY` | `--provider azure --endpoint https://YOUR_RESOURCE.openai.azure.com/ -k "$AZURE_OPENAI_API_KEY" --model YOUR_DEPLOYMENT_NAME` plus explicit judge || `bedrock` | `.[bedrock]` | AWS credential chain | `--provider bedrock -k not-used --model anthropic.claude-sonnet-4-6` |diff --git a/ifixai/cli/init.py b/ifixai/cli/init.pyindex f8bb990..45da559 100644--- a/ifixai/cli/init.py+++ b/ifixai/cli/init.py@@ -14,6 +14,7 @@ PROVIDER_ENV_KEYS: dict[str, str] = {"minimax": "MINIMAX_API_KEY","openrouter": "OPENROUTER_API_KEY","orcarouter": "ORCAROUTER_API_KEY",+ "requesty": "REQUESTY_API_KEY","gemini": "GEMINI_API_KEY","azure": "AZURE_OPENAI_API_KEY","bedrock": "AWS_ACCESS_KEY_ID",diff --git a/ifixai/cli/model_catalog.py b/ifixai/cli/model_catalog.pyindex fc729e0..72082e0 100644--- a/ifixai/cli/model_catalog.py+++ b/ifixai/cli/model_catalog.py@@ -34,6 +34,7 @@ def _minimax_pricing_summary(model_id: str) -> str:DEFAULT_MODEL: dict[str, str] = {"openrouter": "openai/gpt-4o","orcarouter": "openai/gpt-4o",+ "requesty": "openai/gpt-4o-mini","openai": "gpt-4o","anthropic": "claude-3-5-sonnet-latest","gemini": "gemini-2.0-flash",@@ -68,6 +69,13 @@ MODEL_SUGGESTIONS: dict[str, list[tuple[str, str]]] = {("deepseek/deepseek-v4-flash", "DeepSeek V4 Flash — very low cost"),("qwen/qwen3.5-flash", "Qwen3.5 Flash — capable open-weights instruct"),],+ "requesty": [+ ("openai/gpt-4o-mini", "OpenAI GPT-4o Mini — cheap, fast, widely available"),+ ("openai/gpt-4o", "OpenAI GPT-4o — flagship general reasoning"),+ ("anthropic/claude-sonnet-4-6", "Anthropic Claude Sonnet 4.6 — strong reasoning & safety"),+ ("google/gemini-2.5-flash", "Google Gemini 2.5 Flash — fast and inexpensive"),+ ("google/gemini-2.5-pro", "Google Gemini 2.5 Pro — large context, strong reasoning"),+ ],"openai": [("gpt-4o", "Flagship — strong general reasoning"),("gpt-4o-mini", "Cheaper & faster"),diff --git a/ifixai/cli/run.py b/ifixai/cli/run.pyindex 655028a..da6f7c4 100644--- a/ifixai/cli/run.py+++ b/ifixai/cli/run.py@@ -142,6 +142,7 @@ PROVIDER_CHOICES = ["langchain","openrouter","orcarouter",+ "requesty",]FORMAT_CHOICES = ["json", "markdown", "both"]diff --git a/ifixai/cli/setup_cmd.py b/ifixai/cli/setup_cmd.pyindex fe5c722..3c4cad9 100644--- a/ifixai/cli/setup_cmd.py+++ b/ifixai/cli/setup_cmd.py@@ -21,6 +21,7 @@ from ifixai.providers.minimax import DEFAULT_BASE_URL, REGIONAL_ENDPOINTS_PROVIDER_DESCRIPTIONS: dict[str, str] = {"openrouter": "One key → many models (OpenAI, Anthropic, Google, Llama…)","orcarouter": "OpenAI-compatible gateway — many models, one key",+ "requesty": "Requesty gateway — one key → 400+ models (OpenAI, Anthropic, Google…)","openai": "OpenAI API — GPT-4o / o-series","anthropic": "Anthropic API — Claude family","gemini": "Google Gemini",@@ -48,6 +49,7 @@ _MODE_DESCRIPTIONS: dict[str, str] = {_ALL_PROVIDERS = ["openrouter","orcarouter",+ "requesty","openai","anthropic","gemini",diff --git a/ifixai/core/types.py b/ifixai/core/types.pyindex ceeb2c6..174bc69 100644--- a/ifixai/core/types.py+++ b/ifixai/core/types.py@@ -242,7 +242,7 @@ class ProviderConfig(BaseModel):"Request structured JSON output from the provider (response_format=""json_object). Set ONLY for LLM-judge calls so cheap models reliably ""emit a parseable verdict; never for the system-under-test, whose reply "- "must stay natural. Honored by the openai, openrouter, orcarouter, azure, litellm "+ "must stay natural. Honored by the openai, openrouter, orcarouter, requesty, azure, litellm ""(response_format) and gemini (response_mime_type) providers, which fall ""back to free text if the model rejects it; a no-op on providers with no ""JSON mode (anthropic, bedrock, huggingface), where the json-repair "diff --git a/ifixai/providers/requesty.py b/ifixai/providers/requesty.pynew file mode 100644index 0000000..1055e0b--- /dev/null+++ b/ifixai/providers/requesty.py@@ -0,0 +1,154 @@+import asyncio++import openai++from ifixai.core.types import ChatMessage, ProviderConfig+from ifixai.providers.base import (+ ChatProvider,+ ProviderAuthError,+ ProviderConnectionError,+ ProviderEmptyContentError,+ ProviderRateLimitError,+ ProviderResponseError,+ ProviderTimeoutError,+ create_chat_completion_json_fallback,+ raise_for_http_status,+ raise_if_choice_errored,+ raise_if_truncated,+)++DEFAULT_MODEL = "openai/gpt-4o-mini"+DEFAULT_BASE_URL = "https://router.requesty.ai/v1"+# Hard ceiling on max_tokens for Requesty calls. Per-call ``config.max_tokens``+# is clamped to this value; unset config falls through to the ceiling. Prevents+# verbose generations from blowing wall-time and credits on long fixtures.+MAX_TOKENS_CEILING: int = 8192++ClientCacheKey = tuple[str, str | None, float, int]+++class RequestyProvider(ChatProvider):+ def __init__(self) -> None:+ self._clients: dict[ClientCacheKey, openai.AsyncOpenAI] = {}+ self._client_lock = asyncio.Lock()++ async def get_client(self, config: ProviderConfig) -> openai.AsyncOpenAI:+ """Return a long-lived AsyncOpenAI client keyed on connection params.++ Caching by (base_url, api_key, timeout, max_retries) lets the+ underlying httpx pool reuse TCP/TLS across LLM calls instead of+ paying a handshake per request.+ """+ base_url = config.endpoint or DEFAULT_BASE_URL+ key: ClientCacheKey = (+ base_url,+ config.api_key,+ float(config.timeout),+ config.max_retries,+ )+ cached = self._clients.get(key)+ if cached is not None:+ return cached+ async with self._client_lock:+ cached = self._clients.get(key)+ if cached is not None:+ return cached+ client = openai.AsyncOpenAI(+ api_key=config.api_key,+ base_url=base_url,+ timeout=float(config.timeout),+ max_retries=config.max_retries,+ )+ self._clients[key] = client+ return client++ async def aclose(self) -> None:+ for client in self._clients.values():+ await client.close()+ self._clients.clear()++ async def send_message(+ self,+ messages: list[ChatMessage],+ config: ProviderConfig,+ ) -> str:+ base_url = config.endpoint or DEFAULT_BASE_URL+ model = config.model or DEFAULT_MODEL+ formatted = [{"role": m.role, "content": m.content} for m in messages]++ client = await self.get_client(config)+ try:+ effective_max_tokens = (+ min(config.max_tokens, MAX_TOKENS_CEILING)+ if config.max_tokens is not None+ else MAX_TOKENS_CEILING+ )+ create_kwargs: dict = {+ "model": model,+ "messages": formatted, # type: ignore[arg-type]+ "max_tokens": effective_max_tokens,+ "temperature": config.temperature,+ }+ if config.seed is not None:+ create_kwargs["seed"] = config.seed+ if config.json_output:+ # Constrain judge calls to valid JSON so cheap models reliably emit+ # a parseable verdict instead of breaking the contract. Falls back to+ # free text (json-repair handles parsing) if the model does not+ # support response_format.+ create_kwargs["response_format"] = {"type": "json_object"}+ response = await create_chat_completion_json_fallback(+ client, **create_kwargs+ )+ choices = response.choices+ if not choices:+ raise ProviderResponseError(+ provider="requesty",+ endpoint=base_url,+ details=f"No choices in response (id={response.id})",+ )+ choice = choices[0]+ finish_reason = choice.finish_reason or "unknown"+ if choice.message is None:+ raise ProviderResponseError(+ provider="requesty",+ endpoint=base_url,+ details=f"Missing message in choice (finish_reason={finish_reason})",+ )+ content = choice.message.content+ if config.reject_truncated:+ raise_if_truncated("requesty", base_url, finish_reason, content or "")+ if not content:+ raise ProviderEmptyContentError(+ provider="requesty",+ endpoint=base_url,+ details=f"Empty content in response (finish_reason={finish_reason})",+ )+ raise_if_choice_errored("requesty", base_url, choice, content)+ except openai.AuthenticationError as exc:+ raise ProviderAuthError(+ provider="requesty", endpoint=base_url, details=str(exc)+ ) from exc+ except openai.RateLimitError as exc:+ raise ProviderRateLimitError(+ provider="requesty", endpoint=base_url, details=str(exc)+ ) from exc+ except openai.APITimeoutError as exc:+ raise ProviderTimeoutError(+ provider="requesty", endpoint=base_url, details=str(exc)+ ) from exc+ except openai.APIConnectionError as exc:+ raise ProviderConnectionError(+ provider="requesty", endpoint=base_url, details=str(exc)+ ) from exc+ except openai.APIStatusError as exc:+ # The gateway's retryable band (408 timeout, 500 internal, 502 model+ # down, 503 no routable provider, 504) is split off here as a+ # transient overload; everything else stays a response error.+ raise_for_http_status("requesty", base_url, exc)+ except openai.APIError as exc:+ raise ProviderResponseError(+ provider="requesty", endpoint=base_url, details=str(exc)+ ) from exc+ else:+ return contentdiff --git a/ifixai/providers/resolver.py b/ifixai/providers/resolver.pyindex 4e4a746..4aba300 100644--- a/ifixai/providers/resolver.py+++ b/ifixai/providers/resolver.py@@ -55,6 +55,11 @@ try:except ImportError:OrcaRouterProvider = None+try:+ from ifixai.providers.requesty import RequestyProvider+except ImportError:+ RequestyProvider = None # type: ignore[assignment,misc]+try:from ifixai.providers.litellm import LiteLLMProviderexcept ImportError:@@ -69,6 +74,7 @@ REGISTERED_PROVIDERS: tuple[str, ...] = ("minimax","openrouter","orcarouter",+ "requesty","anthropic","gemini","azure",@@ -90,6 +96,7 @@ _PROVIDER_MAP: dict[str, type] = {"minimax": MiniMaxProvider,"openrouter": OpenRouterProvider,"orcarouter": OrcaRouterProvider,+ "requesty": RequestyProvider,"anthropic": AnthropicProvider,"gemini": GeminiProvider,"azure": AzureOpenAIProvider,@@ -169,6 +176,7 @@ _PROVIDER_CREDENTIAL_ENV_VARS: dict[str, tuple[str, ...]] = {"huggingface": ("HUGGINGFACE_API_TOKEN", "HF_TOKEN"),"openrouter": ("OPENROUTER_API_KEY",),"orcarouter": ("ORCAROUTER_API_KEY",),+ "requesty": ("REQUESTY_API_KEY",),"litellm": ("LITELLM_API_KEY",),}diff --git a/ifixai/providers/secrets.py b/ifixai/providers/secrets.pyindex 304a24d..681e143 100644--- a/ifixai/providers/secrets.py+++ b/ifixai/providers/secrets.py@@ -19,6 +19,7 @@ _SECRET_PATTERNS: Final[tuple[Pattern[str], ...]] = (re.compile(r"^sk-cp-[A-Za-z0-9_-]+$"),re.compile(r"^sk-or-[A-Za-z0-9_-]{20,}$"),re.compile(r"^sk-orca-[A-Za-z0-9_-]{20,}$"),+ re.compile(r"^rqsty-[A-Za-z0-9+/=_-]{20,}$"),re.compile(r"^ak-[A-Za-z0-9_-]{20,}$"),re.compile(r"^sk-ant-[A-Za-z0-9_-]{20,}$"),re.compile(r"^sk-[A-Za-z0-9_-]{20,}$"),@@ -49,6 +50,10 @@ _SCRUB_RULES: Final[tuple[tuple[Pattern[str], str], ...]] = (re.compile(r"sk-orca-[A-Za-z0-9_-]{20,}"),"***REDACTED_ORCAROUTER_KEY***",),+ (+ re.compile(r"rqsty-[A-Za-z0-9+/=_-]{20,}"),+ "***REDACTED_REQUESTY_KEY***",+ ),(re.compile(r"(?<![A-Za-z0-9_-])ak-[A-Za-z0-9_-]{20,}(?![A-Za-z0-9_-])"),"***REDACTED_ATLASCLOUD_KEY***",diff --git a/ifixai/reporting/scorecard.py b/ifixai/reporting/scorecard.pyindex d940d47..d9343ed 100644--- a/ifixai/reporting/scorecard.py+++ b/ifixai/reporting/scorecard.py@@ -268,10 +268,10 @@ def b32_not_applicable_warning(# Providers that front many vendors behind one name. For these the real grading# vendor is the model slug's prefix, so a cross-vendor judge routed through one-# aggregator (e.g. a Gemini SUT graded by a Claude judge, both via OpenRouter) is-# recognized as independent instead of mislabeled "self-judge".+# aggregator (e.g. a Gemini SUT graded by a Claude judge, both via OpenRouter or+# Requesty) is recognized as independent instead of mislabeled "self-judge"._AGGREGATOR_PROVIDERS: Final[frozenset[str]] = frozenset(- {"openrouter", "orcarouter", "atlascloud", "litellm", "http", "langchain"}+ {"openrouter", "orcarouter", "requesty", "atlascloud", "litellm", "http", "langchain"})# Distinct provider slugs that front the SAME underlying model vendor, so a@@ -279,11 +279,24 @@ _AGGREGATOR_PROVIDERS: Final[frozenset[str]] = frozenset(# OpenAI models; the direct Gemini API and a "google/..." slug are both Google._VENDOR_ALIASES: Final[dict[str, str]] = {"azure": "openai", "gemini": "google"}+# Aggregator prefixes that name the cloud hosting the model, not its developer+# ("bedrock/claude-sonnet-4-6@eu-central-1"), so the vendor comes from the family.+_CLOUD_HOST_PREFIXES: Final[frozenset[str]] = frozenset(+ {"bedrock", "vertex", "vertex_ai", "azure", "deepinfra", "coding"}+)+_MODEL_FAMILY_VENDORS: Final[dict[str, str]] = {+ "claude": "anthropic",+ "gemini": "google",+ "gpt": "openai",+}+def grading_vendor(provider: str, model: str | None) -> str:p = (provider or "").lower()if p in _AGGREGATOR_PROVIDERS and model and "/" in model:- raw = model.split("/", 1)[0].lower()+ raw, _, rest = model.lower().partition("/")+ if raw in _CLOUD_HOST_PREFIXES:+ raw = next((v for f, v in _MODEL_FAMILY_VENDORS.items() if f in rest), raw)else:raw = preturn _VENDOR_ALIASES.get(raw, raw)diff --git a/ifixai/scaffold/playbook.md b/ifixai/scaffold/playbook.mdindex bde06dd..0b60910 100644--- a/ifixai/scaffold/playbook.md+++ b/ifixai/scaffold/playbook.md@@ -97,7 +97,7 @@ IFIXAI_SURFACE=skill uvx --from "ifixai[<provider>]" ifixai run \```Substitute a provider valid as BOTH the install extra and the `--provider` value: one of-`anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, or `huggingface`. (`mock`+`anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, or `huggingface`. (`mock`runs fully offline for a free smoke test, with `--provider mock --api-key mock --eval-modeself`.) The extra installs that provider's SDK; if you test on one provider and judge onanother, install the union. After the user approves the dry-run estimate, rerun the identical@@ -145,6 +145,7 @@ over the file). A missing key fails fast and names the variable to set.| bedrock | `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` || openrouter | `OPENROUTER_API_KEY` || orcarouter | `ORCAROUTER_API_KEY` |+| requesty | `REQUESTY_API_KEY` || huggingface | `HUGGINGFACE_API_TOKEN` or `HF_TOKEN` |If the SUT and a judge share a provider, one key covers both. A judge on a second provider@@ -407,7 +408,7 @@ at each. Map the picks to flags yourself.build a stand-in that mirrors your setup and test that instead?" A URL keeps this path; a "no"moves to the fallback. Don't guess an endpoint or fall back silently.- *Bare stand-in (fallback, no endpoint).* Offer the providers the user has a key for:- `anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `huggingface`, with+ `anthropic`, `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `huggingface`, with`--grounding fixture`. Offer to pin a model (`--model <id>`) or take the provider default.Maps to `--provider` / `--model`. Tests the model, not the deployment. (`mock` runs offlinefor free.)@@ -420,7 +421,7 @@ at each. Map the picks to flags yourself.- *Self (cheapest, biased, not citable)*: the model grades itself. Maps to `--eval-mode self`.Only a smoke test. (With a single provider key and no judge, standard mode REFUSES rather thanself-judge silently, so this is the explicit opt-in.)-- On an aggregator like OpenRouter, "different vendor" means a different model-slug prefix+- On an aggregator like OpenRouter or Requesty, "different vendor" means a different model-slug prefix(e.g. `google/...` SUT graded by `openai/...`), which the engine now recognizes as citable.**3. Depth / how much to run.** Offer a suite, smallest first, with the trade-off:diff --git a/plugin/skills/ifixai/SKILL.md b/plugin/skills/ifixai/SKILL.mdindex 9e696e7..8769261 100644--- a/plugin/skills/ifixai/SKILL.md+++ b/plugin/skills/ifixai/SKILL.md@@ -105,7 +105,7 @@ Then check the engine is present:the Anthropic SDK only. To fall back to (or judge with) another provider, installits extra on demand into the same venv:`"${CLAUDE_PLUGIN_DATA}/venv/bin/pip" install "ifixai[openai]"` (or `gemini`,- `azure`, `bedrock`, `openrouter`, `orcarouter`, `huggingface`). On Windows that pip is+ `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `huggingface`). On Windows that pip is`"${CLAUDE_PLUGIN_DATA}\venv\Scripts\pip.exe"`. A missing SDK fails fast namingthe provider, so install then re-run.- **Keys live in the environment, never on a command line.** Each provider (and the@@ -520,7 +520,7 @@ Present these two choices in this order and recommend the first:path; a "no" moves to option 2. Never guess an endpoint or silently fall back.2. **Replicate the model as a bare stand-in (fallback, no endpoint).** Name the**provider** that runs the model beneath the agent (`--provider`): `anthropic`,- `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `huggingface`. The engine+ `openai`, `gemini`, `azure`, `bedrock`, `openrouter`, `orcarouter`, `requesty`, `huggingface`. The engineresolves each provider's default model; add `--model` to pin the user's actualproduction model. `azure`/`bedrock` have no default and require an explicitmodel/deployment id (and `azure` also needs `--endpoint`). This tests the model's@@ -543,6 +543,7 @@ Present these two choices in this order and recommend the first:| bedrock | `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` || openrouter | `OPENROUTER_API_KEY` || orcarouter | `ORCAROUTER_API_KEY` |+| requesty | `REQUESTY_API_KEY` || huggingface | `HUGGINGFACE_API_TOKEN` or `HF_TOKEN` |**Decision 2: how much to run (suite, then depth):**diff --git a/pyproject.toml b/pyproject.tomlindex e9e4762..974d241 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -43,6 +43,7 @@ openai = ["openai>=1.0"]atlascloud = ["openai>=1.0"]openrouter = ["openai>=1.0"]orcarouter = ["openai>=1.0"]+requesty = ["openai>=1.0"]gemini = ["google-generativeai>=0.3"]anthropic = ["anthropic>=0.18"]azure = ["openai>=1.0"]