Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

Explain an HTTP 404 from the default LLM model (#514)

Christopher Kevin93d595eb724d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3a8f89fa..5e651005 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -81,6 +81,9 @@ All notable changes to SkillEvaluator are documented in this file.
### Changed
+- The HTTP 404 message for an unavailable LLM model now says the provider's default
+ may not be callable with the key and points to `SKILL_EVAL_LLM_MODEL` (`llm-model`
+ in the reusable workflow); `ci/README.md` notes the same.
- Update the managed SkillSpector scanner and Tier 1 CI image from 2.13.3 to
the latest release, 2.13.4, with matching verified wheel and source hashes.
This fixes false LP1 findings for scoped `allowed-tools` grants and improves
diff --git a/ci/README.md b/ci/README.md
index 26a47de1..8702d070 100644
--- a/ci/README.md
+++ b/ci/README.md
@@ -322,7 +322,10 @@ workflow and GitHub artifact metadata also remain required.
The generated `skill-checks-llm.yml` is a separate trusted manual run. Set
`paths`, choose `llm-provider` (`openai`, `anthropic` or `nv_build`), optionally
set `llm-model`, and explicitly map the repository secret `LLM_API_KEY`. Missing
-credentials fail explicitly. Avoid passing secrets to untrusted PR content;
+credentials fail explicitly. When `llm-model` is empty the provider's default model
+is used, and that model may not be callable with every key (for example, an NVIDIA
+Build key can list a model that still answers HTTP 404). If the LLM checks report
+"Model or endpoint unavailable", set `llm-model` to a model your key can call. Avoid passing secrets to untrusted PR content;
the basic PR workflow remains keyless. Provider billing and content transmission
apply. A configured LLM run is not proof of Tier 2 or Tier 3 evaluation.
diff --git a/src/skillevaluator/inference/diagnostics.py b/src/skillevaluator/inference/diagnostics.py
index 05f69abb..27028e2f 100644
--- a/src/skillevaluator/inference/diagnostics.py
+++ b/src/skillevaluator/inference/diagnostics.py
@@ -72,7 +72,9 @@ def llm_failure_diagnostic(exc: Exception) -> str:
remedy = "Check the selected LLM provider credentials and model access."
elif status in (404, 410):
remedy = (
- "Model or endpoint unavailable. Check SKILL_EVAL_LLM_MODEL against the provider's available models."
+ "Model or endpoint unavailable. If no model was set, the provider's default may not be callable "
+ "with this key; set SKILL_EVAL_LLM_MODEL (llm-model in CI) to a model the key can use. "
+ "Otherwise check SKILL_EVAL_LLM_MODEL against the provider's available models."
)
elif status == 429:
remedy = "The LLM provider rejected the request due to rate or quota limits. Check quota and retry later."
diff --git a/tests/inference/test_diagnostics.py b/tests/inference/test_diagnostics.py
index e8e6519b..1bc78be4 100644
--- a/tests/inference/test_diagnostics.py
+++ b/tests/inference/test_diagnostics.py
@@ -30,6 +30,15 @@ def test_http_status_without_echoing_response_or_request(status):
assert "example.invalid" not in diagnostic
[email protected]("status", [404, 410])
+def test_unavailable_model_points_to_the_model_setting(status):
+ response = httpx.Response(status, request=httpx.Request("POST", "https://example.invalid"))
+ diagnostic = llm_failure_diagnostic(APIStatusError("x", response=response, body=None))
+ assert "default may not be callable" in diagnostic
+ assert "SKILL_EVAL_LLM_MODEL" in diagnostic
+ assert "llm-model" in diagnostic
+
+
@pytest.mark.parametrize(
"body",
[