Skip to content

Create, Evaluate, and Connect AI Skills

Update prompts.py

Xu Haominga16badda8f0c
diff --git a/skillnet-ai/src/skillnet_ai/prompts.py b/skillnet-ai/src/skillnet_ai/prompts.py
index 92ea868..c311aa9 100644
--- a/skillnet-ai/src/skillnet_ai/prompts.py
+++ b/skillnet-ai/src/skillnet_ai/prompts.py
@@ -204,6 +204,7 @@ Evaluation dimensions and how to judge them (apply these rules even if the overa
- **Do NOT rate Poor solely because "No runnable python scripts found"**. Many skills (security guidelines, ideation, policies, design workflows) are instruction-only: the agent reads SKILL.md and follows the guidance with typical tools. For such skills, if the instructions are clear and actionable, executability should be Good.
- If script execution fails due to an obvious documentation placeholder in an example command (e.g., tokens like "[options]", "<file>", "<pattern>", "{{path}}") or an argument parsing error caused by such placeholders, do NOT automatically set executability to Poor. Prefer Average and explain that the script likely needs real inputs or a concrete runnable example; only use Poor if there is additional evidence the workflow is not realistically executable.
- If script execution fails due to missing Python dependencies (e.g., errors like "ModuleNotFoundError" or "ImportError" such as "No module named 'X'"), treat this primarily as an unmet prerequisite in the evaluator/runtime environment (which may not support auto-installing dependencies) rather than a flaw in the Skill package itself. Prefer executability = Average and explicitly cite the missing dependency and the need to pre-install it. Do NOT speculate that the Skill's installation instructions are "incorrect" or contain "typos" based on guesswork; only call them incorrect if the provided content explicitly demonstrates a contradiction. Otherwise describe the gap as an environment/setup prerequisite.
+ - If script execution fails because required input files or arguments are not available in the evaluation environment (e.g., errors like "FileNotFoundError", "No such file or directory", missing required positional arguments, or example commands that use placeholders like "<file>" without concrete sample inputs), treat this primarily as an unmet runtime prerequisite rather than a flaw in the Skill package itself. Prefer executability = Average and state that the workflow is runnable once real inputs are provided; only set executability to Poor when there is additional evidence of script defects (e.g., truncated/incomplete implementation, syntax errors, or contradictions between instructions and code).
- If you detect any CRITICAL CORRECTNESS ERROR in a core formula, algorithm, or code snippet (e.g., Python using "^" for exponentiation or other language-level mistakes that would produce wrong results or runtime failures), executability MUST be "Poor".
- If allowed_tools grants broader permissions than what the Skill clearly needs (e.g., allows "bash" or other powerful tools but the described workflow and examples do not require them), reduce executability by at least one level due to environment/permission ambiguity.
- When reading formulas and code snippets, audit them line-by-line in the context of their target language and typical runtime environment; if you find subtle traps or inconsistencies that would mislead an implementer or cause incorrect behavior, choose a lower (more conservative) executability rating.