Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
Add opt-in Unicode character facts for Tier 3 judges (#242)
Narendran Raghavan58953e99298d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b38fad5e..e798397e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,6 +14,50 @@ All notable changes to SkillEvaluator are documented in this file.numeric and nullable fields. Existing prompts remain unchanged when thesetting is unset.+- `tier3 evaluate` now leads with a case-first summary: a verdict line and one+ row per `evals.json` case. With one agent, each row shows, for with and+ without the skill, the pass count, per-attempt marks (`✓` pass, `✗` fail,+ `⊘` did not run, `·` skipped after a pass) and mean score, plus an effect+ column (`helps`, `hurts`, `both fail`, `better score`, `worse score`,+ `no change`, `inconclusive`). With several agents, each agent gets a column+ showing without-skill → with-skill marks and the pass-rate change. Success+ rate, quality score, and dimensions follow. A "How to improve" list follows,+ ranked by attempts lost and capped at three; each item names the case,+ agent, issue type (`skill`, `skill not used`, `skill hurts`, `environment`,+ `run error`, `eval case`, `missing input`, `goal not met`), one action, and an evidence+ path, and is also written to `what_to_fix.json` in the run directory. When+ nothing needs fixing it prints a one-line "nothing to fix" instead. The+ verdict is `inconclusive` when attempts did not run. The per-evaluator and+ per-dimension tables and metric findings moved behind the new `--verbose`+ flag and remain in `report.html`.+- `goal not met` flags with-skill attempts that pass on the aggregate score+ while the judge scored goal accuracy 0.00 (for example, the agent only asked+ for details).+- `report.html` leads with the same case-based findings: "Verdict at a glance"+ adds the per-case result, the highest-scoring agent versus the biggest skill+ lift, and the goal-not-met count; "Conclusion & Next Steps" and the Insights+ recommendations list the `what_to_fix.json` fixes (or "Nothing to fix").+ Data fixes elsewhere: Pass@K shows attempts passed, a regressed passing+ dimension reads "PASS · regressed" with its weakest signal, resource totals+ are formatted with units and percent change, Attempt Details groups each+ case's with/without rows and marks passes that missed the goal, and Error+ Recovery lists only trials with faults, retries or warnings.+- Reusable GitHub Actions onboarding for complete Tier 1 checks, with+ pinned runtime/workflow consumer templates, affected-skill PR selection,+ optional hosted reports and updated PR feedback. The new guide separates+ basic scanning from publisher setup; public image publication and generated+ release files require a reviewed manual release before consumer use. The+ Tier 1 image is a multi-stage build with precompiled bytecode that ships only+ the locked Semgrep, SkillSpector and SkillEvaluator environments, Gitleaks,+ `ci/` and the license files. A release builds it without cached layers and+ pushes the image it smoke-tested.+- The reusable Tier 1 workflow scans up to four skills at once. The+ `scan-workers` input (1 to 16) changes this, and `1` scans one skill at a+ time. Each worker gets its own private home and temporary directory. Logs,+ annotations and report rows keep skill order, and skills not started when the+ scan time budget runs out are still reported as not evaluated. A timeout,+ Ctrl-C or an error while saving results ends running scans together with+ the scanner tools they started.- `--quality-config PATH` on `validate`, `quality-check`, and `tier1` loads thedeterministic quality check's expected Markdown headings from a YAML fileinstead of the built-in list. Each `expected_headers` entry sets a `heading`diff --git a/src/skillevaluator/tier3/eval_core/character_facts.py b/src/skillevaluator/tier3/eval_core/character_facts.pyindex dc5f1a4a..18cde6e3 100644--- a/src/skillevaluator/tier3/eval_core/character_facts.py+++ b/src/skillevaluator/tier3/eval_core/character_facts.py@@ -44,7 +44,7 @@ def format_character_facts(context = text[max(0, index - context_radius) : index + context_radius + 1]entries.append(f"- line {line}, column {column}: U+{ord(character):04X} "- f"{unicodedata.name(character, 'UNNAMED')}; "+ f"{unicodedata.name(character, '<control>')}; "f"escaped context: {json.dumps(context, ensure_ascii=True)}")if character == "\n":