Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

Merge remote-tracking branch 'origin/main' into fix/christopherk/issue-30-keyword-boundaries

Narendran Raghavanec41fbc3ac5a
diff --cc CHANGELOG.md
index 6e154d47,aa891065..232996b2
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -81,10 -81,35 +81,39 @@@ All notable changes to SkillEvaluator a
### Fixed
+- Quality scoring now uses boundary-aware and context-aware matching for XML tags,
+ reserved names, MCP guidance, README references, time references, exclusivity
+ language, instruction action verbs, and nested Markdown links, avoiding
+ incidental-word score changes.
+ - Tier 3 generated tasks now stage only an entry's declared `files`, preventing
+ undeclared fixtures from the shared `evals/files/` directory from appearing
+ in that task's `/workspace/input/`, while preserving copy-all behavior for
+ legacy entries that omit the field. Agent-visible target, reference, and
+ workspace skill projections now omit evaluator-owned `evals/` directories
+ from every staged skill package, including sanitized `--copy-repo` contexts,
+ while graders, native tasks, custom
+ environments, and declared inputs continue to load from the source dataset.
+ Authenticated historical result trees are also excluded after output rotation,
+ invalid markers fail closed, and late Codex, Cline, Goose, and Qwen
+ skill-discovery roots are reset before agent execution. Pre-upgrade custom
+ result roots outside `evals/` have no authenticity marker and cannot be
+ distinguished safely from authored runtime content. Move or delete that old
+ content before `--copy-repo` or other full-context evaluation, then rerun with
+ this version if replacement evidence is needed. Explicit task inputs cannot
+ select evaluator-owned datasets, configuration, graders, tests, native tasks,
+ environments, or results. Every agent and baseline arm now reads from one
+ private, selective evaluator snapshot containing the active control files,
+ task-source data, consumed fixtures and grader, and the complete authored
+ custom environment. Legacy omitted-file entries retain the full shared files
+ corpus. Unrelated evaluator subtrees and generated results stay outside the
+ snapshot. MCP configuration and completed-run artifacts are
+ read through bounded descriptor-anchored roots; on Windows, selected file
+ handles deny concurrent writes and deletes while live. Historical unmarked
+ runs created before canonical run-level `result.json` remain discoverable only
+ when their stable configuration and summaries satisfy the complete historical
+ schema. Pre-status scored summaries remain consumable, coherent status-era
+ failures remain visible without contributing scores, and marked current
+ partial runs continue to fail closed.
- Public benchmark cards now omit policy profiles, redact absolute host paths,
and normalize imported internal or retired metadata before publication.
- Previous-version validation now rejects catalog-wide scalar reuse and removal