Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
test(tier3): accept either cooperative cancellation result (#707)
Christopher Kevin7c4eabd75bfe
diff --git a/tests/tier3/test_budget_reservation.py b/tests/tier3/test_budget_reservation.pyindex 41fd9ea3..a839e6d4 100644--- a/tests/tier3/test_budget_reservation.py+++ b/tests/tier3/test_budget_reservation.py@@ -373,7 +373,9 @@ def test_cancellation_wakes_waiter_before_in_flight_job_finishes(monkeypatch, tmfinally:finish_job.set()budget.cancel()- with pytest.raises(BudgetCancelledError):+ # Both workers observe cancellation. Either the waiting budget admission or+ # the in-flight Harbor job can surface first through as_completed().+ with pytest.raises((BudgetCancelledError, runner._HarborRunCancelledError)):future.result(timeout=5)assert budget.metadata()["reservation"]["in_flight_attempts"] == 0assert budget.metadata()["not_run"] == []