Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

test(tier3): accept either cooperative cancellation result (#707)

Christopher Kevin7c4eabd75bfe
diff --git a/tests/tier3/test_budget_reservation.py b/tests/tier3/test_budget_reservation.py
index 41fd9ea3..a839e6d4 100644
--- a/tests/tier3/test_budget_reservation.py
+++ b/tests/tier3/test_budget_reservation.py
@@ -373,7 +373,9 @@ def test_cancellation_wakes_waiter_before_in_flight_job_finishes(monkeypatch, tm
finally:
finish_job.set()
budget.cancel()
- with pytest.raises(BudgetCancelledError):
+ # Both workers observe cancellation. Either the waiting budget admission or
+ # the in-flight Harbor job can surface first through as_completed().
+ with pytest.raises((BudgetCancelledError, runner._HarborRunCancelledError)):
future.result(timeout=5)
assert budget.metadata()["reservation"]["in_flight_attempts"] == 0
assert budget.metadata()["not_run"] == []