Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): retain price provenance and charge coverage (#596)
Christopher Kevinf718dd39f929
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1a4f415c..1336b5e2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -447,6 +447,10 @@ All notable changes to SkillEvaluator are documented in this file.- Grader qualification reports null behavior-step results with a matching`not_checked` expectation as insufficient evidence instead of errors. Rawscores and unjudgeable rates are unchanged.+- Tier 3 cost budgets now retain user price-file provenance, actually applied+ rates, and attempt counts by cost derivation. Budget summaries distinguish+ agent inference accounting from billed spend and document excluded charges;+ cost limits remain admission boundaries, not hard currency ceilings.- Tier 3 token usage rows are now validated against the evaluation dataset's owncase IDs. Every case row uses one of those IDs at most once, and an attemptwhose only name is Harbor's 32-character cut of a case name is attributed todiff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex 62a2ca12..d45a90fd 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -341,8 +341,8 @@ The following flags are forwarded to the live-eval engine when Tier 3 is enabled| `--max-agent-tokens INTEGER` | unset | Maximum measured agent-model tokens. || `--max-judge-calls INTEGER` | unset | Maximum recorded non-salvage judge attempts; excludes transport retries and RAGAS-internal requests. || `--max-attempts-total INTEGER` | unset | Maximum logical Harbor attempts admitted across agents and arms, including private comparison arms. |-| `--max-cost-usd FLOAT` | unset | Maximum agent model spend; requires a price book and excludes judge spend, setup, and container time. |-| `--price-file FILE` | unset | JSON price book keyed by exact resolved model identifier; see [cost limits](/skills/skillevaluator/tier3-live-evaluation#cost-limits). |+| `--max-cost-usd FLOAT` | unset | Agent inference accounting admission limit, not billed spend or a hard currency ceiling; requires a price book. Judge, setup, container, backend, and infrastructure charges are excluded. |+| `--price-file FILE` | unset | User-supplied agent inference rates, not billed spend; keyed by exact resolved model identifier. File name/hash and applied rates are retained in `budget.pricing`; see [cost limits](/skills/skillevaluator/tier3-live-evaluation#cost-limits). || `--stop-on-pass / --no-stop-on-pass` | unset | Stop a case's remaining attempts once one passes. || `--model TEXT` | unset | Global agent model override. || `--agent-model TEXT` | unset | Per-agent model override, `AGENT=MODEL` (repeatable). |@@ -632,8 +632,8 @@ Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing| `--max-agent-tokens INTEGER` | unset | Maximum measured agent-model tokens. || `--max-judge-calls INTEGER` | unset | Maximum recorded non-salvage judge attempts; excludes transport retries and RAGAS-internal requests. || `--max-attempts-total INTEGER` | unset | Maximum logical Harbor attempts admitted across agents and arms, including private comparison arms. |-| `--max-cost-usd FLOAT` | unset | Maximum agent model spend; requires a price book and excludes judge spend, setup, and container time. |-| `--price-file FILE` | unset | JSON price book keyed by exact resolved model identifier; see [cost limits](/skills/skillevaluator/tier3-live-evaluation#cost-limits). |+| `--max-cost-usd FLOAT` | unset | Agent inference accounting admission limit, not billed spend or a hard currency ceiling; requires a price book. Judge, setup, container, backend, and infrastructure charges are excluded. |+| `--price-file FILE` | unset | User-supplied agent inference rates, not billed spend; keyed by exact resolved model identifier. File name/hash and applied rates are retained in `budget.pricing`; see [cost limits](/skills/skillevaluator/tier3-live-evaluation#cost-limits). || `--stop-on-pass / --no-stop-on-pass` | unset | Stop a case's remaining attempts once one passes. || `--n-concurrent INTEGER` | unset | Concurrent eval cases per agent. || `--max-agents INTEGER` | unset | Maximum agents to run in parallel. |diff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdxindex 99e857b2..adf28b81 100644--- a/docs/eval-datasets.mdx+++ b/docs/eval-datasets.mdx@@ -533,8 +533,8 @@ grading:| `harbor.max_agent_tokens` | integer ≥ 0 | Maximum observed agent-model tokens across the run. || `harbor.max_judge_calls` | integer ≥ 0 | Maximum recorded non-salvage judge attempts; excludes transport retries and RAGAS-internal requests. || `harbor.max_attempts_total` | integer ≥ 0 | Maximum logical Harbor attempts admitted across agents and arms, including private comparison arms. |-| `harbor.max_cost_usd` | finite number ≥ 0 | Agent model spend ceiling; requires `harbor.price_file` and excludes judge spend, setup, and container time. |-| `harbor.price_file` | path | JSON price book keyed by exact resolved model identifier. Relative paths resolve from the source skill directory. Each model requires `input_per_mtok` and `output_per_mtok`; optional `cached_input_per_mtok` defaults to the input rate. |+| `harbor.max_cost_usd` | finite number ≥ 0 | Agent inference accounting admission limit, not a hard currency ceiling or billed spend; requires `harbor.price_file`. Judge, setup, container, backend, and infrastructure charges are excluded. |+| `harbor.price_file` | path | User-supplied JSON rates per million tokens, keyed by exact resolved model identifier, not verified billing prices. Relative paths resolve from the source skill directory. Each model requires `input_per_mtok` and `output_per_mtok`; optional `cached_input_per_mtok` defaults to the input rate. File name/hash and actually applied rates are retained in `budget.pricing`; see [cost limits](/skills/skillevaluator/tier3-live-evaluation#cost-limits). || `harbor.timeout_multiplier` | finite number > 0 | Scales task timeouts. Values that would overflow Harbor's mandatory lifecycle timeouts are rejected before execution. || `harbor.agent_runtime_preflight` | boolean | Opt-in extra agent-only execution of the first staged task before the measured evaluation. Default: `false`; the CLI flag overrides this value. || `harbor.tolerate_baseline_timeouts` | boolean | Keep an agent scored on its with-skill arm when only baseline attempts hit the agent execution timeout; its baseline and Skill Lift are not reported (see [Baseline timeouts](/skills/skillevaluator/tier3-live-evaluation#baseline-timeouts)). Default: `false`; the CLI flag overrides this value. |diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 0c5d068c..2d08a7fa 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -1036,6 +1036,12 @@ least one bounded usage dimension still unmeasured). `estimate_denials` countslogical units refused after estimate-based exhaustion; `denied_unit_size`records the size of the unit that first caused that exhaustion.+Cost summaries cover agent inference accounting only, using user-supplied+rates with a provider-reported cost fallback, not billed spend. Judge inference,+setup, containers, backend fees, and infrastructure charges are not included.+Unknown agent cost is not treated as zero; measured subtotals remain visible,+and unknown cost stops further admission when a cost limit is configured.+`budget.status` and `exhausted_limit` describe admission decisions;`budget.partial` records attempts left not run. Separately,`budget.observed_overruns` maps each exceeded limit to its measured `used`,@@ -1774,8 +1780,33 @@ tokens use the input rate. A model key must exactly match the resolved agentmodel. Relative `harbor.price_file` paths resolve from the source skilldirectory.-Token and cost limits measure agent model spend only; they exclude judge-spend, environment setup, and container time. `--max-judge-calls` counts+The price file is user-supplied, not a verified billing rate card. Its currency+is as supplied by the file, labelled USD by `--max-cost-usd`; no currency+conversion or billing verification occurs. `budget.pricing` records+`source: user_price_file`, the file name without its absolute path, the SHA-256+of the exact loaded bytes, and `rates_per_mtok` for models whose rates actually+derived an attempt cost (empty before any such derivation). The effective cached+rate is recorded even when it defaults to the input rate.+`verified_against_provider_billing` is always false.++For each attempt, complete agent input/output token telemetry takes precedence+over provider-reported cost. Cost is uncached input times the input rate, plus+cached input times the cached rate, plus output times the output rate, divided+by one million. Cached tokens are clamped to input tokens; missing cached counts+are treated as uncached. If input/output telemetry is missing, reported agent+cost is used instead; otherwise cost is unmeasured, never invented or treated as+zero. An unknown attempt retains earlier measured subtotals and stops further+cost-bounded admission.++`budget.charge_coverage.attempts_by_derivation` counts completed accounting+records as `price_file`, `provider_reported`, or `unmeasured`. The coverage policy+excludes judge inference, environment setup, containers, backend fees, and+infrastructure. Judge evidence does not supply structured token counts for+pricing; prompt/reply character counts are not converted to tokens or money.+Neither the computed cost nor a provider-reported fallback verifies billed+spend, and the admission boundary is not a hard currency ceiling.++Token and cost limits cover agent inference only. `--max-judge-calls` countsrecorded judge attempts, excluding `salvage` entries, transport retries, andrequests made internally by RAGAS. Measured usage totals are retained evenwhen another attempt has no telemetry; that attempt is tracked as unmeasured.diff --git a/src/skillevaluator/tier3/harbor/run_budget.py b/src/skillevaluator/tier3/harbor/run_budget.pyindex 7c5fe1a5..f2b21896 100644--- a/src/skillevaluator/tier3/harbor/run_budget.py+++ b/src/skillevaluator/tier3/harbor/run_budget.py@@ -5,6 +5,7 @@from __future__ import annotations+import hashlibimport jsonimport mathimport threading@@ -31,12 +32,15 @@ class Price:@dataclass(frozen=True)class PriceBook:prices: dict[str, Price]+ file_name: str | None = None+ sha256: str | None = None@classmethoddef load(cls, path: Path) -> PriceBook:try:- raw = json.loads(path.read_text(encoding="utf-8"))- except (OSError, json.JSONDecodeError) as exc:+ contents = path.read_bytes()+ raw = json.loads(contents.decode("utf-8"))+ except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:raise BudgetConfigurationError(f"{path}: invalid price file: {exc}") from excif not isinstance(raw, dict) or not raw:raise BudgetConfigurationError(f"{path}: price file must be a non-empty object")@@ -62,7 +66,7 @@ class PriceBook:):raise BudgetConfigurationError(f"{path}: {model} prices must be finite and non-negative")prices[model] = Price(input_price, output_price, cached_input_price)- return cls(prices)+ return cls(prices, file_name=path.name, sha256=hashlib.sha256(contents).hexdigest())def price_for(self, model: str) -> Price:try:@@ -114,6 +118,10 @@ class BudgetController:unmeasured_agent_attempts: int = 0unmeasured_judge_attempts: int = 0unmeasured_cost_attempts: int = 0+ _cost_derivations: dict[str, int] = field(+ default_factory=lambda: {"price_file": 0, "provider_reported": 0, "unmeasured": 0}, repr=False+ )+ _applied_prices: dict[str, Price] = field(default_factory=dict, repr=False)_lock: threading.Lock = field(default_factory=threading.Lock, repr=False)_condition: threading.Condition = field(init=False, repr=False)_in_flight: dict[tuple[Any, ...], None] = field(default_factory=dict, repr=False)@@ -252,6 +260,7 @@ class BudgetController:tokens = input_tokens + output_tokensjudge_calls = _usage_number(usage, "judge_calls", "llm_call_count")cost = None+ derivation = "unmeasured"if self.price_book is not None and model and input_tokens is not None and output_tokens is not None:price = self.price_book.price_for(model)cached_tokens = _usage_number(usage, "cached_input_tokens", "total_cached_tokens", "cached_tokens") or 0@@ -265,8 +274,12 @@ class BudgetController:+ cached_tokens * cached_price+ output_tokens * price.output_per_mtok) / 1_000_000+ self._applied_prices[model] = Price(price.input_per_mtok, price.output_per_mtok, cached_price)+ derivation = "price_file"if cost is None:cost = _usage_number(usage, "cost_usd", "total_cost_usd")+ if cost is not None:+ derivation = "provider_reported"if tokens is None:self.unmeasured_agent_attempts += 1else:@@ -282,6 +295,7 @@ class BudgetController:else:self.used_cost_usd = (self.used_cost_usd or 0.0) + float(cost)self._cost_usage_seen = True+ self._cost_derivations[derivation] += 1for dimension, value in (("agent_tokens", tokens), ("judge_calls", judge_calls), ("cost_usd", cost)):if value is not None:self._estimates[dimension] = max(self._estimates.get(dimension, 0), value)@@ -324,6 +338,16 @@ class BudgetController:"exhausted_basis": "estimate" if self._denied_unit_size else None,"attempts_not_run": len(self.not_run),"not_run": list(self.not_run),+ "charge_coverage": {+ "agent_inference": "user-price-file token pricing; otherwise provider-reported cost; otherwise not measured",+ "attempts_by_derivation": dict(self._cost_derivations),+ "unmeasured_agent_cost": "not estimated or treated as zero; measured subtotal retained; stops further cost-bounded admission",+ "judge_inference": "not priced; bounded only by --max-judge-calls when configured",+ "environment_setup": "not included",+ "container": "not included",+ "backend": "not included",+ "infrastructure": "not included",+ },"reservation": {"policy": "largest_measured_attempt","estimate_per_attempt": {@@ -338,6 +362,22 @@ class BudgetController:"denied_unit_size": self._denied_unit_size,},}+ if self.price_book is not None and self.price_book.file_name is not None:+ metadata["pricing"] = {+ "source": "user_price_file",+ "file_name": self.price_book.file_name,+ "sha256": self.price_book.sha256,+ "rates_per_mtok": {+ model: {+ "input_per_mtok": price.input_per_mtok,+ "output_per_mtok": price.output_per_mtok,+ "cached_input_per_mtok": price.cached_input_per_mtok,+ }+ for model, price in self._applied_prices.items()+ },+ "currency": "as supplied by the price file (labelled USD by the --max-cost-usd flag)",+ "verified_against_provider_billing": False,+ }metadata["observed_overruns"] = _budget_observations(metadata)[0]return metadata@@ -477,4 +517,7 @@ def budget_line(metadata: dict[str, Any]) -> str:if observations or metadata.get("status") == "exhausted":line += "; configured limits not fully verified"line += f" ({'; '.join(incomplete)})"+ limits = metadata.get("limits")+ if isinstance(limits, dict) and limits.get("max_cost_usd") is not None:+ line += "; agent inference only (user-supplied rates; provider-cost fallback), not billed spend"return linediff --git a/tests/tier3/test_budget_reservation.py b/tests/tier3/test_budget_reservation.pyindex 21dafd53..41fd9ea3 100644--- a/tests/tier3/test_budget_reservation.py+++ b/tests/tier3/test_budget_reservation.py@@ -104,9 +104,14 @@ def test_idle_projection_denial_has_estimate_basis_and_honest_budget_line(dimensassert metadata["observed_overruns"] == {}label = {"agent_tokens": "agent tokens", "judge_calls": "judge calls", "cost_usd": "cost"}[dimension]used, cap, estimate = ("$10.00", "$25.00", "$20.00") if dimension == "cost_usd" else ("10", "25", "20")+ qualifier = (+ "; agent inference only (user-supplied rates; provider-cost fallback), not billed spend"+ if dimension == "cost_usd"+ else ""+ )assert budget_line(metadata) == (f"Budget exhausted ({label} {used} of {cap}; next unit estimated {estimate} for 2 attempts)"- ": 2 of 6 attempts not run; results are partial"+ f": 2 of 6 attempts not run; results are partial{qualifier}")assert not budget.admit_many(unit("later", 1))assert budget.metadata()["reservation"]["estimate_denials"] == 2diff --git a/tests/tier3/test_price_provenance.py b/tests/tier3/test_price_provenance.pynew file mode 100644index 00000000..203463f4--- /dev/null+++ b/tests/tier3/test_price_provenance.py@@ -0,0 +1,130 @@+import hashlib+import json++import pytest+from rich.console import Console++from skillevaluator.tier3.harbor import runner+from skillevaluator.tier3.harbor.run_budget import BudgetConfigurationError, BudgetController, PriceBook, budget_line+from skillevaluator.tier3.result_display import render_evaluation_result+++def load_prices(tmp_path):+ path = tmp_path / "prices.json"+ path.write_text(+ json.dumps(+ {+ "priced": {"input_per_mtok": 2, "output_per_mtok": 4, "cached_input_per_mtok": 1},+ "fallback": {"input_per_mtok": 1, "output_per_mtok": 2},+ "unused": {"input_per_mtok": 9, "output_per_mtok": 9},+ }+ )+ )+ return path, PriceBook.load(path)+++def test_file_provenance_hashes_loaded_bytes_and_reports_only_applied_rates(tmp_path):+ path, prices = load_prices(tmp_path)+ original = path.read_bytes()+ budget = BudgetController(price_book=prices)+ assert budget.metadata()["pricing"]["rates_per_mtok"] == {}+ budget.record_usage({"input_tokens": 1000, "output_tokens": 100, "cached_input_tokens": 400}, model="priced")+ budget.record_usage({"input_tokens": 1000, "output_tokens": 0, "cached_input_tokens": 500}, model="fallback")+ path.write_text("{}")+ assert budget.metadata()["pricing"] == {+ "source": "user_price_file",+ "file_name": "prices.json",+ "sha256": hashlib.sha256(original).hexdigest(),+ "rates_per_mtok": {+ "priced": {"input_per_mtok": 2, "output_per_mtok": 4, "cached_input_per_mtok": 1},+ "fallback": {"input_per_mtok": 1, "output_per_mtok": 2, "cached_input_per_mtok": 1},+ },+ "currency": "as supplied by the price file (labelled USD by the --max-cost-usd flag)",+ "verified_against_provider_billing": False,+ }+ assert budget.metadata()["used"]["cost_usd"] == pytest.approx(0.003)+ assert str(tmp_path) not in json.dumps(budget.metadata())+++def test_derivation_counts_follow_pricing_precedence_fallback_and_unknown_usage(tmp_path):+ _, prices = load_prices(tmp_path)+ budget = BudgetController(price_book=prices)+ assert budget.metadata()["charge_coverage"]["attempts_by_derivation"] == {+ "price_file": 0,+ "provider_reported": 0,+ "unmeasured": 0,+ }+ budget.record_usage(+ {"input_tokens": 1000, "output_tokens": 100, "cached_input_tokens": 400, "cost_usd": 99}, model="priced"+ )+ budget.record_usage({"total_cost_usd": 0.5}, model="fallback")+ budget.record_usage(None, model="unused")+ metadata = budget.metadata()+ assert metadata["charge_coverage"]["attempts_by_derivation"] == {+ "price_file": 1,+ "provider_reported": 1,+ "unmeasured": 1,+ }+ assert sum(metadata["charge_coverage"]["attempts_by_derivation"].values()) == metadata["finished_attempts"] == 3+ assert metadata["used"]["cost_usd"] == pytest.approx(0.502)+ assert metadata["unmeasured_attempts"]["cost_usd"] == 1+ assert set(metadata["pricing"]["rates_per_mtok"]) == {"priced"}+ metadata["charge_coverage"]["attempts_by_derivation"]["price_file"] = 99+ assert budget.metadata()["charge_coverage"]["attempts_by_derivation"]["price_file"] == 1++[email protected]("usage,derivation", [({"cost_usd": 0}, "provider_reported"), ({}, "unmeasured")])+def test_no_price_file_has_no_pricing_object(usage, derivation):+ budget = BudgetController()+ budget.record_usage(usage)+ metadata = budget.metadata()+ assert "pricing" not in metadata+ assert metadata["charge_coverage"]["attempts_by_derivation"][derivation] == 1+ assert metadata["charge_coverage"]["judge_inference"].startswith("not priced")+ for charge in ("environment_setup", "container", "backend", "infrastructure"):+ assert metadata["charge_coverage"][charge] == "not included"++[email protected]("contents", [b"\xff", b"{", b"{}"])+def test_invalid_price_file_fails_without_provenance(tmp_path, contents):+ path = tmp_path / "invalid.json"+ path.write_bytes(contents)+ with pytest.raises(BudgetConfigurationError):+ PriceBook.load(path)+++def test_judge_token_or_cost_fields_do_not_feed_agent_pricing(tmp_path):+ _, prices = load_prices(tmp_path)+ job = tmp_path / "job"+ verifier = job / "trial" / "verifier"+ verifier.mkdir(parents=True)+ (verifier / "judge_attempts.json").write_text(+ json.dumps(+ {+ "attempts": [+ {"method": "llm", "usage": {"input_tokens": 1_000_000, "output_tokens": 1_000_000}, "cost_usd": 99}+ ]+ }+ )+ )+ usage = runner._budget_usage(job)+ assert usage == {"judge_calls": 1}+ budget = BudgetController(price_book=prices)+ budget.record_usage(usage, model="priced")+ assert budget.metadata()["used"]["cost_usd"] == "not measured"+ assert budget.metadata()["charge_coverage"]["attempts_by_derivation"]["unmeasured"] == 1++[email protected]("cap", [0, 1])+def test_cost_qualifier_reaches_terminal_and_unknown_usage_stops_admission(tmp_path, cap):+ _, prices = load_prices(tmp_path)+ budget = BudgetController(max_cost_usd=cap, price_book=prices)+ budget.record_usage(None)+ assert not budget.admit(agent="a", arm="with", case_id="c", attempt=1)+ assert budget.metadata()["charge_coverage"]["attempts_by_derivation"]["unmeasured"] == 1+ qualifier = "agent inference only (user-supplied rates; provider-cost fallback), not billed spend"+ assert budget_line(budget.metadata()).endswith(qualifier)+ output = Console(record=True, force_terminal=False, width=500)+ render_evaluation_result({"execution_status": "failed", "budget": budget.metadata()}, console=output)+ assert qualifier in output.export_text()+ assert qualifier not in budget_line(BudgetController(price_book=prices).metadata())diff --git a/tests/tier3/test_run_budget.py b/tests/tier3/test_run_budget.pyindex cdf0c794..4087484d 100644--- a/tests/tier3/test_run_budget.py+++ b/tests/tier3/test_run_budget.py@@ -35,6 +35,7 @@ def test_token_and_cost_admission():assert (budget_line(budget.metadata())== "Budget exhausted (agent tokens 10 of 10): 1 of 2 attempts not run; results are partial"+ "; agent inference only (user-supplied rates; provider-cost fallback), not billed spend")@@ -634,7 +635,12 @@ def test_legacy_overrun_boundaries_include_zero_and_subcent_costs(key, cap, valu"attempts": "max_attempts_total",}[key]metadata = {"limits": {limit: cap}, "used": {key: value}, "started_attempts": value}- assert budget_line(metadata) == f"Budget overrun observed ({expected})"+ qualifier = (+ "; agent inference only (user-supplied rates; provider-cost fallback), not billed spend"+ if key == "cost_usd"+ else ""+ )+ assert budget_line(metadata) == f"Budget overrun observed ({expected}){qualifier}"@pytest.mark.parametrize("invalid", [True, False, -1, "[bold]bad[/bold]", float("nan"), float("inf"), [], {}])