Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
feat(evals): repeated sampling for nondeterministic targets (#1550)
Kevinf6657709e9bf
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 52aaee9a..9e3652da 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -18,6 +18,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Persistent-cache keys are process-stable: relation/triplet extraction now folds entity/relation inputs into the key via a deterministic SHA-256 fingerprint instead of the process-randomized built-in `hash()`, so entries reliably rehit after a restart. `ttl=0` expires immediately on both backends, and corrupt/undeserializable rows are dropped rather than retained- **Security / trust model:** the sqlite file is deserialized back into the process (default serializer `pickle`), so it must point at a trusted, user-private location and may hold sensitive extraction results in the clear (documented in the module usage guide). The default database lives under a per-user private directory (`$XDG_CACHE_HOME`/`~/.cache`, `0o700`); the db file is created **atomically** with `0o600` (`O_CREAT|O_EXCL`, no validate-then-open window) and an existing symlink, non-regular file, or file owned by another user is refused (falling back to in-memory). sqlite access uses a busy-timeout with bounded retry, rolls back within the same lock scope on failure, and contains all errors (reads degrade to a miss, writes to a skipped update)- No new runtime dependencies (stdlib `sqlite3`). New public exports: `CacheBackend`, `InMemoryBackend`, `SqliteCacheBackend`, `configure_cache`. New `tests/semantic_extract/test_cache_backends.py`++- **Repeated sampling for nondeterministic evals (#1545)**+ - New `evaluate_repeated(cases, evaluators, config=None, target_fn=None, runs=10)` in `semantica.evals`: reruns `target_fn` per case `n` times and aggregates per-evaluator `SampleStats` — `n`, `passes`, `errors`, `pass_rate`, `mean_score`, `stddev`, and the observed `any_passed` (pass@n) / `all_passed` (pass^n). Verdicts classify each case as `stable_pass` / `flaky` / `stable_fail` / `error`, with a `RepeatedSummary` over the suite+ - Nondeterminism stays in `target_fn`; evaluators remain pure. A case with a non-null static `actual` and `runs > 1` raises `ValueError` up front; an `actual` of `None` is treated as absent, matching `evaluate()`. Objective config applies per run and the same objective gates the aggregate `pass_rate` via `SampleStats.objective_passed`. `evaluate()` is untouched; new result types (`SampleStats`, `RepeatedCaseResult`, `RepeatedSummary`) are exported additively+ - New `tests/evals/test_repeater.py`+- **Schema-guided extraction validation** (#1510) by @Besokus- New `SchemaValidator` (`semantica.semantic_extract`, lazy export): a deterministic sibling of `ExtractionValidator` that checks extraction output for *conformance to a domain ontology* — an axis orthogonal to `ExtractionValidator`'s confidence checks. It mirrors the same interface (`validate_entities()` / `validate_relations()` returning `ValidationResult`, batch-aware), so the two compose back-to-back- Entity labels must be concepts in the schema; relation predicates must be in the schema and satisfy their `domain` / `range`. Violations are reported in `ValidationResult.errors` with counts in `metrics` and `score` = conformance ratio; `filter_by_schema()` / `filter_relations_by_schema()` return the conforming subset (mirroring `filter_by_confidence`). No LLM requireddiff --git a/docs/modules.md b/docs/modules.mdindex 9c47f1cc..7ef5a6ad 100644--- a/docs/modules.md+++ b/docs/modules.md@@ -506,7 +506,7 @@ summary = evaluate(cases, evaluators=["levenshtein"])print(summary.total, summary.passed, summary.pass_rate)```-**Public API:** `evaluate(cases, evaluators, config=None)`, `list_evaluators()`, `get_evaluator(name)`, and the `EvalMetric` / `CaseResult` / `EvalSummary` result types. See the [Evals reference](/reference/evals).+**Public API:** `evaluate(cases, evaluators, config=None)`, `evaluate_repeated(cases, evaluators, config=None, target_fn=None, runs=10)` for repeated sampling over nondeterministic targets, `list_evaluators()`, `get_evaluator(name)`, and the `EvalMetric` / `CaseResult` / `EvalSummary` / `SampleStats` / `RepeatedCaseResult` / `RepeatedSummary` result types. See the [Evals reference](/reference/evals).### Corediff --git a/docs/reference/evals.md b/docs/reference/evals.mdindex a465bb68..5c59a346 100644--- a/docs/reference/evals.md+++ b/docs/reference/evals.md@@ -30,15 +30,19 @@ assert on in tests, or track across runs.| Name | Kind | Role || :--- | :--- | :--- || `evaluate(cases, evaluators, config=None, target_fn=None)` | function | Run named evaluators over each case, return an `EvalSummary` |+| `evaluate_repeated(cases, evaluators, config=None, target_fn=None, runs=10)` | function | Rerun `target_fn` per case and aggregate stats, return a `RepeatedSummary` || `list_evaluators()` | function | Sorted names of every registered evaluator || `get_evaluator(name)` | function | Look up a single evaluator function by name || `EvalMetric` | dataclass (frozen) | One evaluator's result: `score`, `passed`, `meta` || `CaseResult` | namedtuple | One case's result: `case_id`, `status`, `metrics`, `details` || `EvalSummary` | dataclass | Aggregate across cases: `total`, `passed`, `failed`, `errors`, `pass_rate`, `cases` |+| `SampleStats` | dataclass (frozen) | Per-evaluator statistics over repeated runs: `n`, `passes`, `errors`, `pass_rate`, `mean_score`, `stddev`, `any_passed`, `all_passed`, `objective_passed`, `samples` |+| `RepeatedCaseResult` | dataclass (frozen) | Repeated outcome for one case: `case_id`, `verdict`, `stats` |+| `RepeatedSummary` | dataclass | Aggregate across repeated-sampled cases: `runs`, `stable_pass`, `flaky`, `stable_fail`, `errors`, `cases` |```pythonimport semantica.evals as evals-from semantica.evals import evaluate, list_evaluators, get_evaluator+from semantica.evals import evaluate, evaluate_repeated, list_evaluators, get_evaluator```## Built-in evaluators@@ -204,10 +208,54 @@ for case in summary.cases:print(metric.meta.get("reasons", {})) # per-sub-check failure reasons```-`EvalMetric` is frozen (`score: float`, `passed: bool`, `meta: dict`). `CaseResult`-is a namedtuple, and `EvalSummary` is a plain dataclass, so all three are+`EvalMetric`, `SampleStats`, and `RepeatedCaseResult` are frozen dataclasses+(`score: float`, `passed: bool`, `meta: dict`, ...). `CaseResult` is a+namedtuple and the `*Summary` classes are plain dataclasses, so all arestraightforward to serialize for logging or regression tracking.+## Repeated sampling++For nondeterministic targets (LLM-backed extraction, agent pipelines,+sampling-based judges) a single verdict says little. `evaluate_repeated` reruns+`target_fn` per case `n` times and aggregates per-evaluator statistics:++```python+from semantica.evals import evaluate_repeated++summary = evaluate_repeated(+ cases, # must be dict cases; see below+ evaluators=["exact_match"],+ target_fn=pipeline_run, # called once per run to produce a fresh `actual`+ runs=10,+)++summary.runs # 10+summary.cases[0].verdict # "stable_pass" | "flaky" | "stable_fail" | "error"+summary.cases[0].stats["exact_match"].pass_rate # e.g. 0.8+summary.cases[0].stats["exact_match"].stddev # e.g. 0.13+```++Per-evaluator `SampleStats` carries `n`, `passes`, `errors`, `pass_rate`+(`passes / n`), `mean_score`, `stddev`, and the derived booleans `any_passed`+(observed pass@n) and `all_passed` (observed pass^n). Verdicts pool all samples:+`stable_pass` when every run passed every evaluator, `stable_fail` when none+passed, `flaky` when the pool mixes passes and fails, and `error` when a run+errored.++Requirements and edge cases:++- Each case can be a dict or a `(expected, actual)` tuple, matching `evaluate()`.+ A case that carries a **non-null** static `actual` has nothing to sample; using+ it with `runs > 1` raises `ValueError`. An `actual` of `None` is treated as+ absent, so it falls back to the resolver exactly as in `evaluate()`.+- `target_fn` exceptions and evaluator failures become per-run error samples and+ mark the case `error`; they never crash the run.+- The objective layer applies per run as in `evaluate()`, and the same objective+ gates the aggregate `pass_rate` through `SampleStats.objective_passed` (e.g.+ `{direction: maximize, threshold: 0.8}` requires an 80% pass rate across runs).+- At least one evaluator is required, and evaluator names must be unique.+- `runs` must be `>= 1`.+## Notes- `llm_as_judge` needs `config["judge_fn"]`, a callablediff --git a/semantica/evals/__init__.py b/semantica/evals/__init__.pyindex af1dd7af..352422c0 100644--- a/semantica/evals/__init__.py+++ b/semantica/evals/__init__.py@@ -7,15 +7,26 @@ runner for measuring decision records, audit trails, and reasoning output.from . import decision_evaluators # noqa: F401 (registers decision_scores)from . import evaluators # noqa: F401 (registers the generic evaluators)from .registry import get_evaluator, list_evaluators-from .runner import evaluate-from .types import CaseResult, EvalMetric, EvalSummary+from .runner import evaluate, evaluate_repeated+from .types import (+ CaseResult,+ EvalMetric,+ EvalSummary,+ RepeatedCaseResult,+ RepeatedSummary,+ SampleStats,+)__version__ = "0.1.0"__all__ = ["evaluate",+ "evaluate_repeated","get_evaluator","list_evaluators","CaseResult","EvalMetric","EvalSummary",+ "RepeatedCaseResult",+ "RepeatedSummary",+ "SampleStats",]diff --git a/semantica/evals/runner.py b/semantica/evals/runner.pyindex 815c4762..a77ac949 100644--- a/semantica/evals/runner.py+++ b/semantica/evals/runner.py@@ -4,7 +4,14 @@ import mathfrom typing import Any, Callable, Dict, List, Optional, Tuple, Unionfrom .registry import get_evaluator-from .types import CaseResult, EvalMetric, EvalSummary+from .types import (+ CaseResult,+ EvalMetric,+ EvalSummary,+ RepeatedCaseResult,+ RepeatedSummary,+ SampleStats,+)Case = Union[Dict[str, Any], Tuple[Any, Any]]@@ -64,12 +71,18 @@ def _parse_objective(name, eval_config):raise ValueError(f"objective for '{name}': 'minimize' requires a 'threshold'")- return {"direction": "minimize", "threshold": _coerce_threshold(name, threshold)}+ return {+ "direction": "minimize",+ "threshold": _coerce_threshold(name, threshold),+ }if direction == "maximize":if threshold is None:# no bar to re-decide against; treat as absent (evaluator default stands)return None- return {"direction": "maximize", "threshold": _coerce_threshold(name, threshold)}+ return {+ "direction": "maximize",+ "threshold": _coerce_threshold(name, threshold),+ }raise ValueError(f"objective for '{name}': 'direction' must be 'maximize' or 'minimize' "f"(got {direction!r})"@@ -98,7 +111,9 @@ def _extract(case: Case, target_fn: Optional[Callable]):return case_id, expected, actual, config, per_fn-def _merge_config(default_config: Dict[str, Any], case_config: Dict[str, Any]) -> Dict[str, Any]:+def _merge_config(+ default_config: Dict[str, Any], case_config: Dict[str, Any]+) -> Dict[str, Any]:"""Deep-merge per-case config over the global config (two levels deep).Level 1 (top-level keys, e.g. evaluator names): merged key-by-key so a@@ -185,7 +200,9 @@ def evaluate(metric = get_evaluator(name)(actual, expected, config=eval_config)objective = objective_by_name.get(name)if objective is not None and "error" not in metric.meta:- metric = EvalMetric(metric.score, _apply_objective(metric, objective), metric.meta)+ metric = EvalMetric(+ metric.score, _apply_objective(metric, objective), metric.meta+ )metrics[name] = metricif "error" in metric.meta:errored = True@@ -206,6 +223,189 @@ def evaluate(errors = sum(1 for c in case_results if c.status == "error")pass_rate = (passed / total) if total else 1.0return EvalSummary(- total, passed, failed, errors, pass_rate,+ total,+ passed,+ failed,+ errors,+ pass_rate,cases=case_results,)+++def _stddev(values: List[float]) -> float:+ """Population standard deviation over ``values`` (0.0 for <2 samples)."""+ if len(values) < 2:+ return 0.0+ mean = sum(values) / len(values)+ variance = sum((v - mean) ** 2 for v in values) / len(values)+ return math.sqrt(variance)+++def evaluate_repeated(+ cases: List[Case],+ evaluators: List[str],+ config: Optional[Dict[str, Any]] = None,+ target_fn: Optional[Callable] = None,+ runs: int = 10,+) -> RepeatedSummary:+ """Rerun ``target_fn`` per case ``runs`` times and aggregate statistics.++ Non-determinism in an evaluation lives in ``target_fn`` (an LLM call, a+ sampling-based pipeline), not in the evaluators, which are pure functions+ over a given ``actual``. So ``evaluate_repeated`` re-invokes ``target_fn``+ on each run to draw a fresh ``actual``, then scores it with the named+ evaluators. Aggregating those scores across runs surfaces distribution-level+ evidence (pass rate, mean/stddev) and a per-case verdict:++ - ``stable_pass``: every run passed every evaluator+ - ``flaky``: at least one pass and one fail across runs/evaluators+ - ``stable_fail``: no run passed any evaluator+ - ``error``: a run errored++ Verdict for a multi-evaluator case pools samples across all evaluators: it+ is ``stable_pass`` only when every sample of every evaluator passed, and+ ``flaky`` when the pool mixes passes and fails (e.g. one evaluator passing+ every run while another passes none).++ A case that carries a non-null static ``actual`` (rather than a ``target_fn``)+ has nothing to sample, so with ``runs > 1`` it raises ``ValueError``, matching+ the fail-fast style of invalid objective config. An ``actual`` of ``None`` is+ treated as absent, exactly as in ``evaluate()``: it falls back to the+ resolver. ``evaluate()`` is untouched; this is an additive entry point over+ the same helpers.+ """+ if runs < 1:+ raise ValueError(f"runs must be >= 1 (got {runs!r})")+ if not evaluators:+ raise ValueError("evaluate_repeated requires at least one evaluator")+ if len(evaluators) != len(set(evaluators)):+ raise ValueError(+ f"evaluators must be unique (got duplicates: "+ f"{sorted({n for n in evaluators if evaluators.count(n) > 1})})"+ )+ default_config = config or {}+ repeated_results: List[RepeatedCaseResult] = []++ # Validate object config and sampling prerequisites for every case up front+ # so an invalid objective or an unsamplable static ``actual`` rejects the+ # run before any target_fn or evaluator executes (fail-fast), regardless of+ # which case carries it.+ pre_resolved = []+ for case in cases:+ case_id, _, static_actual, case_config, _ = _extract(case, target_fn)+ if static_actual is not None and runs > 1:+ raise ValueError(+ f"case '{case_id}': repeated sampling requires a streaming "+ f"``target_fn``; a static ``actual`` can only be run once "+ f"(runs={runs!r})"+ )+ merged = _merge_config(default_config, case_config)+ pre_resolved.append(+ {+ name: _parse_objective(name, merged.get(name) or {})+ for name in evaluators+ }+ )++ for case, objective_by_name in zip(cases, pre_resolved):+ case_id, expected, static_actual, case_config, per_fn = _extract(+ case, target_fn+ )+ merged = _merge_config(default_config, case_config)+ if expected is None:+ expected = merged.get("expected")+ resolver = per_fn or target_fn+ has_actual = static_actual is not None++ # All samples for this case, per evaluator.+ per_eval: Dict[str, List[EvalMetric]] = {name: [] for name in evaluators}+ errored_any = False+ for _ in range(runs):+ if not has_actual and resolver is None:+ for name in evaluators:+ per_eval[name].append(+ EvalMetric(0.0, False, {"error": "no target_fn to sample"})+ )+ errored_any = True+ continue+ try:+ actual = static_actual if has_actual else resolver(case)+ except Exception as exc: # noqa: BLE001+ errored_any = True+ for name in evaluators:+ per_eval[name].append(+ EvalMetric(0.0, False, {"error": f"target_fn: {exc}"})+ )+ continue+ for name in evaluators:+ eval_config = merged.get(name) or {}+ objective = objective_by_name.get(name)+ try:+ metric = get_evaluator(name)(actual, expected, config=eval_config)+ if objective is not None and "error" not in metric.meta:+ metric = EvalMetric(+ metric.score,+ _apply_objective(metric, objective),+ metric.meta,+ )+ except Exception as exc: # noqa: BLE001+ metric = EvalMetric(0.0, False, {"error": str(exc)})+ if "error" in metric.meta:+ errored_any = True+ per_eval[name].append(metric)++ # Aggregate per-evaluator stats.+ stats: Dict[str, SampleStats] = {}+ verdict = "error" if errored_any else None+ for name in evaluators:+ samples = per_eval[name]+ n = len(samples)+ passes = sum(1 for m in samples if m.passed and "error" not in m.meta)+ errors = sum(1 for m in samples if "error" in m.meta)+ scores = [m.score for m in samples if "error" not in m.meta]+ pass_rate = passes / n if n else 0.0+ mean_score = sum(scores) / len(scores) if scores else 0.0+ objective = objective_by_name.get(name)+ objective_passed = None+ if objective is not None and "direction" in objective:+ objective_passed = _apply_objective(+ EvalMetric(pass_rate, True), objective+ )+ stats[name] = SampleStats(+ n=n,+ passes=passes,+ errors=errors,+ pass_rate=pass_rate,+ mean_score=mean_score,+ stddev=_stddev(scores),+ any_passed=passes > 0,+ all_passed=passes == n and n > 0 and errors == 0,+ objective_passed=objective_passed,+ samples=samples,+ )+ if verdict is None:+ # Verdict across all evaluators: flaky if any mixed, else uniform.+ passes_total = sum(stats[name].passes for name in evaluators)+ n_total = sum(stats[name].n for name in evaluators)+ if passes_total == 0:+ verdict = "stable_fail"+ elif passes_total == n_total:+ verdict = "stable_pass"+ else:+ verdict = "flaky"+ repeated_results.append(+ RepeatedCaseResult(case_id=case_id, verdict=verdict, stats=stats)+ )++ stable_pass = sum(1 for r in repeated_results if r.verdict == "stable_pass")+ flaky = sum(1 for r in repeated_results if r.verdict == "flaky")+ stable_fail = sum(1 for r in repeated_results if r.verdict == "stable_fail")+ errors = sum(1 for r in repeated_results if r.verdict == "error")+ return RepeatedSummary(+ runs=runs,+ stable_pass=stable_pass,+ flaky=flaky,+ stable_fail=stable_fail,+ errors=errors,+ cases=repeated_results,+ )diff --git a/semantica/evals/types.py b/semantica/evals/types.pyindex 54b01edf..37216a4d 100644--- a/semantica/evals/types.py+++ b/semantica/evals/types.py@@ -4,7 +4,7 @@ Defines the metric and result shapes produced by the evals module."""from dataclasses import dataclass, field-from typing import Any, Dict, List, NamedTuple+from typing import Any, Dict, List, NamedTuple, Optional@dataclass(frozen=True)@@ -35,3 +35,56 @@ class EvalSummary:errors: intpass_rate: floatcases: List[CaseResult] = field(default_factory=list)+++@dataclass(frozen=True)+class SampleStats:+ """Per-evaluator statistics over repeated runs of one case.++ ``n`` is the number of runs, ``passes`` the number that passed, and the+ rest are derived views over those runs. ``samples`` keeps the raw metrics.+ ``objective_passed`` is None unless a ``direction`` objective is configured,+ in which case it gates the aggregate ``pass_rate`` with the same threshold.+ """++ n: int+ passes: int+ errors: int+ pass_rate: float+ mean_score: float+ stddev: float+ any_passed: bool+ all_passed: bool+ objective_passed: Optional[bool] = None+ samples: List[EvalMetric] = field(default_factory=list)+++@dataclass(frozen=True)+class RepeatedCaseResult:+ """Per-case outcome after repeated sampling.++ ``verdict`` classifies the case across runs:+ ``stable_pass`` (all runs passed), ``flaky`` (mixed), ``stable_fail``+ (none passed), or ``error`` (a run errored). ``stats`` holds per-evaluator+ ``SampleStats``.+ """++ case_id: str+ verdict: str+ stats: Dict[str, SampleStats]+++@dataclass+class RepeatedSummary:+ """Aggregate outcome across cases from repeated sampling."""++ runs: int+ stable_pass: int+ flaky: int+ stable_fail: int+ errors: int+ cases: List[RepeatedCaseResult] = field(default_factory=list)++ @property+ def total(self) -> int:+ return len(self.cases)diff --git a/semantica/evals/usage.md b/semantica/evals/usage.mdindex aab237ae..b7e15d65 100644--- a/semantica/evals/usage.md+++ b/semantica/evals/usage.md@@ -86,6 +86,38 @@ summary = evaluate(cases, ["decision_scores"])`llm_as_judge`; per-case or top-level `config` may carry per-evaluator settings(e.g. `config={"exact_match": {...}}`).+## Repeated sampling for nondeterministic targets++When your target is nondeterministic (LLM-backed extraction, agent pipeline),+a single verdict says little about stability. `evaluate_repeated` reruns+`target_fn` per case `n` times and aggregates per-evaluator statistics:++```python+from semantica.evals import evaluate_repeated++summary = evaluate_repeated(+ cases, # dict or (expected, actual) tuple cases+ evaluators=["exact_match"],+ target_fn=pipeline_run, # called once per run for a fresh `actual`+ runs=10,+)++summary.cases[0].verdict # "stable_pass" | "flaky" | "stable_fail" | "error"+stat = summary.cases[0].stats["exact_match"]+stat.pass_rate # passes / n, e.g. 0.8+stat.any_passed # observed pass@n+stat.all_passed # observed pass^n+stat.mean_score, stat.stddev # distribution over sampled scores+```++`target_fn` must be supplied and case `actual` must be left out: a case with a+non-null static `actual` (rather than a `target_fn`) has nothing to sample and+raises `ValueError` with `runs > 1`. An `actual` of `None` is treated as absent,+matching `evaluate()`, so it falls back to the resolver. Objective config applies+per run as in `evaluate()` and the same objective gates the aggregate pass rate:+`stat.objective_passed` reports `pass_rate >= 0.8` for a+`{direction: maximize, threshold: 0.8}` objective.+## Set per-evaluator objectivesBy default each evaluator decides its own pass/fail. To override thatdiff --git a/tests/evals/test_repeater.py b/tests/evals/test_repeater.pynew file mode 100644index 00000000..2ef74cf4--- /dev/null+++ b/tests/evals/test_repeater.py@@ -0,0 +1,369 @@+"""Tests for ``evaluate_repeated`` (repeated-sampling evaluation)."""++import math++import pytest++from semantica.evals import (+ RepeatedCaseResult,+ RepeatedSummary,+ SampleStats,+ evaluate_repeated,+)+from semantica.evals.registry import EVALUATORS, register+from semantica.evals.types import EvalMetric++CALLS = {"n": 0}+++def grade_linear():+ """Register a graded evaluator for the mean/stddev test, then remove it."""++ @register("_grade_linear")+ def _grade_linear(actual, expected=None, config=None, **kwargs):+ return EvalMetric(score=float(actual), passed=float(actual) >= 0.0)++ yield _grade_linear+ EVALUATORS.pop("_grade_linear", None)+++def seq(*results):+ """Return a shared-counter target_fn that yields ``results``, repeating last."""++ def _fn(_case):+ CALLS["n"] += 1+ idx = min(CALLS["n"] - 1, len(results) - 1)+ return results[idx]++ return _fn+++def make_seq(*results):+ """Independent-counter target_fn yielding ``results``, repeating the last."""+ state = {"i": 0}++ def _fn(_case):+ idx = min(state["i"], len(results) - 1)+ state["i"] += 1+ return results[idx]++ return _fn+++def always(actual):+ return lambda _case: actual+++def _reset():+ CALLS["n"] = 0+++def test_exports():+ from semantica.evals import evaluate, evaluate_repeated++ assert callable(evaluate) and callable(evaluate_repeated)+++def test_stable_pass_when_all_runs_pass():+ _reset()+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ target_fn=always("ok"),+ runs=5,+ )+ assert isinstance(summary, RepeatedSummary)+ assert summary.runs == 5+ assert summary.stable_pass == 1+ assert summary.flaky == 0+ assert summary.stable_fail == 0+ result = summary.cases[0]+ assert isinstance(result, RepeatedCaseResult)+ assert result.verdict == "stable_pass"+ stat = result.stats["exact_match"]+ assert isinstance(stat, SampleStats)+ assert stat.n == 5+ assert stat.passes == 5+ assert stat.pass_rate == 1.0+ assert stat.any_passed is True+ assert stat.all_passed is True+ assert stat.stddev == 0.0+++def test_flaky_detection_mixed_results():+ _reset()+ # 4 pass, 1 fail -> flaky+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ target_fn=seq("ok", "ok", "bad", "ok", "ok"),+ runs=5,+ )+ assert summary.cases[0].verdict == "flaky"+ stat = summary.cases[0].stats["exact_match"]+ assert stat.passes == 4+ assert stat.pass_rate == pytest.approx(0.8)+ assert stat.any_passed is True+ assert stat.all_passed is False+ assert summary.flaky == 1+++def test_stable_fail_when_no_run_passes():+ _reset()+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ target_fn=always("bad"),+ runs=5,+ )+ assert summary.cases[0].verdict == "stable_fail"+ stat = summary.cases[0].stats["exact_match"]+ assert stat.passes == 0+ assert stat.pass_rate == 0.0+ assert stat.any_passed is False+ assert stat.all_passed is False+++def test_static_actual_with_runs_gt_1_raises():+ _reset()+ with pytest.raises(ValueError, match=r"static ``actual``"):+ evaluate_repeated(+ [{"id": "c1", "expected": "ok", "actual": "ok"}],+ ["exact_match"],+ runs=3,+ )+++def test_static_actual_runs_1_is_allowed():+ _reset()+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok", "actual": "ok"}],+ ["exact_match"],+ runs=1,+ )+ assert summary.cases[0].verdict == "stable_pass"+ assert summary.cases[0].stats["exact_match"].n == 1+++def test_runs_zero_raises():+ _reset()+ with pytest.raises(ValueError, match="runs must be >= 1"):+ evaluate_repeated([], ["exact_match"], runs=0)+++def test_target_fn_exception_becomes_error_verdict():+ _reset()++ def boom(_case):+ raise RuntimeError("boom")++ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ target_fn=boom,+ runs=5,+ )+ result = summary.cases[0]+ assert result.verdict == "error"+ assert summary.errors == 1+ # every errored sample carries the target_fn failure message+ stat = result.stats["exact_match"]+ assert stat.errors == 5+ assert all(m.meta.get("error", "").startswith("target_fn:") for m in stat.samples)+++def test_error_takes_precedence_over_flaky():+ _reset()++ def sometimes_boom(case, _state={"i": 0}):+ _state["i"] += 1+ if _state["i"] == 2:+ raise RuntimeError("boom")+ return "ok"++ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ target_fn=sometimes_boom,+ runs=5,+ )+ assert summary.cases[0].verdict == "error"+++def test_objective_applies_per_run():+ # objective applies per run as in evaluate(); this test pins that the+ # maximize threshold still flips the per-run verdict for exact_match+ _reset()+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ config={+ "exact_match": {"objective": {"direction": "maximize", "threshold": 0.6}}+ },+ target_fn=seq("ok", "ok", "bad", "ok", "ok"),+ runs=5,+ )+ # exact_match scores 1.0/0.0; pass_rate aggregates the per-run passes+ stat = summary.cases[0].stats["exact_match"]+ assert stat.passes == 4+ # the same objective also gates the aggregate pass_rate (0.8 >= 0.6)+ assert stat.objective_passed is True+++def test_aggregate_objective_gates_pass_rate():+ # a nudge below the threshold flips objective_passed without changing the+ # distribution verdict: 3/5 passes gives pass_rate 0.6 < 0.8+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ config={+ "exact_match": {"objective": {"direction": "maximize", "threshold": 0.8}}+ },+ target_fn=make_seq("ok", "ok", "bad", "ok", "bad"),+ runs=5,+ )+ stat = summary.cases[0].stats["exact_match"]+ assert stat.passes == 3+ assert stat.pass_rate == pytest.approx(0.6)+ assert stat.objective_passed is False+ # verdict still reflects distribution, not the aggregate threshold+ assert summary.cases[0].verdict == "flaky"+++def test_mean_and_stddev_reflect_scores(grade_linear):+ # _grade_linear scores mirror the numeric actual, so 1..5 -> mean 3, stddev sqrt(2)+ summary = evaluate_repeated(+ [{"id": "c1"}],+ ["_grade_linear"],+ target_fn=make_seq(1.0, 2.0, 3.0, 4.0, 5.0),+ runs=5,+ )+ stat = summary.cases[0].stats["_grade_linear"]+ assert stat.mean_score == pytest.approx(3.0)+ assert stat.stddev == pytest.approx(math.sqrt(2.0))+ assert stat.any_passed is True+ assert stat.all_passed is True+++def test_multiple_cases_aggregation():+ # each case owns an independent target_fn, so verdicts are deterministic+ summary = evaluate_repeated(+ [+ {"id": "a", "expected": "ok", "target_fn": make_seq("ok", "ok", "ok")},+ {"id": "b", "expected": "ok", "target_fn": make_seq("ok", "bad", "ok")},+ {"id": "c", "expected": "ok", "target_fn": make_seq("bad", "bad", "bad")},+ ],+ ["exact_match"],+ runs=3,+ )+ assert summary.total == 3+ verdicts = {r.case_id: r.verdict for r in summary.cases}+ assert verdicts["a"] == "stable_pass"+ assert verdicts["b"] == "flaky"+ assert verdicts["c"] == "stable_fail"+++def test_missing_target_fn_is_error():+ _reset()+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ runs=3,+ )+ assert summary.cases[0].verdict == "error"+ assert summary.errors == 1+++def test_multi_evaluator_flaky_when_verdicts_diverge():+ # exact_match (checks "ok") passes every run; keyword_check (requires "k")+ # fails every run -> pooled across evaluators the case is flaky.+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match", "keyword_check"],+ config={"keyword_check": {"required": ["k"]}},+ target_fn=always("ok"),+ runs=4,+ )+ result = summary.cases[0]+ assert result.verdict == "flaky"+ assert result.stats["exact_match"].all_passed is True+ assert result.stats["keyword_check"].all_passed is False+++def test_minimize_objective_applies_per_run():+ # exact_match scores 1.0 on a match; a minimize threshold of 0.0 flips that+ # pass to a fail -> every run fails -> stable_fail.+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ config={+ "exact_match": {"objective": {"direction": "minimize", "threshold": 0.0}}+ },+ target_fn=always("ok"),+ runs=3,+ )+ result = summary.cases[0]+ assert result.verdict == "stable_fail"+ assert result.stats["exact_match"].passes == 0+ # aggregate gate mirrors the per-run direction: pass_rate 0.0 <= 0.0+ assert result.stats["exact_match"].objective_passed is True+++def test_empty_evaluators_rejected():+ with pytest.raises(ValueError, match="at least one evaluator"):+ evaluate_repeated([{"id": "c1"}], [], runs=3)+++def test_duplicate_evaluators_rejected():+ with pytest.raises(ValueError, match="must be unique"):+ evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match", "exact_match"],+ target_fn=always("ok"),+ runs=3,+ )+++def test_missing_target_errors_all_runs():+ # no static actual and no resolver: every requested run records an error, so+ # SampleStats.n matches runs rather than stopping after the first.+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["exact_match"],+ runs=4,+ )+ result = summary.cases[0]+ assert result.verdict == "error"+ stat = result.stats["exact_match"]+ assert stat.n == 4+ assert stat.errors == 4+ assert summary.runs == 4+++def test_tuple_case_form_is_rejected_for_sampling():+ # (expected, actual) tuple form always carries a static actual, so with+ # runs > 1 it is rejected up front, consistent with static-actual cases.+ with pytest.raises(ValueError, match="static ``actual``"):+ evaluate_repeated(+ [("ok", "ok")],+ ["exact_match"],+ target_fn=seq("ok", "bad", "ok"),+ runs=3,+ )+++def test_unknown_evaluator_surfaces_as_error_verdict():+ # An unknown evaluator raising from get_evaluator is caught per-run and the+ # case becomes an error verdict, consistent with the single-run evaluate().+ _reset()+ summary = evaluate_repeated(+ [{"id": "c1", "expected": "ok"}],+ ["nope"],+ target_fn=always("ok"),+ runs=2,+ )+ assert summary.cases[0].verdict == "error"+ assert summary.errors == 1+ # the per-run error carries the unknown-evaluator message+ assert summary.cases[0].stats["nope"].samples[0].meta["error"]