Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

feat(evals): repeated sampling for nondeterministic targets (#1550)

Kevinf6657709e9bf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 52aaee9a..9e3652da 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Persistent-cache keys are process-stable: relation/triplet extraction now folds entity/relation inputs into the key via a deterministic SHA-256 fingerprint instead of the process-randomized built-in `hash()`, so entries reliably rehit after a restart. `ttl=0` expires immediately on both backends, and corrupt/undeserializable rows are dropped rather than retained
- **Security / trust model:** the sqlite file is deserialized back into the process (default serializer `pickle`), so it must point at a trusted, user-private location and may hold sensitive extraction results in the clear (documented in the module usage guide). The default database lives under a per-user private directory (`$XDG_CACHE_HOME`/`~/.cache`, `0o700`); the db file is created **atomically** with `0o600` (`O_CREAT|O_EXCL`, no validate-then-open window) and an existing symlink, non-regular file, or file owned by another user is refused (falling back to in-memory). sqlite access uses a busy-timeout with bounded retry, rolls back within the same lock scope on failure, and contains all errors (reads degrade to a miss, writes to a skipped update)
- No new runtime dependencies (stdlib `sqlite3`). New public exports: `CacheBackend`, `InMemoryBackend`, `SqliteCacheBackend`, `configure_cache`. New `tests/semantic_extract/test_cache_backends.py`
+
+- **Repeated sampling for nondeterministic evals (#1545)**
+ - New `evaluate_repeated(cases, evaluators, config=None, target_fn=None, runs=10)` in `semantica.evals`: reruns `target_fn` per case `n` times and aggregates per-evaluator `SampleStats` — `n`, `passes`, `errors`, `pass_rate`, `mean_score`, `stddev`, and the observed `any_passed` (pass@n) / `all_passed` (pass^n). Verdicts classify each case as `stable_pass` / `flaky` / `stable_fail` / `error`, with a `RepeatedSummary` over the suite
+ - Nondeterminism stays in `target_fn`; evaluators remain pure. A case with a non-null static `actual` and `runs > 1` raises `ValueError` up front; an `actual` of `None` is treated as absent, matching `evaluate()`. Objective config applies per run and the same objective gates the aggregate `pass_rate` via `SampleStats.objective_passed`. `evaluate()` is untouched; new result types (`SampleStats`, `RepeatedCaseResult`, `RepeatedSummary`) are exported additively
+ - New `tests/evals/test_repeater.py`
+
- **Schema-guided extraction validation** (#1510) by @Besokus
- New `SchemaValidator` (`semantica.semantic_extract`, lazy export): a deterministic sibling of `ExtractionValidator` that checks extraction output for *conformance to a domain ontology* — an axis orthogonal to `ExtractionValidator`'s confidence checks. It mirrors the same interface (`validate_entities()` / `validate_relations()` returning `ValidationResult`, batch-aware), so the two compose back-to-back
- Entity labels must be concepts in the schema; relation predicates must be in the schema and satisfy their `domain` / `range`. Violations are reported in `ValidationResult.errors` with counts in `metrics` and `score` = conformance ratio; `filter_by_schema()` / `filter_relations_by_schema()` return the conforming subset (mirroring `filter_by_confidence`). No LLM required
diff --git a/docs/modules.md b/docs/modules.md
index 9c47f1cc..7ef5a6ad 100644
--- a/docs/modules.md
+++ b/docs/modules.md
@@ -506,7 +506,7 @@ summary = evaluate(cases, evaluators=["levenshtein"])
print(summary.total, summary.passed, summary.pass_rate)
```
-**Public API:** `evaluate(cases, evaluators, config=None)`, `list_evaluators()`, `get_evaluator(name)`, and the `EvalMetric` / `CaseResult` / `EvalSummary` result types. See the [Evals reference](/reference/evals).
+**Public API:** `evaluate(cases, evaluators, config=None)`, `evaluate_repeated(cases, evaluators, config=None, target_fn=None, runs=10)` for repeated sampling over nondeterministic targets, `list_evaluators()`, `get_evaluator(name)`, and the `EvalMetric` / `CaseResult` / `EvalSummary` / `SampleStats` / `RepeatedCaseResult` / `RepeatedSummary` result types. See the [Evals reference](/reference/evals).
### Core
diff --git a/docs/reference/evals.md b/docs/reference/evals.md
index a465bb68..5c59a346 100644
--- a/docs/reference/evals.md
+++ b/docs/reference/evals.md
@@ -30,15 +30,19 @@ assert on in tests, or track across runs.
| Name | Kind | Role |
| :--- | :--- | :--- |
| `evaluate(cases, evaluators, config=None, target_fn=None)` | function | Run named evaluators over each case, return an `EvalSummary` |
+| `evaluate_repeated(cases, evaluators, config=None, target_fn=None, runs=10)` | function | Rerun `target_fn` per case and aggregate stats, return a `RepeatedSummary` |
| `list_evaluators()` | function | Sorted names of every registered evaluator |
| `get_evaluator(name)` | function | Look up a single evaluator function by name |
| `EvalMetric` | dataclass (frozen) | One evaluator's result: `score`, `passed`, `meta` |
| `CaseResult` | namedtuple | One case's result: `case_id`, `status`, `metrics`, `details` |
| `EvalSummary` | dataclass | Aggregate across cases: `total`, `passed`, `failed`, `errors`, `pass_rate`, `cases` |
+| `SampleStats` | dataclass (frozen) | Per-evaluator statistics over repeated runs: `n`, `passes`, `errors`, `pass_rate`, `mean_score`, `stddev`, `any_passed`, `all_passed`, `objective_passed`, `samples` |
+| `RepeatedCaseResult` | dataclass (frozen) | Repeated outcome for one case: `case_id`, `verdict`, `stats` |
+| `RepeatedSummary` | dataclass | Aggregate across repeated-sampled cases: `runs`, `stable_pass`, `flaky`, `stable_fail`, `errors`, `cases` |
```python
import semantica.evals as evals
-from semantica.evals import evaluate, list_evaluators, get_evaluator
+from semantica.evals import evaluate, evaluate_repeated, list_evaluators, get_evaluator
```
## Built-in evaluators
@@ -204,10 +208,54 @@ for case in summary.cases:
print(metric.meta.get("reasons", {})) # per-sub-check failure reasons
```
-`EvalMetric` is frozen (`score: float`, `passed: bool`, `meta: dict`). `CaseResult`
-is a namedtuple, and `EvalSummary` is a plain dataclass, so all three are
+`EvalMetric`, `SampleStats`, and `RepeatedCaseResult` are frozen dataclasses
+(`score: float`, `passed: bool`, `meta: dict`, ...). `CaseResult` is a
+namedtuple and the `*Summary` classes are plain dataclasses, so all are
straightforward to serialize for logging or regression tracking.
+## Repeated sampling
+
+For nondeterministic targets (LLM-backed extraction, agent pipelines,
+sampling-based judges) a single verdict says little. `evaluate_repeated` reruns
+`target_fn` per case `n` times and aggregates per-evaluator statistics:
+
+```python
+from semantica.evals import evaluate_repeated
+
+summary = evaluate_repeated(
+ cases, # must be dict cases; see below
+ evaluators=["exact_match"],
+ target_fn=pipeline_run, # called once per run to produce a fresh `actual`
+ runs=10,
+)
+
+summary.runs # 10
+summary.cases[0].verdict # "stable_pass" | "flaky" | "stable_fail" | "error"
+summary.cases[0].stats["exact_match"].pass_rate # e.g. 0.8
+summary.cases[0].stats["exact_match"].stddev # e.g. 0.13
+```
+
+Per-evaluator `SampleStats` carries `n`, `passes`, `errors`, `pass_rate`
+(`passes / n`), `mean_score`, `stddev`, and the derived booleans `any_passed`
+(observed pass@n) and `all_passed` (observed pass^n). Verdicts pool all samples:
+`stable_pass` when every run passed every evaluator, `stable_fail` when none
+passed, `flaky` when the pool mixes passes and fails, and `error` when a run
+errored.
+
+Requirements and edge cases:
+
+- Each case can be a dict or a `(expected, actual)` tuple, matching `evaluate()`.
+ A case that carries a **non-null** static `actual` has nothing to sample; using
+ it with `runs > 1` raises `ValueError`. An `actual` of `None` is treated as
+ absent, so it falls back to the resolver exactly as in `evaluate()`.
+- `target_fn` exceptions and evaluator failures become per-run error samples and
+ mark the case `error`; they never crash the run.
+- The objective layer applies per run as in `evaluate()`, and the same objective
+ gates the aggregate `pass_rate` through `SampleStats.objective_passed` (e.g.
+ `{direction: maximize, threshold: 0.8}` requires an 80% pass rate across runs).
+- At least one evaluator is required, and evaluator names must be unique.
+- `runs` must be `>= 1`.
+
## Notes
- `llm_as_judge` needs `config["judge_fn"]`, a callable
diff --git a/semantica/evals/__init__.py b/semantica/evals/__init__.py
index af1dd7af..352422c0 100644
--- a/semantica/evals/__init__.py
+++ b/semantica/evals/__init__.py
@@ -7,15 +7,26 @@ runner for measuring decision records, audit trails, and reasoning output.
from . import decision_evaluators # noqa: F401 (registers decision_scores)
from . import evaluators # noqa: F401 (registers the generic evaluators)
from .registry import get_evaluator, list_evaluators
-from .runner import evaluate
-from .types import CaseResult, EvalMetric, EvalSummary
+from .runner import evaluate, evaluate_repeated
+from .types import (
+ CaseResult,
+ EvalMetric,
+ EvalSummary,
+ RepeatedCaseResult,
+ RepeatedSummary,
+ SampleStats,
+)
__version__ = "0.1.0"
__all__ = [
"evaluate",
+ "evaluate_repeated",
"get_evaluator",
"list_evaluators",
"CaseResult",
"EvalMetric",
"EvalSummary",
+ "RepeatedCaseResult",
+ "RepeatedSummary",
+ "SampleStats",
]
diff --git a/semantica/evals/runner.py b/semantica/evals/runner.py
index 815c4762..a77ac949 100644
--- a/semantica/evals/runner.py
+++ b/semantica/evals/runner.py
@@ -4,7 +4,14 @@ import math
from typing import Any, Callable, Dict, List, Optional, Tuple, Union
from .registry import get_evaluator
-from .types import CaseResult, EvalMetric, EvalSummary
+from .types import (
+ CaseResult,
+ EvalMetric,
+ EvalSummary,
+ RepeatedCaseResult,
+ RepeatedSummary,
+ SampleStats,
+)
Case = Union[Dict[str, Any], Tuple[Any, Any]]
@@ -64,12 +71,18 @@ def _parse_objective(name, eval_config):
raise ValueError(
f"objective for '{name}': 'minimize' requires a 'threshold'"
)
- return {"direction": "minimize", "threshold": _coerce_threshold(name, threshold)}
+ return {
+ "direction": "minimize",
+ "threshold": _coerce_threshold(name, threshold),
+ }
if direction == "maximize":
if threshold is None:
# no bar to re-decide against; treat as absent (evaluator default stands)
return None
- return {"direction": "maximize", "threshold": _coerce_threshold(name, threshold)}
+ return {
+ "direction": "maximize",
+ "threshold": _coerce_threshold(name, threshold),
+ }
raise ValueError(
f"objective for '{name}': 'direction' must be 'maximize' or 'minimize' "
f"(got {direction!r})"
@@ -98,7 +111,9 @@ def _extract(case: Case, target_fn: Optional[Callable]):
return case_id, expected, actual, config, per_fn
-def _merge_config(default_config: Dict[str, Any], case_config: Dict[str, Any]) -> Dict[str, Any]:
+def _merge_config(
+ default_config: Dict[str, Any], case_config: Dict[str, Any]
+) -> Dict[str, Any]:
"""Deep-merge per-case config over the global config (two levels deep).
Level 1 (top-level keys, e.g. evaluator names): merged key-by-key so a
@@ -185,7 +200,9 @@ def evaluate(
metric = get_evaluator(name)(actual, expected, config=eval_config)
objective = objective_by_name.get(name)
if objective is not None and "error" not in metric.meta:
- metric = EvalMetric(metric.score, _apply_objective(metric, objective), metric.meta)
+ metric = EvalMetric(
+ metric.score, _apply_objective(metric, objective), metric.meta
+ )
metrics[name] = metric
if "error" in metric.meta:
errored = True
@@ -206,6 +223,189 @@ def evaluate(
errors = sum(1 for c in case_results if c.status == "error")
pass_rate = (passed / total) if total else 1.0
return EvalSummary(
- total, passed, failed, errors, pass_rate,
+ total,
+ passed,
+ failed,
+ errors,
+ pass_rate,
cases=case_results,
)
+
+
+def _stddev(values: List[float]) -> float:
+ """Population standard deviation over ``values`` (0.0 for <2 samples)."""
+ if len(values) < 2:
+ return 0.0
+ mean = sum(values) / len(values)
+ variance = sum((v - mean) ** 2 for v in values) / len(values)
+ return math.sqrt(variance)
+
+
+def evaluate_repeated(
+ cases: List[Case],
+ evaluators: List[str],
+ config: Optional[Dict[str, Any]] = None,
+ target_fn: Optional[Callable] = None,
+ runs: int = 10,
+) -> RepeatedSummary:
+ """Rerun ``target_fn`` per case ``runs`` times and aggregate statistics.
+
+ Non-determinism in an evaluation lives in ``target_fn`` (an LLM call, a
+ sampling-based pipeline), not in the evaluators, which are pure functions
+ over a given ``actual``. So ``evaluate_repeated`` re-invokes ``target_fn``
+ on each run to draw a fresh ``actual``, then scores it with the named
+ evaluators. Aggregating those scores across runs surfaces distribution-level
+ evidence (pass rate, mean/stddev) and a per-case verdict:
+
+ - ``stable_pass``: every run passed every evaluator
+ - ``flaky``: at least one pass and one fail across runs/evaluators
+ - ``stable_fail``: no run passed any evaluator
+ - ``error``: a run errored
+
+ Verdict for a multi-evaluator case pools samples across all evaluators: it
+ is ``stable_pass`` only when every sample of every evaluator passed, and
+ ``flaky`` when the pool mixes passes and fails (e.g. one evaluator passing
+ every run while another passes none).
+
+ A case that carries a non-null static ``actual`` (rather than a ``target_fn``)
+ has nothing to sample, so with ``runs > 1`` it raises ``ValueError``, matching
+ the fail-fast style of invalid objective config. An ``actual`` of ``None`` is
+ treated as absent, exactly as in ``evaluate()``: it falls back to the
+ resolver. ``evaluate()`` is untouched; this is an additive entry point over
+ the same helpers.
+ """
+ if runs < 1:
+ raise ValueError(f"runs must be >= 1 (got {runs!r})")
+ if not evaluators:
+ raise ValueError("evaluate_repeated requires at least one evaluator")
+ if len(evaluators) != len(set(evaluators)):
+ raise ValueError(
+ f"evaluators must be unique (got duplicates: "
+ f"{sorted({n for n in evaluators if evaluators.count(n) > 1})})"
+ )
+ default_config = config or {}
+ repeated_results: List[RepeatedCaseResult] = []
+
+ # Validate object config and sampling prerequisites for every case up front
+ # so an invalid objective or an unsamplable static ``actual`` rejects the
+ # run before any target_fn or evaluator executes (fail-fast), regardless of
+ # which case carries it.
+ pre_resolved = []
+ for case in cases:
+ case_id, _, static_actual, case_config, _ = _extract(case, target_fn)
+ if static_actual is not None and runs > 1:
+ raise ValueError(
+ f"case '{case_id}': repeated sampling requires a streaming "
+ f"``target_fn``; a static ``actual`` can only be run once "
+ f"(runs={runs!r})"
+ )
+ merged = _merge_config(default_config, case_config)
+ pre_resolved.append(
+ {
+ name: _parse_objective(name, merged.get(name) or {})
+ for name in evaluators
+ }
+ )
+
+ for case, objective_by_name in zip(cases, pre_resolved):
+ case_id, expected, static_actual, case_config, per_fn = _extract(
+ case, target_fn
+ )
+ merged = _merge_config(default_config, case_config)
+ if expected is None:
+ expected = merged.get("expected")
+ resolver = per_fn or target_fn
+ has_actual = static_actual is not None
+
+ # All samples for this case, per evaluator.
+ per_eval: Dict[str, List[EvalMetric]] = {name: [] for name in evaluators}
+ errored_any = False
+ for _ in range(runs):
+ if not has_actual and resolver is None:
+ for name in evaluators:
+ per_eval[name].append(
+ EvalMetric(0.0, False, {"error": "no target_fn to sample"})
+ )
+ errored_any = True
+ continue
+ try:
+ actual = static_actual if has_actual else resolver(case)
+ except Exception as exc: # noqa: BLE001
+ errored_any = True
+ for name in evaluators:
+ per_eval[name].append(
+ EvalMetric(0.0, False, {"error": f"target_fn: {exc}"})
+ )
+ continue
+ for name in evaluators:
+ eval_config = merged.get(name) or {}
+ objective = objective_by_name.get(name)
+ try:
+ metric = get_evaluator(name)(actual, expected, config=eval_config)
+ if objective is not None and "error" not in metric.meta:
+ metric = EvalMetric(
+ metric.score,
+ _apply_objective(metric, objective),
+ metric.meta,
+ )
+ except Exception as exc: # noqa: BLE001
+ metric = EvalMetric(0.0, False, {"error": str(exc)})
+ if "error" in metric.meta:
+ errored_any = True
+ per_eval[name].append(metric)
+
+ # Aggregate per-evaluator stats.
+ stats: Dict[str, SampleStats] = {}
+ verdict = "error" if errored_any else None
+ for name in evaluators:
+ samples = per_eval[name]
+ n = len(samples)
+ passes = sum(1 for m in samples if m.passed and "error" not in m.meta)
+ errors = sum(1 for m in samples if "error" in m.meta)
+ scores = [m.score for m in samples if "error" not in m.meta]
+ pass_rate = passes / n if n else 0.0
+ mean_score = sum(scores) / len(scores) if scores else 0.0
+ objective = objective_by_name.get(name)
+ objective_passed = None
+ if objective is not None and "direction" in objective:
+ objective_passed = _apply_objective(
+ EvalMetric(pass_rate, True), objective
+ )
+ stats[name] = SampleStats(
+ n=n,
+ passes=passes,
+ errors=errors,
+ pass_rate=pass_rate,
+ mean_score=mean_score,
+ stddev=_stddev(scores),
+ any_passed=passes > 0,
+ all_passed=passes == n and n > 0 and errors == 0,
+ objective_passed=objective_passed,
+ samples=samples,
+ )
+ if verdict is None:
+ # Verdict across all evaluators: flaky if any mixed, else uniform.
+ passes_total = sum(stats[name].passes for name in evaluators)
+ n_total = sum(stats[name].n for name in evaluators)
+ if passes_total == 0:
+ verdict = "stable_fail"
+ elif passes_total == n_total:
+ verdict = "stable_pass"
+ else:
+ verdict = "flaky"
+ repeated_results.append(
+ RepeatedCaseResult(case_id=case_id, verdict=verdict, stats=stats)
+ )
+
+ stable_pass = sum(1 for r in repeated_results if r.verdict == "stable_pass")
+ flaky = sum(1 for r in repeated_results if r.verdict == "flaky")
+ stable_fail = sum(1 for r in repeated_results if r.verdict == "stable_fail")
+ errors = sum(1 for r in repeated_results if r.verdict == "error")
+ return RepeatedSummary(
+ runs=runs,
+ stable_pass=stable_pass,
+ flaky=flaky,
+ stable_fail=stable_fail,
+ errors=errors,
+ cases=repeated_results,
+ )
diff --git a/semantica/evals/types.py b/semantica/evals/types.py
index 54b01edf..37216a4d 100644
--- a/semantica/evals/types.py
+++ b/semantica/evals/types.py
@@ -4,7 +4,7 @@ Defines the metric and result shapes produced by the evals module.
"""
from dataclasses import dataclass, field
-from typing import Any, Dict, List, NamedTuple
+from typing import Any, Dict, List, NamedTuple, Optional
@dataclass(frozen=True)
@@ -35,3 +35,56 @@ class EvalSummary:
errors: int
pass_rate: float
cases: List[CaseResult] = field(default_factory=list)
+
+
+@dataclass(frozen=True)
+class SampleStats:
+ """Per-evaluator statistics over repeated runs of one case.
+
+ ``n`` is the number of runs, ``passes`` the number that passed, and the
+ rest are derived views over those runs. ``samples`` keeps the raw metrics.
+ ``objective_passed`` is None unless a ``direction`` objective is configured,
+ in which case it gates the aggregate ``pass_rate`` with the same threshold.
+ """
+
+ n: int
+ passes: int
+ errors: int
+ pass_rate: float
+ mean_score: float
+ stddev: float
+ any_passed: bool
+ all_passed: bool
+ objective_passed: Optional[bool] = None
+ samples: List[EvalMetric] = field(default_factory=list)
+
+
+@dataclass(frozen=True)
+class RepeatedCaseResult:
+ """Per-case outcome after repeated sampling.
+
+ ``verdict`` classifies the case across runs:
+ ``stable_pass`` (all runs passed), ``flaky`` (mixed), ``stable_fail``
+ (none passed), or ``error`` (a run errored). ``stats`` holds per-evaluator
+ ``SampleStats``.
+ """
+
+ case_id: str
+ verdict: str
+ stats: Dict[str, SampleStats]
+
+
+@dataclass
+class RepeatedSummary:
+ """Aggregate outcome across cases from repeated sampling."""
+
+ runs: int
+ stable_pass: int
+ flaky: int
+ stable_fail: int
+ errors: int
+ cases: List[RepeatedCaseResult] = field(default_factory=list)
+
+ @property
+ def total(self) -> int:
+ return len(self.cases)
diff --git a/semantica/evals/usage.md b/semantica/evals/usage.md
index aab237ae..b7e15d65 100644
--- a/semantica/evals/usage.md
+++ b/semantica/evals/usage.md
@@ -86,6 +86,38 @@ summary = evaluate(cases, ["decision_scores"])
`llm_as_judge`; per-case or top-level `config` may carry per-evaluator settings
(e.g. `config={"exact_match": {...}}`).
+## Repeated sampling for nondeterministic targets
+
+When your target is nondeterministic (LLM-backed extraction, agent pipeline),
+a single verdict says little about stability. `evaluate_repeated` reruns
+`target_fn` per case `n` times and aggregates per-evaluator statistics:
+
+```python
+from semantica.evals import evaluate_repeated
+
+summary = evaluate_repeated(
+ cases, # dict or (expected, actual) tuple cases
+ evaluators=["exact_match"],
+ target_fn=pipeline_run, # called once per run for a fresh `actual`
+ runs=10,
+)
+
+summary.cases[0].verdict # "stable_pass" | "flaky" | "stable_fail" | "error"
+stat = summary.cases[0].stats["exact_match"]
+stat.pass_rate # passes / n, e.g. 0.8
+stat.any_passed # observed pass@n
+stat.all_passed # observed pass^n
+stat.mean_score, stat.stddev # distribution over sampled scores
+```
+
+`target_fn` must be supplied and case `actual` must be left out: a case with a
+non-null static `actual` (rather than a `target_fn`) has nothing to sample and
+raises `ValueError` with `runs > 1`. An `actual` of `None` is treated as absent,
+matching `evaluate()`, so it falls back to the resolver. Objective config applies
+per run as in `evaluate()` and the same objective gates the aggregate pass rate:
+`stat.objective_passed` reports `pass_rate >= 0.8` for a
+`{direction: maximize, threshold: 0.8}` objective.
+
## Set per-evaluator objectives
By default each evaluator decides its own pass/fail. To override that
diff --git a/tests/evals/test_repeater.py b/tests/evals/test_repeater.py
new file mode 100644
index 00000000..2ef74cf4
--- /dev/null
+++ b/tests/evals/test_repeater.py
@@ -0,0 +1,369 @@
+"""Tests for ``evaluate_repeated`` (repeated-sampling evaluation)."""
+
+import math
+
+import pytest
+
+from semantica.evals import (
+ RepeatedCaseResult,
+ RepeatedSummary,
+ SampleStats,
+ evaluate_repeated,
+)
+from semantica.evals.registry import EVALUATORS, register
+from semantica.evals.types import EvalMetric
+
+CALLS = {"n": 0}
+
+
+def grade_linear():
+ """Register a graded evaluator for the mean/stddev test, then remove it."""
+
+ @register("_grade_linear")
+ def _grade_linear(actual, expected=None, config=None, **kwargs):
+ return EvalMetric(score=float(actual), passed=float(actual) >= 0.0)
+
+ yield _grade_linear
+ EVALUATORS.pop("_grade_linear", None)
+
+
+def seq(*results):
+ """Return a shared-counter target_fn that yields ``results``, repeating last."""
+
+ def _fn(_case):
+ CALLS["n"] += 1
+ idx = min(CALLS["n"] - 1, len(results) - 1)
+ return results[idx]
+
+ return _fn
+
+
+def make_seq(*results):
+ """Independent-counter target_fn yielding ``results``, repeating the last."""
+ state = {"i": 0}
+
+ def _fn(_case):
+ idx = min(state["i"], len(results) - 1)
+ state["i"] += 1
+ return results[idx]
+
+ return _fn
+
+
+def always(actual):
+ return lambda _case: actual
+
+
+def _reset():
+ CALLS["n"] = 0
+
+
+def test_exports():
+ from semantica.evals import evaluate, evaluate_repeated
+
+ assert callable(evaluate) and callable(evaluate_repeated)
+
+
+def test_stable_pass_when_all_runs_pass():
+ _reset()
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ target_fn=always("ok"),
+ runs=5,
+ )
+ assert isinstance(summary, RepeatedSummary)
+ assert summary.runs == 5
+ assert summary.stable_pass == 1
+ assert summary.flaky == 0
+ assert summary.stable_fail == 0
+ result = summary.cases[0]
+ assert isinstance(result, RepeatedCaseResult)
+ assert result.verdict == "stable_pass"
+ stat = result.stats["exact_match"]
+ assert isinstance(stat, SampleStats)
+ assert stat.n == 5
+ assert stat.passes == 5
+ assert stat.pass_rate == 1.0
+ assert stat.any_passed is True
+ assert stat.all_passed is True
+ assert stat.stddev == 0.0
+
+
+def test_flaky_detection_mixed_results():
+ _reset()
+ # 4 pass, 1 fail -> flaky
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ target_fn=seq("ok", "ok", "bad", "ok", "ok"),
+ runs=5,
+ )
+ assert summary.cases[0].verdict == "flaky"
+ stat = summary.cases[0].stats["exact_match"]
+ assert stat.passes == 4
+ assert stat.pass_rate == pytest.approx(0.8)
+ assert stat.any_passed is True
+ assert stat.all_passed is False
+ assert summary.flaky == 1
+
+
+def test_stable_fail_when_no_run_passes():
+ _reset()
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ target_fn=always("bad"),
+ runs=5,
+ )
+ assert summary.cases[0].verdict == "stable_fail"
+ stat = summary.cases[0].stats["exact_match"]
+ assert stat.passes == 0
+ assert stat.pass_rate == 0.0
+ assert stat.any_passed is False
+ assert stat.all_passed is False
+
+
+def test_static_actual_with_runs_gt_1_raises():
+ _reset()
+ with pytest.raises(ValueError, match=r"static ``actual``"):
+ evaluate_repeated(
+ [{"id": "c1", "expected": "ok", "actual": "ok"}],
+ ["exact_match"],
+ runs=3,
+ )
+
+
+def test_static_actual_runs_1_is_allowed():
+ _reset()
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok", "actual": "ok"}],
+ ["exact_match"],
+ runs=1,
+ )
+ assert summary.cases[0].verdict == "stable_pass"
+ assert summary.cases[0].stats["exact_match"].n == 1
+
+
+def test_runs_zero_raises():
+ _reset()
+ with pytest.raises(ValueError, match="runs must be >= 1"):
+ evaluate_repeated([], ["exact_match"], runs=0)
+
+
+def test_target_fn_exception_becomes_error_verdict():
+ _reset()
+
+ def boom(_case):
+ raise RuntimeError("boom")
+
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ target_fn=boom,
+ runs=5,
+ )
+ result = summary.cases[0]
+ assert result.verdict == "error"
+ assert summary.errors == 1
+ # every errored sample carries the target_fn failure message
+ stat = result.stats["exact_match"]
+ assert stat.errors == 5
+ assert all(m.meta.get("error", "").startswith("target_fn:") for m in stat.samples)
+
+
+def test_error_takes_precedence_over_flaky():
+ _reset()
+
+ def sometimes_boom(case, _state={"i": 0}):
+ _state["i"] += 1
+ if _state["i"] == 2:
+ raise RuntimeError("boom")
+ return "ok"
+
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ target_fn=sometimes_boom,
+ runs=5,
+ )
+ assert summary.cases[0].verdict == "error"
+
+
+def test_objective_applies_per_run():
+ # objective applies per run as in evaluate(); this test pins that the
+ # maximize threshold still flips the per-run verdict for exact_match
+ _reset()
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ config={
+ "exact_match": {"objective": {"direction": "maximize", "threshold": 0.6}}
+ },
+ target_fn=seq("ok", "ok", "bad", "ok", "ok"),
+ runs=5,
+ )
+ # exact_match scores 1.0/0.0; pass_rate aggregates the per-run passes
+ stat = summary.cases[0].stats["exact_match"]
+ assert stat.passes == 4
+ # the same objective also gates the aggregate pass_rate (0.8 >= 0.6)
+ assert stat.objective_passed is True
+
+
+def test_aggregate_objective_gates_pass_rate():
+ # a nudge below the threshold flips objective_passed without changing the
+ # distribution verdict: 3/5 passes gives pass_rate 0.6 < 0.8
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ config={
+ "exact_match": {"objective": {"direction": "maximize", "threshold": 0.8}}
+ },
+ target_fn=make_seq("ok", "ok", "bad", "ok", "bad"),
+ runs=5,
+ )
+ stat = summary.cases[0].stats["exact_match"]
+ assert stat.passes == 3
+ assert stat.pass_rate == pytest.approx(0.6)
+ assert stat.objective_passed is False
+ # verdict still reflects distribution, not the aggregate threshold
+ assert summary.cases[0].verdict == "flaky"
+
+
+def test_mean_and_stddev_reflect_scores(grade_linear):
+ # _grade_linear scores mirror the numeric actual, so 1..5 -> mean 3, stddev sqrt(2)
+ summary = evaluate_repeated(
+ [{"id": "c1"}],
+ ["_grade_linear"],
+ target_fn=make_seq(1.0, 2.0, 3.0, 4.0, 5.0),
+ runs=5,
+ )
+ stat = summary.cases[0].stats["_grade_linear"]
+ assert stat.mean_score == pytest.approx(3.0)
+ assert stat.stddev == pytest.approx(math.sqrt(2.0))
+ assert stat.any_passed is True
+ assert stat.all_passed is True
+
+
+def test_multiple_cases_aggregation():
+ # each case owns an independent target_fn, so verdicts are deterministic
+ summary = evaluate_repeated(
+ [
+ {"id": "a", "expected": "ok", "target_fn": make_seq("ok", "ok", "ok")},
+ {"id": "b", "expected": "ok", "target_fn": make_seq("ok", "bad", "ok")},
+ {"id": "c", "expected": "ok", "target_fn": make_seq("bad", "bad", "bad")},
+ ],
+ ["exact_match"],
+ runs=3,
+ )
+ assert summary.total == 3
+ verdicts = {r.case_id: r.verdict for r in summary.cases}
+ assert verdicts["a"] == "stable_pass"
+ assert verdicts["b"] == "flaky"
+ assert verdicts["c"] == "stable_fail"
+
+
+def test_missing_target_fn_is_error():
+ _reset()
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ runs=3,
+ )
+ assert summary.cases[0].verdict == "error"
+ assert summary.errors == 1
+
+
+def test_multi_evaluator_flaky_when_verdicts_diverge():
+ # exact_match (checks "ok") passes every run; keyword_check (requires "k")
+ # fails every run -> pooled across evaluators the case is flaky.
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match", "keyword_check"],
+ config={"keyword_check": {"required": ["k"]}},
+ target_fn=always("ok"),
+ runs=4,
+ )
+ result = summary.cases[0]
+ assert result.verdict == "flaky"
+ assert result.stats["exact_match"].all_passed is True
+ assert result.stats["keyword_check"].all_passed is False
+
+
+def test_minimize_objective_applies_per_run():
+ # exact_match scores 1.0 on a match; a minimize threshold of 0.0 flips that
+ # pass to a fail -> every run fails -> stable_fail.
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ config={
+ "exact_match": {"objective": {"direction": "minimize", "threshold": 0.0}}
+ },
+ target_fn=always("ok"),
+ runs=3,
+ )
+ result = summary.cases[0]
+ assert result.verdict == "stable_fail"
+ assert result.stats["exact_match"].passes == 0
+ # aggregate gate mirrors the per-run direction: pass_rate 0.0 <= 0.0
+ assert result.stats["exact_match"].objective_passed is True
+
+
+def test_empty_evaluators_rejected():
+ with pytest.raises(ValueError, match="at least one evaluator"):
+ evaluate_repeated([{"id": "c1"}], [], runs=3)
+
+
+def test_duplicate_evaluators_rejected():
+ with pytest.raises(ValueError, match="must be unique"):
+ evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match", "exact_match"],
+ target_fn=always("ok"),
+ runs=3,
+ )
+
+
+def test_missing_target_errors_all_runs():
+ # no static actual and no resolver: every requested run records an error, so
+ # SampleStats.n matches runs rather than stopping after the first.
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["exact_match"],
+ runs=4,
+ )
+ result = summary.cases[0]
+ assert result.verdict == "error"
+ stat = result.stats["exact_match"]
+ assert stat.n == 4
+ assert stat.errors == 4
+ assert summary.runs == 4
+
+
+def test_tuple_case_form_is_rejected_for_sampling():
+ # (expected, actual) tuple form always carries a static actual, so with
+ # runs > 1 it is rejected up front, consistent with static-actual cases.
+ with pytest.raises(ValueError, match="static ``actual``"):
+ evaluate_repeated(
+ [("ok", "ok")],
+ ["exact_match"],
+ target_fn=seq("ok", "bad", "ok"),
+ runs=3,
+ )
+
+
+def test_unknown_evaluator_surfaces_as_error_verdict():
+ # An unknown evaluator raising from get_evaluator is caught per-run and the
+ # case becomes an error verdict, consistent with the single-run evaluate().
+ _reset()
+ summary = evaluate_repeated(
+ [{"id": "c1", "expected": "ok"}],
+ ["nope"],
+ target_fn=always("ok"),
+ runs=2,
+ )
+ assert summary.cases[0].verdict == "error"
+ assert summary.errors == 1
+ # the per-run error carries the unknown-evaluator message
+ assert summary.cases[0].stats["nope"].samples[0].meta["error"]