DSPy: The framework for programming—not prompting—language models
Python
Add experimental ReAnchor calibration optimizer for Predict (#10475)
Isaac Millere8ec009c6bd9
diff --git a/docs/docs/api/experimental/DecisionTypes.md b/docs/docs/api/experimental/DecisionTypes.mdindex 9e104de91..e756a4bfb 100644--- a/docs/docs/api/experimental/DecisionTypes.md+++ b/docs/docs/api/experimental/DecisionTypes.md@@ -284,5 +284,7 @@ annotations. This restriction applies to outputs, not structured input data.Use TypeSafe through `Predict`, rather than calling chat adapters directly.Generation-specific wrappers such as `BestOfN` and `Refine`, generative optimizers,and fine-tuning are unsupported with TypeSafe.+[ReAnchor](ReAnchor.md) calibrates decision outputs with TypeSafe or a+generative LM, and it sends no generation settings.Wrappers that assume LM generation settings can currently raise `AttributeError`rather than a capability-specific error.diff --git a/docs/docs/api/experimental/ReAnchor.md b/docs/docs/api/experimental/ReAnchor.mdnew file mode 100644index 000000000..f31139ee3--- /dev/null+++ b/docs/docs/api/experimental/ReAnchor.md@@ -0,0 +1,148 @@+# dspy.experimental.ReAnchor++!!! warning "Experimental API"+ `ReAnchor` is experimental and may change without warning.++`ReAnchor` fits the numeric decision settings in a program to your metric. These+settings are the `threshold`, `cuts`, and `weights` entries in each predictor's+`fields`. They are described in [Decision types and System One models](DecisionTypes.md).+ReAnchor does not change any instructions, descriptions, or demos.++You give ReAnchor a metric, a program, and training examples. The program can be+a single `dspy.Predict` or any module that holds predictors with decision+outputs. A decision output is a `Noul`, `Score`, or `Choice` output, or a native+`bool` or `Literal` output. `RLM` does not support decision outputs, so ReAnchor+does not support RLM programs either.++```python+import dspy+from dspy.experimental import ReAnchor, TypeSafe+++class Match(dspy.Signature):+ """Decide whether two product listings describe the same item."""++ pair: str = dspy.InputField(desc="Two listings.")+ match: bool = dspy.OutputField(desc="Are they the same item?")+++def metric(example, prediction, trace=None):+ return float(prediction.match == example.match)+++dspy.configure(lm=TypeSafe("jev-latest"))+optimizer = ReAnchor(metric)+tuned = optimizer.compile(dspy.Predict(Match), trainset=trainset, valset=valset)+print(optimizer.report)+```++## What ReAnchor fits++ReAnchor searches each output's numeric settings against the whole-program+metric. A new setting must improve the training score and pass a fold check.++For every compatible output, ReAnchor saves the original configuration and score,+enables probability-based execution, and fits the numeric settings. It then compares+the fitted behavior against the original using a fold check. If rejected, it restores+the exact original configuration, including removing an entry that was originally absent.+The adapter handles the backend used on each call; calibration does not inspect it.++Before fitting each output, ReAnchor runs the program and records that output's+probabilities on every call. A threshold anywhere between two neighboring P(True) values makes the+same decisions, so ReAnchor tries the midpoint of each gap between them. For+example, when the model only returns P(True) of 0.98 and 1.0, ReAnchor tries+0.5 and 0.99.++- For a Boolean output, ReAnchor tries each midpoint between the observed+ P(True) values.+- For a Score output, the level depends on the mean level index. ReAnchor tries+ each `cut` at the midpoints between the observed mean indexes, and it keeps+ the cuts in order. The cuts choose `.level` and do not change `.value`.+- For a Choice output, ReAnchor moves one option's multiplier at a time. It+ tries the multipliers that fall between the points where that option's pick+ would flip on some call.++The current setting stays unless another setting scores strictly better and+passes a fold check. The check splits the training examples into up to five parts.+For each part, ReAnchor picks a setting using the remaining parts and scores+that pick on the held-out part. A new setting is kept only when the combined held-out+score beats the current setting. The check discourages gains confined to small+portions of the dataset, but can accept them when they recur across folds.+Among improving candidates with equal scores, ReAnchor prefers the widest gap,+which leaves the most room on either side. Each search step tries at most+40 gap midpoints, thinning large lists to settings spaced evenly through+the observed values. For Boolean outputs, it also tries threshold zero when+P(True)=0 is observed, since that boundary is the only way to classify those+answers as True. When two observed probabilities are adjacent floats, it tries+the upper value as a threshold: `p >= threshold` separates them without a midpoint.++## Requests and the cache++The numeric settings are not part of the request. Repeated identical requests+reuse cached answers. In a composed program, changing an upstream decision can+change downstream inputs or which predictors run, producing new requests and+additional backend calls. Native-output promotion also changes the request.++`compile` requires caching by default to avoid repeating identical backend+calls; it does not guarantee a fixed request count. Pass `require_cache=False`+to run without this check. The cache check inspects statically bound or globally+configured clients. For programs selecting clients inside `forward()`, disable+the check and manage caching on those clients.++## Native outputs on a generative LM++A generative LM answers a native `bool` or `Literal` output with a single value.+It does not report probabilities, so ReAnchor has nothing to fit. When an output+has an entry in the predictor's `fields`, `Predict` asks the LM for+probabilities instead. `Predict` then applies the threshold or weights to pick+the value, and the output still returns a `bool` or a `Literal` member.++The first pass with probabilities sends new requests, because the request changes.+If the fitted behavior is rejected, restoring the original configuration returns+an unconfigured native field to direct generation.++```python+dspy.configure(lm=dspy.LM("your-provider/your-model")) # Use your model's identifier.+tuned = ReAnchor(metric).compile(dspy.Predict(Match), trainset=trainset)+```++A System One model such as Jev always returns probabilities. Enabling probability+execution leaves its request unchanged; restoring a field restores its original+decoding settings, not direct generation.++## Errors++ReAnchor stops at the first error from the program or the metric. On a+generative LM, a malformed answer counts as an error. For example, `Predict`+raises when a `Score` answer leaves out a probability for any level. Use a+model that follows JSON schemas reliably, and set a client `timeout` that fits+your backend's load.++## Results++`compile` returns a copy of the program and leaves your program unchanged. After+`compile`, `report` holds:++- `train_score_before` and `train_score`, the mean metric score on the training+ examples before and after calibration.+- `val_score_before` and `val_score`, the same scores on `valset` when you pass+ one. ReAnchor never fits settings on `valset`.+- `fitted`, one row per output with the fitted value, or the reason ReAnchor+ skipped it. Each fitted row has an `observed` entry with the number of calls+ and settings tried; threshold and cut reports also summarize observed values. Its+ `fold_check` entry counts the search steps whose better training score passed+ or failed the fold check. Per-field `train_score_original` is the score before+ enabling probabilities, `train_score_at_start` is the probability-based baseline,+ and `train_score` describes the retained behavior. Rejected configurations have+ a `skipped` reason instead of a fitted `value`.++Your metric may return a number or a `dspy.Prediction` with a `score`.++When you set `log_dir`, ReAnchor writes `report.json` to that folder.++The fitted settings are part of each predictor's `fields`, so the tuned program+saves and loads like any other `Predict` program.++::: dspy.experimental.ReAnchor+ options:+ members: [__init__, compile]diff --git a/docs/docs/api/experimental/index.md b/docs/docs/api/experimental/index.mdindex d152c6625..1c6a2f318 100644--- a/docs/docs/api/experimental/index.md+++ b/docs/docs/api/experimental/index.md@@ -6,4 +6,5 @@ API reference for DSPy experimental. Select a page below.- [Citations](Citations.md)- [Decision types and System One models](DecisionTypes.md)- [Document](Document.md)+- [ReAnchor](ReAnchor.md)<!-- END_API_INDEX -->diff --git a/docs/mkdocs.yml b/docs/mkdocs.ymlindex 70e047012..b57e76dfa 100644--- a/docs/mkdocs.yml+++ b/docs/mkdocs.yml@@ -113,6 +113,7 @@ nav:- Citations: api/experimental/Citations.md- Decision types and System One models: api/experimental/DecisionTypes.md- Document: api/experimental/Document.md+ - ReAnchor: api/experimental/ReAnchor.md- Models:- api/models/index.md- BaseLM: api/models/BaseLM.mddiff --git a/dspy/adapters/decision.py b/dspy/adapters/decision.pyindex 5bafa93ba..b8dc80526 100644--- a/dspy/adapters/decision.py+++ b/dspy/adapters/decision.py@@ -2,13 +2,28 @@import copyimport json+import threadingimport warnings+from contextlib import contextmanagerfrom functools import lru_cachefrom pydantic import BaseModel, ConfigDict, Field, create_modelfrom dspy.adapters.decision_state import DecisionStatefrom dspy.adapters.types.decision import Choice, Noul, Probability, Score+from dspy.dsp.utils.settings import settings+++@contextmanager+def record_evidence():+ """Collect decision evidence in this execution context and its DSPy workers.++ Yields a list that receives one ``(predictor, output name, evidence)`` entry per decoded+ output. ``predictor`` is the innermost calling module. Nested collectors are independent.+ """+ log = []+ with settings.context(_decision_evidence=(log, threading.Lock())):+ yield log@lru_cache(maxsize=256)@@ -60,7 +75,6 @@ class DecisionAdapter:def _prepare(self, signature, demos, inputs, lm_kwargs):from dspy.adapters.utils import get_field_description_string- from dspy.dsp.utils.settings import settingsfrom dspy.predict.predict import serialize_objectif settings.send_stream is not None:@@ -112,6 +126,12 @@ class DecisionAdapter:raise ValueError(f"Invalid Score distribution for {name!r}.")answer["probabilities"] = {int(k): v for k, v in answer["probabilities"].items()}answers[name] = answer+ recorder = settings.get("_decision_evidence")+ if recorder is not None:+ log, lock = recorder+ caller = (settings.caller_modules or [None])[-1]+ with lock:+ log.extend((caller, name, copy.deepcopy(answer)) for name, answer in answers.items())results.append({**completion, **self.state._decode(answers)})return resultsdiff --git a/dspy/experimental/__init__.py b/dspy/experimental/__init__.pyindex 76481e2d5..e99c24437 100644--- a/dspy/experimental/__init__.py+++ b/dspy/experimental/__init__.py@@ -2,6 +2,7 @@ from dspy.adapters.types.citation import Citationsfrom dspy.adapters.types.decision import Choice, Noul, Scorefrom dspy.adapters.types.document import Documentfrom dspy.clients.typesafe import TypeSafe+from dspy.teleprompt.reanchor import ReAnchor__all__ = ["Citations",@@ -10,4 +11,5 @@ __all__ = ["Noul","Score","TypeSafe",+ "ReAnchor",]diff --git a/dspy/teleprompt/reanchor/__init__.py b/dspy/teleprompt/reanchor/__init__.pynew file mode 100644index 000000000..34598adf4--- /dev/null+++ b/dspy/teleprompt/reanchor/__init__.py@@ -0,0 +1,3 @@+from dspy.teleprompt.reanchor.reanchor import ReAnchor++__all__ = ["ReAnchor"]diff --git a/dspy/teleprompt/reanchor/calibrate.py b/dspy/teleprompt/reanchor/calibrate.pynew file mode 100644index 000000000..c27e934a1--- /dev/null+++ b/dspy/teleprompt/reanchor/calibrate.py@@ -0,0 +1,354 @@+"""Fit the numeric decision parameters of every Predict in a program against the metric.++Each `Predict` holds per-output parameters in `fields` that reinterpret the backend's+probabilities without adding request parameters. Each candidate reruns the program over the+training set, reusing cached answers for identical requests. Changed upstream decisions can+produce new downstream requests. The search runs directly against the metric, whatever shape+the program's outputs have.++Three kinds of parameter are fitted, all in the `Predict`'s `fields[field]` configuration:+- `threshold`, a Boolean output's cut point on P(True).+- `cuts`, a Score's boundaries on the mean level index. They pick the returned `.level` and leave+ `.value` alone.+- `weights`, a Choice's multiplier per option label, applied to the probabilities before the+ option is picked. An option picked too often gets a multiplier below 1.++The candidate settings come from the training set. One pass records the probabilities the output+is decoded from on every call. A setting anywhere between two neighbouring observed values makes+the same decisions, so the candidates are the midpoints of those gaps: between P(True) values for+a threshold, between mean level indexes for a cut, and between the points where an option's pick+flips for a weight. Among equal scores the candidate in the widest gap wins.++A setting stays unless a candidate scores strictly better and the gain holds across folds. The+fold check splits the training set into FOLDS parts. For each part, it picks a setting on the+other parts and scores that pick on the held-out part. A candidate replaces the current setting+only when those held-out scores beat the current setting's. This discourages gains confined to+small portions of the dataset, but can accept them when they recur across folds.++Calibration enables probability-based execution for every compatible output by adding an entry+in `fields`. After fitting, it compares against the original behavior under the same fold check+and restores the original field configuration unless the fitted behavior wins. The adapter+handles probability requests for the backend used on each call.++"""++import copy+import itertools+import math+import random+from typing import Any, Callable++import dspy+from dspy.adapters.decision import record_evidence+from dspy.adapters.decision_state import DecisionState+from dspy.adapters.types.decision import Choice, Noul, decision_type+from dspy.predict.predict import Predict+from dspy.utils.parallelizer import ParallelExecutor++MAX_CANDIDATES = 40 # candidate settings per parameter; more distinct values are thinned to quantiles+WEIGHT_RANGE = (1e-3, 1e3) # the smallest and largest Choice multiplier tried+FOLDS = 5 # training-set parts the fold check holds out in turn+++def decision_outputs(predict: Predict) -> dict[str, type]:+ """The outputs a decision parameter can apply to, with their decision types."""+ kinds = {name: decision_type(field) for name, field in predict.signature.output_fields.items()}+ return {name: kind for name, kind in kinds.items() if kind is not None}+++def predictors(program) -> list[tuple[str, Predict]]:+ """Every Predict with a decision output, by the name `named_parameters` gives it. A bare Predict is `self`."""+ return [(name, p) for name, p in program.named_parameters() if isinstance(p, Predict) and decision_outputs(p)]+++def caches(predict: Predict) -> bool:+ """Check caching for a statically bound or configured client."""+ lm = predict.lm or dspy.settings.lm+ if lm is None:+ raise ValueError(+ "ReAnchor cannot check caching without a bound or globally configured client. "+ "For runtime client selection, pass require_cache=False."+ )+ enabled = predict.config.get("cache", getattr(lm, "cache", False))+ return bool(enabled) and getattr(lm, "_cache_responses", True)+++def effective(predict: Predict, field: str) -> dict:+ """The output's full configuration: its stored overrides on top of its type's defaults."""+ fields = dict(predict.fields)+ fields.setdefault(field, {})+ return DecisionState(predict.signature, fields).fields[field]+++def metric_value(metric: Callable, example, pred) -> float:+ """The metric's score as a float, from a number or from a prediction carrying `score`."""+ verdict = metric(example, pred)+ return verdict.score if hasattr(verdict, "score") else float(verdict)+++def scores(+ program, examples: list, metric: Callable, num_threads: int | None = None, progress: bool = False+) -> list[float]:+ """The program's metric score on each example. Any program or metric error fails the pass."""++ def one(example):+ with dspy.context(trace=[]):+ pred = program(**example.inputs())+ return metric_value(metric, example, pred)++ executor = ParallelExecutor(num_threads=num_threads, max_errors=1, disable_progress_bar=not progress)+ values = executor.execute(one, examples)+ if not all(math.isfinite(s) for s in values):+ raise ValueError("ReAnchor requires finite metric values.")+ return values+++def run(program, examples: list, metric: Callable, num_threads: int | None = None, progress: bool = False) -> float:+ """The program's mean metric score over `examples`."""+ values = scores(program, examples, metric, num_threads, progress)+ return math.fsum(values) / len(values)+++def calibrate(+ program,+ trainset: list,+ metric: Callable,+ num_threads: int | None = None,+) -> list[dict[str, Any]]:+ """Fit every threshold, cut, and weight in place.++ Returns one report row per output: the fitted value, or why it was skipped.+ """+ report = []++ def score() -> list[float]:+ return scores(program, trainset, metric, num_threads)++ for name, predict in predictors(program):+ for field, kind in decision_outputs(predict).items():+ original = copy.deepcopy(predict.fields.get(field))+ before = score()+ predict.fields[field] = effective(predict, field)+ evidence = _observe(program, predict, field, trainset, metric, num_threads)+ row = {"predictor": name, "field": field}+ parameter, fit = (+ ("threshold", _fit_threshold)+ if issubclass(kind, Noul)+ else ("weights", _fit_weights)+ if issubclass(kind, Choice)+ else ("cuts", _fit_cuts)+ )+ base = score()+ best, fitted, observed, check = fit(predict, field, kind, evidence, score, base)+ row.update(parameter=parameter, value=best, train_score=_mean(fitted), train_score_at_start=_mean(base))+ row.update(observed=observed, fold_check=check, train_score_original=_mean(before))+ if _select(before, [(fitted, (), None)])[0] is None:+ _restore(predict, field, original)+ row.pop("value")+ row.update(skipped="fitted behavior did not beat the original", train_score=_mean(before))+ report.append(row)+ return report+++def _restore(predict: Predict, field: str, original: dict | None) -> None:+ """Put back the output's entry in `fields` as it was before calibration, or remove it."""+ if original is None:+ del predict.fields[field]+ else:+ predict.fields[field] = original+++def _observe(program, predict: Predict, field: str, trainset: list, metric: Callable, num_threads) -> list[dict]:+ """The evidence the output is decoded from on each training call, from one pass at the current setting."""+ with record_evidence() as log:+ scores(program, trainset, metric, num_threads)+ return [evidence for caller, name, evidence in log if caller is predict and name == field]+++def _tidy(x: float, lo: float, hi: float) -> float:+ """The shortest decimal strictly between `lo` and `hi`, rounded from `x`."""+ for digits in range(1, 12):+ rounded = round(x, digits)+ if lo < rounded < hi:+ return rounded+ return x+++def _gaps(values: list[float], lo: float, hi: float, geometric: bool = False) -> list[tuple[float, float]]:+ """Candidate settings and the width of the gap each sits in.++ The candidates are the midpoints between neighbouring distinct values, with `lo` and `hi`+ closing the range. A setting anywhere inside one gap makes the same decisions as its midpoint,+ so these cover every outcome. More than MAX_CANDIDATES gaps are thinned to evenly spaced+ quantiles of the values. `geometric` takes midpoints and widths on a log scale.+ """+ inner = sorted({v for v in values if lo < v < hi})+ if len(inner) > MAX_CANDIDATES - 1:+ step = (len(inner) - 1) / (MAX_CANDIDATES - 2)+ inner = sorted({inner[round(i * step)] for i in range(MAX_CANDIDATES - 1)})+ points = [lo, *inner, hi]+ gaps = []+ for a, b in itertools.pairwise(points):+ middle, width = (math.sqrt(a * b), math.log(b / a)) if geometric else ((a + b) / 2, b - a)+ if a < middle < b: # Adjacent floats can round the midpoint onto a boundary.+ gaps.append((_tidy(middle, a, b), width))+ return gaps+++def _summary(values: list[float]) -> dict[str, Any]:+ distinct = sorted(set(values))+ if not distinct:+ return {"calls": 0}+ return {+ "calls": len(values),+ "distinct": len(distinct),+ "min": round(distinct[0], 4),+ "max": round(distinct[-1], 4),+ }+++def _mean(values: list[float]) -> float:+ return round(math.fsum(values) / len(values), 4)+++def _folds(n: int) -> list[list[int]]:+ """Example indexes split into up to FOLDS parts, the same way on every run."""+ order = list(range(n))+ random.Random(0).shuffle(order)+ k = min(FOLDS, n)+ return [order[i::k] for i in range(k)]+++def _pick(start: list[float], tried: list[tuple], rows) -> tuple | None:+ """The best entry of `tried` on `rows`, or None when none scores strictly better than `start` there.++ Each entry is (per-example scores, tie-break key, setting). Equal totals go to the larger key.+ """++ def total(values):+ return math.fsum(values[i] for i in rows)++ best = max(tried, key=lambda t: (total(t[0]), t[1]), default=None)+ return best if best is not None and total(best[0]) > total(start) else None+++def _select(start: list[float], tried: list[tuple]) -> tuple[tuple | None, bool]:+ """The entry of `tried` to keep, or None to keep the start, and whether the fold check refused a better entry.++ The entry that scores best on the whole training set is kept only when picking on all folds but+ one, and scoring the pick on the held-out fold, beats the start on those same examples.+ """+ best = _pick(start, tried, range(len(start)))+ if best is None:+ return None, False+ held, held_start = 0.0, 0.0+ for fold in _folds(len(start)):+ out = set(fold)+ pick = _pick(start, tried, [i for i in range(len(start)) if i not in out])+ held += math.fsum((pick[0] if pick else start)[i] for i in fold)+ held_start += math.fsum(start[i] for i in fold)+ return (best, False) if held > held_start else (None, True)+++def _fit_threshold(+ predict: Predict, field: str, kind: type, evidence: list[dict], score: Callable, base: list[float]+) -> tuple[float, list[float], dict, dict]:+ """The threshold that scores best, between the observed probabilities.++ The starting threshold stays unless a candidate scores strictly better and passes the fold+ check. Among equal scores, the candidate in the widest gap wins, since it leaves the most room+ on either side.+ """+ config = predict.fields[field]+ start = config["threshold"]+ probabilities = [e["noul"] for e in evidence]+ tried = []+ candidates = _gaps(probabilities, 0.0, 1.0)+ for lower, upper in itertools.pairwise(sorted(set(probabilities))):+ if math.nextafter(lower, upper) == upper:+ # No interior float exists, but p >= upper still separates the pair.+ candidates.append((upper, upper - lower))+ if 0.0 in probabilities:+ candidates.append((0.0, 0.0)) # P(True) >= threshold makes zero a distinct outcome.+ for t, width in candidates:+ config["threshold"] = t+ tried.append((score(), (width, -abs(t - start), t), t))+ kept, refused = _select(base, tried)+ best, values = (kept[2], kept[0]) if kept else (start, base)+ config["threshold"] = best+ observed = {**_summary(probabilities), "candidates": len(tried)}+ return best, values, observed, {"passed": int(kept is not None), "failed": int(refused)}+++def _fit_weights(+ predict: Predict, field: str, kind: type, evidence: list[dict], score: Callable, base: list[float]+) -> tuple[dict, list[float], dict, dict]:+ """The Choice multipliers that score best on the training set.++ Each option's multiplier moves in turn while the others hold. An option's pick on a call flips+ where its weighted probability meets the strongest rival's, so the candidates sit between those+ flip points. A candidate replaces the current multipliers only when it scores strictly better+ and passes the fold check.+ """+ labels = list(kind.criteria())+ config = predict.fields[field]+ best = {label: config["weights"].get(label, 1.0) for label in labels}+ best_scores, candidates, check = base, 0, {"passed": 0, "failed": 0}+ for label in labels:+ flips = []+ for e in evidence:+ p = e["probabilities"]+ rival = max(best[other] * p[other] for other in labels if other != label)+ if p[label] > 0 and rival > 0:+ flips.append(rival / p[label])+ low, high = WEIGHT_RANGE+ if flips:+ low, high = max(low, min(flips) / 4), min(high, max(flips) * 4)+ tried = []+ for w, width in _gaps(flips, low, high, geometric=True):+ if w == best[label]:+ continue+ candidate = {**best, label: w}+ config["weights"] = candidate+ tried.append((score(), width, candidate))+ candidates += len(tried)+ kept, refused = _select(best_scores, tried)+ check["passed"] += kept is not None+ check["failed"] += refused+ if kept:+ best_scores, _, best = kept+ config["weights"] = best+ return best, best_scores, {"calls": len(evidence), "candidates": candidates}, check+++def _fit_cuts(+ predict: Predict, field: str, kind: type, evidence: list[dict], score: Callable, base: list[float]+) -> tuple[list[float], list[float], dict, dict]:+ """The Score cuts that score best on the training set.++ A call's level depends only on its mean level index. Each cut moves in turn between the+ observed means that lie between its neighbours, and stays strictly between them. A candidate+ replaces the current cuts only when it scores strictly better and passes the fold check.+ """+ config = predict.fields[field]+ top = len(kind.criteria()) - 1+ means = [sum(i * p for i, p in e["probabilities"].items()) / sum(e["probabilities"].values()) for e in evidence]+ best, best_scores, candidates, check = list(config["cuts"]), base, 0, {"passed": 0, "failed": 0}+ for i in range(len(best)):+ below = best[i - 1] if i else 0+ above = best[i + 1] if i + 1 < len(best) else top+ tried = []+ for c, width in _gaps(means, below, above):+ if c == best[i]:+ continue+ candidate = [*best[:i], c, *best[i + 1 :]]+ config["cuts"] = candidate+ tried.append((score(), width, candidate))+ candidates += len(tried)+ kept, refused = _select(best_scores, tried)+ check["passed"] += kept is not None+ check["failed"] += refused+ if kept:+ best_scores, _, best = kept+ config["cuts"] = best+ return best, best_scores, {**_summary(means), "candidates": candidates}, checkdiff --git a/dspy/teleprompt/reanchor/reanchor.py b/dspy/teleprompt/reanchor/reanchor.pynew file mode 100644index 000000000..bb51e6b7e--- /dev/null+++ b/dspy/teleprompt/reanchor/reanchor.py@@ -0,0 +1,87 @@+"""Fit a program's decision thresholds, cuts, and weights against the metric."""++import json+import logging+from pathlib import Path+from typing import Any++from dspy.teleprompt.reanchor.calibrate import caches, calibrate, predictors, run+from dspy.teleprompt.teleprompt import Teleprompter+from dspy.utils.annotation import experimental++logger = logging.getLogger(__name__)+++@experimental+class ReAnchor(Teleprompter):+ """Calibrate a program's decisions: fit every threshold, Score cut, and Choice weight against the metric.++ The program can be a single `Predict` or any `dspy.Module` holding predictors with decision+ outputs, and the metric is the only supervision. Each parameter changes how `Predict` reads the+ backend's probabilities without adding request parameters. Identical requests reuse cached+ answers, but changed upstream decisions can produce new downstream requests.++ ReAnchor enables probability-based execution for every compatible output through the predictor's+ `fields`. It keeps each fitted configuration only when it beats the original behavior and passes+ a fold check; otherwise it restores the original configuration, including absent field entries.++ Args:+ metric: Per-example metric to maximize, as in `dspy.Evaluate`. It may return a number, or a+ `dspy.Prediction` with a `score`.+ num_threads: Evaluation concurrency, as in `dspy.Evaluate`.+ log_dir: When set, report.json is written here.+ require_cache: When True, check caching on statically bound or globally configured clients.+ Set it to False for runtime client selection or to allow uncached requests.++ After `compile`, `report` holds the fitted parameters and the metric's mean before and after+ calibration, on the training set and on the validation set when one is given.+ """++ def __init__(self, metric, *, num_threads=None, log_dir=None, require_cache=True):+ super().__init__()+ self.metric = metric+ self.num_threads = num_threads+ self.log_dir = Path(log_dir) if log_dir else None+ self.require_cache = require_cache+ self.report: dict[str, Any] = {}++ def compile(self, student, *, trainset, valset=None):+ """Return a calibrated copy; leave the student unchanged.++ Args:+ student: A `Predict`, or a module holding predictors with decision outputs.+ trainset: Examples the parameters are fitted on.+ valset: Examples scored before and after calibration for the report. Nothing is fitted on them.+ """+ if not trainset:+ raise ValueError("trainset must contain at least one example.")+ program = student.deepcopy()+ found = predictors(program)+ if not found:+ raise ValueError("The student must contain at least one Predict with a decision output.")+ uncached = [name for name, predict in found if not caches(predict)] if self.require_cache else []+ if self.require_cache and uncached:+ raise ValueError(+ f"Predictors {uncached} do not cache responses, so every candidate setting would send new requests. "+ "Enable the client's cache, or pass require_cache=False to calibrate anyway."+ )++ logger.info("answering %d training examples", len(trainset))+ before = self._score(program, trainset, progress=True)+ val_before = self._score(program, valset) if valset else None+ logger.info("fitting thresholds, cuts, and weights")+ fitted = calibrate(program, trainset, self.metric, num_threads=self.num_threads)+ self.report = {"train_score_before": before, "train_score": self._score(program, trainset), "fitted": fitted}+ logger.info("calibrated: train %s -> %s", before, self.report["train_score"])+ if valset:+ self.report.update(val_score_before=val_before, val_score=self._score(program, valset))+ logger.info("validation: %s -> %s", val_before, self.report["val_score"])++ if self.log_dir:+ self.log_dir.mkdir(parents=True, exist_ok=True)+ (self.log_dir / "report.json").write_text(json.dumps(self.report, indent=2, default=str), encoding="utf-8")+ program._compiled = True+ return program++ def _score(self, program, examples: list, progress: bool = False) -> float:+ return round(run(program, examples, self.metric, self.num_threads, progress), 4)diff --git a/tests/teleprompt/reanchor/conftest.py b/tests/teleprompt/reanchor/conftest.pynew file mode 100644index 000000000..c87be1ec2--- /dev/null+++ b/tests/teleprompt/reanchor/conftest.py@@ -0,0 +1,16 @@+import pytest++import dspy+from tests.teleprompt.reanchor.fakes import FakeClient+++def system_one():+ """Configures a FakeClient as the LM for one test."""++ def install(answer, cache=True):+ client = FakeClient(answer, cache=cache)+ dspy.configure(lm=client)+ return client++ return installdiff --git a/tests/teleprompt/reanchor/fakes.py b/tests/teleprompt/reanchor/fakes.pynew file mode 100644index 000000000..6f325cb5e--- /dev/null+++ b/tests/teleprompt/reanchor/fakes.py@@ -0,0 +1,68 @@+"""Stand-in decision clients and generative LMs for the ReAnchor tests."""++import copy+import re+import threading++from dspy.clients.engines.dummy_engine import AsyncDummyEngine, DummyEngine+from dspy.utils.dummies import DummyLM+++class FakeClient:+ """A System One client. Answers each question with `answer(state, name, question)`, and records every request."""++ supports_decision_requests = True++ def __init__(self, answer, cache=True):+ self.answer = answer+ self.cache = cache+ self.calls = []+ self.history = []+ self.callbacks = []++ def __call__(self, state, questions):+ self.calls.append(copy.deepcopy((state, questions)))+ return {name: self.answer(state, name, q) for name, q in questions.items()}++ async def acall(self, state, questions):+ return self(state, questions)+++class _ComputedEngine(DummyEngine):+ _lock = threading.Lock()++ def _complete_messages(self, messages):+ with self._lock:+ self.owner.answers = {"": self.owner.answer(messages)}+ return super()._complete_messages(messages)+++class ComputedLM(DummyLM):+ """A generative LM whose output field values are `answer(inputs, evidence)`.++ `inputs` maps each input field to its text in the final message, and `evidence` is True when+ the request asks for probabilities instead of native values.+ """++ def __init__(self, answer, adapter=None):+ super().__init__({}, adapter=adapter)+ self.answer_fn = answer+ self._engine_spec = _ComputedEngine(self)+ self._async_engine_spec = AsyncDummyEngine(self._engine_spec)++ def answer(self, messages):+ inputs = dict(re.findall(r"\[\[ ## (\w+) ## \]\]\n(.*)", messages[-1]["content"]))+ evidence = any("noul" in m["content"] or "probabilities" in m["content"] for m in messages)+ return self.answer_fn(inputs, evidence)+++def noul(p: float) -> dict:+ return {"noul": p}+++def score(probabilities: dict[int, float], confidence: float = 0.8) -> dict:+ return {"score": 0.0, "confidence": confidence, "probabilities": probabilities}+++def choice(probabilities: dict[str, float], confidence: float = 0.7) -> dict:+ return {"confidence": confidence, "probabilities": probabilities}diff --git a/tests/teleprompt/reanchor/test_reanchor.py b/tests/teleprompt/reanchor/test_reanchor.pynew file mode 100644index 000000000..e5e526fc1--- /dev/null+++ b/tests/teleprompt/reanchor/test_reanchor.py@@ -0,0 +1,291 @@+"""ReAnchor on a stubbed System One client and a stubbed generative LM."""++import copy+import json+import logging++import pytest++import dspy+from dspy.experimental import ReAnchor+from tests.teleprompt.reanchor.fakes import ComputedLM, noul+++def leaning(state, name, q):+ """Jev leans high, 0.9 on a same pair and 0.7 on a different one, and reads a tricky same pair+ as different, so no threshold gets every pair right. The best thresholds lie between 0.7 and 0.9; ReAnchor picks the midpoint, 0.8."""+ pair = state["inputs"]["pair"]+ if pair.endswith("same-tricky"):+ return noul(0.7)+ return noul(0.9 if "same" in pair else 0.7)+++class Sig(dspy.Signature):+ pair: str = dspy.InputField()+ match: bool = dspy.OutputField(desc="Are the two the same?")++[email protected](autouse=True)+def jev(system_one):+ return system_one(leaning)++[email protected](autouse=True)+def settings():+ # Configured rather than set with `dspy.context`, which would pin the client a test installs later.+ dspy.configure(adapter=dspy.JSONAdapter())+++def examples(prefix=""):+ kinds = ["same"] * 6 + ["same-tricky"] * 4 + ["different"] * 8+ return [dspy.Example(pair=prefix + k, match=k.startswith("same")).with_inputs("pair") for k in kinds]+++def metric(gold, pred, trace=None):+ return float(pred.match == gold.match)+++def test_compile_fits_the_threshold_and_reports_the_training_scores():+ optimizer = ReAnchor(metric, num_threads=2)+ program = optimizer.compile(dspy.Predict(Sig), trainset=examples())+ assert isinstance(program, dspy.Predict)+ assert program.fields["match"]["threshold"] == 0.8+ assert optimizer.report["train_score_before"] == 0.5556 and optimizer.report["train_score"] == 0.7778+ assert optimizer.report["fitted"][0]["parameter"] == "threshold"+ assert "val_score" not in optimizer.report+++def test_the_validation_set_is_scored_and_never_fitted_on():+ optimizer = ReAnchor(metric, num_threads=2)+ val = examples("v-")[:10] # same pairs only, where the default threshold is already right+ program = optimizer.compile(dspy.Predict(Sig), trainset=examples(), valset=val)+ assert program.fields["match"]["threshold"] == 0.8+ assert optimizer.report["val_score_before"] == 1.0 and optimizer.report["val_score"] == 0.6+++def test_compile_leaves_the_student_unchanged_and_marks_the_program_compiled():+ student = dspy.Predict(Sig)+ before = copy.deepcopy(student.fields)+ program = ReAnchor(metric, num_threads=2).compile(student, trainset=examples())+ assert student.fields == before+ assert program._compiled and program is not student+++def test_a_metric_returning_a_prediction_is_read_by_its_score():+ graded = lambda gold, pred, trace=None: dspy.Prediction(score=metric(gold, pred), feedback="") # noqa: E731+ program = ReAnchor(graded, num_threads=2).compile(dspy.Predict(Sig), trainset=examples())+ assert program.fields["match"]["threshold"] == 0.8+++def test_a_module_holding_predictors_gets_each_one_calibrated():+ class Wrapper(dspy.Module):+ def __init__(self):+ super().__init__()+ self.judge = dspy.Predict(Sig)+ self.judges = [dspy.Predict(Sig)]++ def forward(self, pair):+ return self.judges[0](pair=pair) if self.judge(pair=pair).match else dspy.Prediction(match=False)++ program = ReAnchor(metric, num_threads=2).compile(Wrapper(), trainset=examples())+ assert isinstance(program, Wrapper)+ assert program.judge.fields["match"]["threshold"] == 0.8+ assert program.judges[0].fields == {} # No further gain after calibrating the first judge.+++class Items(dspy.Module):+ def __init__(self):+ super().__init__()+ self.judge = dspy.Predict(Sig)++ def forward(self, pairs):+ return dspy.Prediction(matches=[self.judge(pair=p).match for p in pairs])+++def test_a_predictor_called_once_per_item_is_calibrated_from_the_program_metric():+ batch = ["same", "different", "different"]+ trainset = [dspy.Example(pairs=batch, matches=[k.startswith("same") for k in batch]).with_inputs("pairs")] * 4++ def items_metric(gold, pred, trace=None):+ return sum(a == b for a, b in zip(pred.matches, gold.matches, strict=True)) / len(gold.matches)++ program = ReAnchor(items_metric, num_threads=2).compile(Items(), trainset=trainset)+ assert isinstance(program, Items) and program.judge.fields["match"]["threshold"] == 0.8+++def test_the_program_keeps_a_bound_client_and_callbacks(system_one):+ client = system_one(leaning)+ dspy.configure(lm=None)+ student = dspy.Predict(Sig, callbacks=[])+ student.set_lm(client)+ program = ReAnchor(metric, num_threads=2).compile(student, trainset=examples())+ assert program.lm is not None and program(pair="same").match is True+++def test_log_dir_holds_the_report(tmp_path):+ ReAnchor(metric, num_threads=2, log_dir=tmp_path).compile(dspy.Predict(Sig), trainset=examples())+ assert json.loads((tmp_path / "report.json").read_text())["train_score"] == 0.7778+ assert {path.name for path in tmp_path.iterdir()} == {"report.json"}+++def test_compile_logs_each_stage():+ lines = []+ handler = logging.Handler()+ handler.emit = lambda record: lines.append(record.getMessage())+ log = logging.getLogger("dspy.teleprompt.reanchor.reanchor")+ log.addHandler(handler)+ try:+ ReAnchor(metric, num_threads=2).compile(dspy.Predict(Sig), trainset=examples(), valset=examples("v-"))+ finally:+ log.removeHandler(handler)+ assert lines == [+ "answering 18 training examples",+ "fitting thresholds, cuts, and weights",+ "calibrated: train 0.5556 -> 0.7778",+ "validation: 0.5556 -> 0.7778",+ ]+++def test_empty_trainset_fails():+ with pytest.raises(ValueError, match="trainset"):+ ReAnchor(metric).compile(dspy.Predict(Sig), trainset=[])++[email protected]("keyword", ["teacher", "unsupported"])+def test_unsupported_compile_arguments_are_not_silently_ignored(keyword):+ with pytest.raises(TypeError, match="unexpected keyword argument"):+ ReAnchor(metric).compile(dspy.Predict(Sig), trainset=examples(), **{keyword: None})+++def test_a_student_without_a_decision_output_fails():+ with pytest.raises(ValueError, match="decision output"):+ ReAnchor(metric).compile(dspy.Predict("pair -> answer: str"), trainset=examples())+++def test_an_uncached_client_fails_unless_the_cache_is_waived(system_one):+ system_one(leaning, cache=False)+ with pytest.raises(ValueError, match="require_cache=False"):+ ReAnchor(metric).compile(dspy.Predict(Sig), trainset=examples())+ program = ReAnchor(metric, num_threads=2, require_cache=False).compile(dspy.Predict(Sig), trainset=examples())+ assert program.fields["match"]["threshold"] == 0.8+++def test_a_predictor_config_that_turns_the_cache_off_fails(system_one):+ with pytest.raises(ValueError, match="require_cache=False"):+ ReAnchor(metric).compile(dspy.Predict(Sig, cache=False), trainset=examples())+++def generative(inputs, evidence):+ """The LM answers True on every pair when asked for a bool. Asked for probabilities, it leans+ the same way as the System One client."""+ if not evidence:+ return {"match": True}+ pair = inputs["pair"]+ return {"match": noul(0.7 if pair.endswith("same-tricky") else 0.9 if "same" in pair else 0.7)}+++def test_a_native_bool_on_a_generative_lm_is_promoted_when_probabilities_score_better():+ dspy.configure(lm=ComputedLM(generative, adapter=dspy.JSONAdapter()))+ student = dspy.Predict(Sig)+ assert student.fields == {}+ optimizer = ReAnchor(metric, num_threads=2, require_cache=False)+ program = optimizer.compile(student, trainset=examples())+ assert student.fields == {}+ assert program.fields == {"match": {"threshold": 0.8}}+ row = optimizer.report["fitted"][0]+ assert "skipped" not in row and row["train_score_original"] == 0.5556 and row["train_score"] == 0.7778++[email protected]("accurate_probabilities", [True, False])+def test_a_native_bool_stays_native_when_probabilities_do_not_score_better(accurate_probabilities):+ def right(inputs, evidence):+ answer = inputs["pair"].startswith("same")+ probability = (0.9 if answer else 0.1) if accurate_probabilities else 0.9+ return {"match": noul(probability)} if evidence else {"match": answer}++ dspy.configure(lm=ComputedLM(right, adapter=dspy.JSONAdapter()))+ optimizer = ReAnchor(metric, num_threads=2, require_cache=False)+ program = optimizer.compile(dspy.Predict(Sig), trainset=examples())+ assert program.fields == {}+ row = optimizer.report["fitted"][0]+ assert row["skipped"] == "fitted behavior did not beat the original"+ assert row["train_score"] == row["train_score_original"] == 1.0+ assert row["train_score_at_start"] == (1.0 if accurate_probabilities else 0.5556)+ assert "value" not in row+++def test_a_native_bool_stays_native_when_probabilities_win_only_one_example():+ pairs = [f"same{i}" for i in range(20)] + [f"different{i}" for i in range(20)]++ def one_miss(inputs, evidence):+ answer = inputs["pair"].startswith("same")+ if evidence:+ return {"match": noul(0.9 if answer else 0.1)}+ return {"match": answer and inputs["pair"] != "same0"}++ dspy.configure(lm=ComputedLM(one_miss, adapter=dspy.JSONAdapter()))+ train = [dspy.Example(pair=p, match=p.startswith("same")).with_inputs("pair") for p in pairs]+ optimizer = ReAnchor(metric, num_threads=2, require_cache=False)+ program = optimizer.compile(dspy.Predict(Sig), trainset=train)+ assert program.fields == {}+ assert optimizer.report["fitted"][0]["skipped"] == "fitted behavior did not beat the original"++[email protected]("binding", ["call", "context"])[email protected]("backend", ["jev", "lm"])+def test_runtime_clients_can_calibrate_without_static_cache_checks(jev, binding, backend):+ client = jev if backend == "jev" else ComputedLM(generative, adapter=dspy.JSONAdapter())++ class Routed(dspy.Module):+ def __init__(self):+ super().__init__()+ self.judge = dspy.Predict(Sig)++ def forward(self, pair):+ if binding == "call":+ return self.judge(pair=pair, lm=client)+ with dspy.context(lm=client):+ return self.judge(pair=pair)++ with dspy.context(lm=None):+ with pytest.raises(ValueError, match="require_cache=False"):+ ReAnchor(metric).compile(Routed(), trainset=examples())+ optimizer = ReAnchor(metric, num_threads=2, require_cache=False)+ program = optimizer.compile(Routed(), trainset=examples())+ assert program.judge.fields == {"match": {"threshold": 0.8}}+ assert program(pair="different").match is False+ assert optimizer.report["train_score"] == 0.7778++[email protected]("original", [None, {}, {"criteria": {"true": {"what": "same item"}}}])+def test_rejected_jev_configuration_is_restored_exactly(original):+ student = dspy.Predict(Sig)+ if original is not None:+ student.fields["match"] = copy.deepcopy(original)+ # All True is already perfect; fitting must not leave materialized defaults behind.+ train = [dspy.Example(pair="same", match=True).with_inputs("pair") for _ in range(10)]+ optimizer = ReAnchor(metric, num_threads=2)+ program = optimizer.compile(student, trainset=train)+ assert program.fields == student.fields+ row = optimizer.report["fitted"][0]+ assert row["skipped"] == "fitted behavior did not beat the original"+ assert row["train_score_original"] == row["train_score"] == 1.0+ assert "value" not in row+++def test_a_partial_field_entry_is_filled_from_the_type_defaults(system_one):+ student = dspy.Predict(Sig)+ student.set_criteria("match", {"true": "The same item.", "false": "Two different items."})+ program = ReAnchor(metric, num_threads=2).compile(student, trainset=examples())+ assert program.fields["match"] == {+ "criteria": {"true": "The same item.", "false": "Two different items."},+ "threshold": 0.8,+ }+++def test_the_calibrated_program_saves_and_loads(tmp_path):+ program = ReAnchor(metric, num_threads=2).compile(dspy.Predict(Sig), trainset=examples())+ program.save(tmp_path / "program.json")+ restored = dspy.Predict(Sig)+ restored.load(tmp_path / "program.json")+ assert restored.fields == {"match": {"threshold": 0.8}}diff --git a/tests/teleprompt/reanchor/test_reanchor_calibrate.py b/tests/teleprompt/reanchor/test_reanchor_calibrate.pynew file mode 100644index 000000000..85149b184--- /dev/null+++ b/tests/teleprompt/reanchor/test_reanchor_calibrate.py@@ -0,0 +1,399 @@+"""Calibration fits each predictor's thresholds, cuts, and weights against the metric, in place."""++import math+from typing import Annotated, Literal++import pytest++import dspy+from dspy.experimental import Choice, Score+from dspy.teleprompt.reanchor.calibrate import calibrate, predictors+from tests.teleprompt.reanchor.fakes import choice, noul, score++Rating = Score["low", "mid", "high"]+++class Match(dspy.Signature):+ pair: str = dspy.InputField()+ match: bool = dspy.OutputField(desc="Are the two the same?")+++def test_a_leaning_noul_gets_a_threshold_between_its_piles(system_one):+ system_one(lambda state, name, q: noul(0.9 if state["inputs"]["pair"] == "same" else 0.7))+ program = dspy.Predict(Match)+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 4]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)+ assert program.fields["match"]["threshold"] == 0.8+ assert report[0]["train_score"] == 1.0++[email protected]("lower", [0.0, 0.7, math.nextafter(0.7, 1.0), math.nextafter(1.0, 0.0)])+def test_adjacent_probabilities_can_be_separated_by_a_boolean_threshold(system_one, lower):+ upper = math.nextafter(lower, 1.0)+ system_one(lambda state, name, q: noul(upper if state["inputs"]["pair"] == "same" else lower))+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 10]+ optimizer = dspy.experimental.ReAnchor(lambda g, p: float(p.match == g.match), num_threads=2)+ program = optimizer.compile(dspy.Predict(Match), trainset=train)+ assert program.fields["match"]["threshold"] == upper+ assert program(pair="same").match is True+ assert program(pair="different").match is False+ assert optimizer.report["train_score_before"] == 0.5+ assert optimizer.report["train_score"] == 1.0++[email protected]("target", [True, False])+def test_zero_probability_can_be_classified_on_either_side(system_one, target):+ system_one(lambda state, name, q: noul(0.0))+ program = dspy.Predict(Match)+ train = [dspy.Example(pair=str(i), match=target).with_inputs("pair") for i in range(10)]+ report = calibrate(program, train, lambda g, p: float(p.match == g.match), num_threads=2)+ assert program.fields == ({"match": {"threshold": 0.0}} if target else {})+ assert report[0]["train_score"] == 1.0+++def test_a_starting_threshold_stays_unless_a_candidate_scores_strictly_better(system_one):+ # 0.73 already splits 0.74 from 0.72, so the candidate in that gap only ties.+ system_one(lambda state, name, q: noul(0.74 if state["inputs"]["pair"] == "same" else 0.72))+ program = dspy.Predict(Match)+ program.fields["match"] = {"threshold": 0.73}+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 4]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)+ assert program.fields["match"]["threshold"] == 0.73+ assert report[0]["train_score_at_start"] == 1.0 and report[0]["train_score"] == 1.0+++class Kind(dspy.Signature):+ item: str = dspy.InputField()+ kind: Literal["x", "y"] = dspy.OutputField(desc="Which kind?")+++def test_choice_multipliers_move_an_overpicked_option(system_one):+ # Jev leans to "x": 0.6 on an "x" item and 0.55 on a "y" item.+ system_one(+ lambda state, name, q: choice(+ {"x": 0.6, "y": 0.4} if state["inputs"]["item"] == "x" else {"x": 0.55, "y": 0.45}+ )+ )+ program = dspy.Predict(Kind)+ train = [dspy.Example(item=k, kind=k).with_inputs("item") for k in ["x", "y"] * 4]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.kind == g.kind), num_threads=2)+ assert report[0]["train_score"] == 1.0+ w = program.fields["kind"]["weights"]+ assert 0.55 * w["x"] < 0.45 * w["y"] and 0.6 * w["x"] > 0.4 * w["y"]+++# Jev leans high: a low item still puts half its mass on the top level (mean index 1.3).+LEANING = {"lo": {0: 0.2, 1: 0.3, 2: 0.5}, "hi": {0: 0.0, 1: 0.2, 2: 0.8}}+++class RateLevel(dspy.Signature):+ item: str = dspy.InputField()+ rating: Rating = dspy.OutputField(desc="How good is it?")+++def test_score_cuts_move_to_fit_a_metric_on_the_level(system_one):+ system_one(lambda state, name, q: score(LEANING[state["inputs"]["item"]]))+ program = dspy.Predict(RateLevel)+ train = [dspy.Example(item=k, rating={"lo": 0, "hi": 2}[k]).with_inputs("item") for k in ["lo", "hi"] * 3]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.rating.level == g.rating), num_threads=2)+ cuts = program.fields["rating"]["cuts"]+ assert report[0]["parameter"] == "cuts"+ assert report[0]["train_score_at_start"] == 0.5 and report[0]["train_score"] == 1.0+ assert 1.3 < cuts[0] < cuts[1] <= 1.8+++def test_score_cuts_stay_at_their_defaults_when_the_metric_ignores_the_level(system_one):+ system_one(lambda state, name, q: score(LEANING[state["inputs"]["item"]]))+ program = dspy.Predict(RateLevel)+ train = [dspy.Example(item=k, rating={"lo": 0, "hi": 2}[k]).with_inputs("item") for k in ["lo", "hi"]]+ report = calibrate(program, train, lambda g, p, trace=None: 1 - abs(p.rating.value - g.rating) / 2, num_threads=2)+ assert program.fields == {}+ assert report[0]["train_score"] == report[0]["train_score_at_start"]+++def test_score_search_does_not_collapse_cuts_at_adjacent_floats(system_one):+ # The midpoint between 0.7 and this observed mean rounds back to 0.7.+ probability = 0.7000000000000001+ system_one(lambda state, name, q: score({0: 1 - probability, 1: probability, 2: 0.0}))+ program = dspy.Predict(RateLevel)+ program.fields["rating"] = {"cuts": [0.7, 1.5]}+ train = [dspy.Example(item=str(i), rating=1).with_inputs("item") for i in range(6)]+ report = calibrate(program, train, lambda g, p: float(p.rating.level == g.rating), num_threads=2)+ assert program.fields["rating"] == {"cuts": [0.7, 1.5]}+ assert report[0]["train_score"] == 1.0+++class Route(dspy.Signature):+ ticket: str = dspy.InputField()+ queue: Annotated[Literal["billing", "tech"], Choice[("billing", "Payments"), ("tech", "Bugs")]] = dspy.OutputField(+ desc="Which queue?"+ )+++def test_a_native_literal_with_choice_criteria_gets_weights_and_stays_native(system_one):+ system_one(lambda state, name, q: choice({"billing": 0.6, "tech": 0.4}))+ program = dspy.Predict(Route)+ train = [dspy.Example(ticket=str(i), queue="tech" if i % 3 else "billing").with_inputs("ticket") for i in range(6)]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.queue == g.queue), num_threads=2)+ assert report[0]["parameter"] == "weights" and report[0]["train_score"] == round(4 / 6, 4)+ assert program(ticket="0").queue == "tech"+++def test_predictors_finds_nested_and_listed_predictors():+ class Wrapper(dspy.Module):+ def __init__(self):+ super().__init__()+ self.judge = dspy.Predict(Match)+ self.judges = [dspy.Predict(Kind)]+ self.writer = dspy.Predict("pair -> note")++ assert [name for name, _ in predictors(Wrapper())] == ["judge", "judges[0]"]+ assert [name for name, _ in predictors(dspy.Predict(Match))] == ["self"]++[email protected]("threads", [1, 4])+def test_calibration_leaves_the_request_unchanged(system_one, threads):+ client = system_one(lambda state, name, q: noul(0.9 if state["inputs"]["pair"] == "same" else 0.7))+ program = dspy.Predict(Match)+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"]]+ calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=threads)+ assert (+ len({repr(c) for c in client.calls}) == 2+ ) # one distinct request per example, however many settings were tried+++def test_a_metric_error_fails_the_pass(system_one):+ system_one(lambda state, name, q: noul(0.9))++ def broken(example, prediction):+ raise RuntimeError("metric failed")++ train = [dspy.Example(pair="same", match=True).with_inputs("pair")]+ with pytest.raises(Exception, match="Execution cancelled"):+ calibrate(dspy.Predict(Match), train, broken, num_threads=1)++[email protected]("value", [float("nan"), float("inf")])+def test_a_nonfinite_metric_fails(system_one, value):+ system_one(lambda state, name, q: noul(0.9))+ train = [dspy.Example(pair="same", match=True).with_inputs("pair")]+ with pytest.raises(ValueError, match="finite metric"):+ calibrate(dspy.Predict(Match), train, lambda g, p, trace=None: value, num_threads=1)+++class TwoFlags(dspy.Signature):+ pair: str = dspy.InputField()+ match: bool = dspy.OutputField(desc="Are the two the same?")+ noise: bool = dspy.OutputField(desc="Is it noisy?")+ kind: Literal["x", "y"] = dspy.OutputField(desc="Which kind?")+++def two_flags(state, name, q):+ if name == "kind":+ return choice({"x": 0.6, "y": 0.4})+ return noul(0.9 if state["inputs"]["pair"] == "same" else 0.7)+++def test_outputs_the_metric_does_not_read_keep_their_numeric_settings(system_one):+ client = system_one(two_flags)+ program = dspy.Predict(TwoFlags)+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 4]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)+ rows = {r["field"]: r for r in report}+ assert rows["noise"]["train_score"] == rows["noise"]["train_score_at_start"]+ assert rows["kind"]["train_score"] == rows["kind"]["train_score_at_start"]+ assert "skipped" not in rows["match"] and program.fields["match"]["threshold"] == 0.8+ assert "noise" not in program.fields+ assert "kind" not in program.fields+ assert len({repr(c) for c in client.calls}) == 2+++def test_balanced_labels_are_not_mistaken_for_an_ignored_output(system_one):+ # All True and all False both score 0.5 on average; the per-example scores still differ.+ system_one(lambda state, name, q: noul(0.9 if state["inputs"]["pair"] == "same" else 0.7))+ program = dspy.Predict(Match)+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 2]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)+ assert "skipped" not in report[0] and program.fields["match"]["threshold"] == 0.8+++def test_equal_scores_at_extremes_do_not_hide_a_useful_threshold(system_one):+ system_one(lambda state, name, q: noul(0.4 if state["inputs"]["pair"] == "same" else 0.2))++ class Batch(dspy.Module):+ def __init__(self):+ super().__init__()+ self.judge = dspy.Predict(Match)++ def forward(self, pairs):+ return dspy.Prediction(matches=[self.judge(pair=p).match for p in pairs])++ program = Batch()+ train = [dspy.Example(pairs=["same", "different"], matches=[True, False]).with_inputs("pairs") for _ in range(10)]+ # Default, all-True, and all-False decisions all fail exact match. Only the interior split wins.+ report = calibrate(program, train, lambda g, p, trace=None: float(p.matches == g.matches), num_threads=2)+ assert program.judge.fields["match"]["threshold"] == pytest.approx(0.3)+ assert report[0]["train_score_at_start"] == 0.0+ assert report[0]["train_score"] == 1.0+++def test_probabilities_bunched_near_one_get_a_threshold_between_them(system_one):+ # Every P(True) is 0.98 or 1.0, above any fixed grid; the gap between them splits the labels.+ system_one(lambda state, name, q: noul(1.0 if state["inputs"]["pair"] == "same" else 0.98))+ program = dspy.Predict(Match)+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 4]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)+ assert program.fields["match"]["threshold"] == 0.99+ assert report[0]["train_score"] == 1.0+ assert report[0]["observed"] == {"calls": 8, "distinct": 2, "min": 0.98, "max": 1.0, "candidates": 2}+++def test_a_choice_multiplier_can_leave_the_range_a_fixed_grid_would_try(system_one):+ # A "y" item still puts 0.95 on "x"; separating the items needs a "y" multiplier above 19.+ system_one(+ lambda state, name, q: choice(+ {"x": 0.99, "y": 0.01} if state["inputs"]["item"] == "x" else {"x": 0.95, "y": 0.05}+ )+ )+ program = dspy.Predict(Kind)+ train = [dspy.Example(item=k, kind=k).with_inputs("item") for k in ["x", "y"] * 4]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.kind == g.kind), num_threads=2)+ w = program.fields["kind"]["weights"]+ assert report[0]["train_score"] == 1.0+ assert 0.95 * w["x"] < 0.05 * w["y"] and 0.99 * w["x"] > 0.01 * w["y"]+++def test_many_distinct_probabilities_are_thinned_to_the_candidate_cap(system_one):+ from dspy.teleprompt.reanchor.calibrate import MAX_CANDIDATES++ system_one(lambda state, name, q: noul(int(state["inputs"]["pair"]) / 200))+ program = dspy.Predict(Match)+ train = [dspy.Example(pair=str(i), match=i >= 120).with_inputs("pair") for i in range(200)]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=4)+ assert report[0]["observed"]["distinct"] == 200+ assert report[0]["observed"]["candidates"] <= MAX_CANDIDATES + 1 # Gap midpoints plus the zero endpoint.+ assert 0.55 < program.fields["match"]["threshold"] <= 0.6 and report[0]["train_score"] >= 0.97+++def test_evidence_is_recorded_for_the_predictor_that_decoded_it(system_one):+ from dspy.adapters.decision import record_evidence++ system_one(lambda state, name, q: noul(0.9) if name == "match" else choice({"x": 0.6, "y": 0.4}))++ class Two(dspy.Module):+ def __init__(self):+ super().__init__()+ self.judge = dspy.Predict(Match)+ self.kind = dspy.Predict(Kind)++ def forward(self, items):+ return [self.judge(pair=i).match for i in items], self.kind(item="a").kind++ program = Two()+ with record_evidence() as log:+ program(items=["a", "b", "c"])+ program(items=["d"])+ assert [(caller is program.judge, name) for caller, name, _ in log].count((True, "match")) == 3+ caller, name, evidence = log[-1]+ assert caller is program.kind and name == "kind" and evidence["probabilities"] == {"x": 0.6, "y": 0.4}+ assert len(log) == 4+++def test_concurrent_evidence_collectors_are_isolated_and_reach_dspy_workers(system_one):+ from concurrent.futures import ThreadPoolExecutor+ from threading import Barrier++ from dspy.adapters.decision import record_evidence+ from dspy.utils.parallelizer import ParallelExecutor++ system_one(lambda state, name, q: noul(0.2 if state["inputs"]["pair"] == "a" else 0.8))+ barrier = Barrier(2)++ def collect(pair):+ predict = dspy.Predict(Match)+ with record_evidence() as log:+ barrier.wait(timeout=10)+ ParallelExecutor(num_threads=2, disable_progress_bar=True).execute(lambda _: predict(pair=pair), range(2))+ barrier.wait(timeout=10)+ return predict, log++ with ThreadPoolExecutor(max_workers=2) as executor:+ results = list(executor.map(collect, ["a", "b"]))+ for (predict, log), probability in zip(results, [0.2, 0.8], strict=True):+ assert len(log) == 2+ assert all(+ caller is predict and name == "match" and evidence == {"noul": probability}+ for caller, name, evidence in log+ )+++def test_nested_evidence_collector_restores_outer_context_after_error(system_one):+ from dspy.adapters.decision import record_evidence++ system_one(lambda state, name, q: noul(float(state["inputs"]["pair"])))+ predict = dspy.Predict(Match)+ with record_evidence() as outer:+ predict(pair="0.1")+ with pytest.raises(RuntimeError, match="stop"):+ with record_evidence() as inner:+ predict(pair="0.2")+ raise RuntimeError("stop")+ predict(pair="0.3")+ predict(pair="0.4")+ assert [evidence["noul"] for _, _, evidence in outer] == [0.1, 0.3]+ assert [evidence["noul"] for _, _, evidence in inner] == [0.2]+ assert dspy.settings.get("_decision_evidence") is None+++def lone_outlier(same: float, different: float, odd: float, count: int = 1):+ """20 same pairs, 19 different pairs, and `count` of the same pairs replaced by `odd` ones."""+ pairs = (+ [f"same{i}" for i in range(20 - count)] + [f"odd{i}" for i in range(count)] + [f"diff{i}" for i in range(19)]+ )++ def answer(state, name, q):+ pair = state["inputs"]["pair"]+ return noul(odd if pair.startswith("odd") else same if pair.startswith("same") else different)++ train = [dspy.Example(pair=p, match=not p.startswith("diff")).with_inputs("pair") for p in pairs]+ return answer, train+++def test_a_threshold_stays_when_its_whole_gain_is_one_example(system_one):+ answer, train = lone_outlier(0.9, 0.1, 0.3)+ system_one(answer)+ program = dspy.Predict(Match)+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)+ assert program.fields == {}+ assert report[0]["fold_check"] == {"passed": 0, "failed": 1}+++def test_a_threshold_moves_when_its_gain_recurs_across_the_training_set(system_one):+ answer, train = lone_outlier(0.9, 0.1, 0.3, count=6)+ system_one(answer)+ program = dspy.Predict(Match)+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)+ assert program.fields["match"]["threshold"] == 0.2+ assert report[0]["fold_check"] == {"passed": 1, "failed": 0}+++def test_score_cuts_stay_when_their_whole_gain_is_one_example(system_one):+ levels = {"lo": {0: 0.8, 1: 0.1, 2: 0.1}, "hi": {0: 0.1, 1: 0.0, 2: 0.9}, "odd": {0: 0.5, 1: 0.3, 2: 0.2}}+ system_one(lambda state, name, q: score(levels[state["inputs"]["item"].rstrip("0123456789")]))+ program = dspy.Predict(RateLevel)+ items = [f"lo{i}" for i in range(19)] + ["odd0"] + [f"hi{i}" for i in range(20)]+ train = [dspy.Example(item=k, rating=2 if k.startswith("hi") else 0).with_inputs("item") for k in items]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.rating.level == g.rating), num_threads=2)+ assert program.fields == {}+ assert report[0]["fold_check"]["failed"] >= 1+++def test_choice_weights_stay_when_their_whole_gain_is_one_example(system_one):+ leans = {"x": {"x": 0.8, "y": 0.2}, "y": {"x": 0.2, "y": 0.8}, "odd": {"x": 0.55, "y": 0.45}}+ system_one(lambda state, name, q: choice(leans[state["inputs"]["item"].rstrip("0123456789")]))+ program = dspy.Predict(Kind)+ items = [f"x{i}" for i in range(20)] + [f"y{i}" for i in range(19)] + ["odd0"]+ train = [dspy.Example(item=k, kind="x" if k.startswith("x") else "y").with_inputs("item") for k in items]+ report = calibrate(program, train, lambda g, p, trace=None: float(p.kind == g.kind), num_threads=2)+ assert program.fields == {}+ assert report[0]["fold_check"]["failed"] >= 1