Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Add experimental ReAnchor calibration optimizer for Predict (#10475)

Isaac Millere8ec009c6bd9
diff --git a/docs/docs/api/experimental/DecisionTypes.md b/docs/docs/api/experimental/DecisionTypes.md
index 9e104de91..e756a4bfb 100644
--- a/docs/docs/api/experimental/DecisionTypes.md
+++ b/docs/docs/api/experimental/DecisionTypes.md
@@ -284,5 +284,7 @@ annotations. This restriction applies to outputs, not structured input data.
Use TypeSafe through `Predict`, rather than calling chat adapters directly.
Generation-specific wrappers such as `BestOfN` and `Refine`, generative optimizers,
and fine-tuning are unsupported with TypeSafe.
+[ReAnchor](ReAnchor.md) calibrates decision outputs with TypeSafe or a
+generative LM, and it sends no generation settings.
Wrappers that assume LM generation settings can currently raise `AttributeError`
rather than a capability-specific error.
diff --git a/docs/docs/api/experimental/ReAnchor.md b/docs/docs/api/experimental/ReAnchor.md
new file mode 100644
index 000000000..f31139ee3
--- /dev/null
+++ b/docs/docs/api/experimental/ReAnchor.md
@@ -0,0 +1,148 @@
+# dspy.experimental.ReAnchor
+
+!!! warning "Experimental API"
+ `ReAnchor` is experimental and may change without warning.
+
+`ReAnchor` fits the numeric decision settings in a program to your metric. These
+settings are the `threshold`, `cuts`, and `weights` entries in each predictor's
+`fields`. They are described in [Decision types and System One models](DecisionTypes.md).
+ReAnchor does not change any instructions, descriptions, or demos.
+
+You give ReAnchor a metric, a program, and training examples. The program can be
+a single `dspy.Predict` or any module that holds predictors with decision
+outputs. A decision output is a `Noul`, `Score`, or `Choice` output, or a native
+`bool` or `Literal` output. `RLM` does not support decision outputs, so ReAnchor
+does not support RLM programs either.
+
+```python
+import dspy
+from dspy.experimental import ReAnchor, TypeSafe
+
+
+class Match(dspy.Signature):
+ """Decide whether two product listings describe the same item."""
+
+ pair: str = dspy.InputField(desc="Two listings.")
+ match: bool = dspy.OutputField(desc="Are they the same item?")
+
+
+def metric(example, prediction, trace=None):
+ return float(prediction.match == example.match)
+
+
+dspy.configure(lm=TypeSafe("jev-latest"))
+optimizer = ReAnchor(metric)
+tuned = optimizer.compile(dspy.Predict(Match), trainset=trainset, valset=valset)
+print(optimizer.report)
+```
+
+## What ReAnchor fits
+
+ReAnchor searches each output's numeric settings against the whole-program
+metric. A new setting must improve the training score and pass a fold check.
+
+For every compatible output, ReAnchor saves the original configuration and score,
+enables probability-based execution, and fits the numeric settings. It then compares
+the fitted behavior against the original using a fold check. If rejected, it restores
+the exact original configuration, including removing an entry that was originally absent.
+The adapter handles the backend used on each call; calibration does not inspect it.
+
+Before fitting each output, ReAnchor runs the program and records that output's
+probabilities on every call. A threshold anywhere between two neighboring P(True) values makes the
+same decisions, so ReAnchor tries the midpoint of each gap between them. For
+example, when the model only returns P(True) of 0.98 and 1.0, ReAnchor tries
+0.5 and 0.99.
+
+- For a Boolean output, ReAnchor tries each midpoint between the observed
+ P(True) values.
+- For a Score output, the level depends on the mean level index. ReAnchor tries
+ each `cut` at the midpoints between the observed mean indexes, and it keeps
+ the cuts in order. The cuts choose `.level` and do not change `.value`.
+- For a Choice output, ReAnchor moves one option's multiplier at a time. It
+ tries the multipliers that fall between the points where that option's pick
+ would flip on some call.
+
+The current setting stays unless another setting scores strictly better and
+passes a fold check. The check splits the training examples into up to five parts.
+For each part, ReAnchor picks a setting using the remaining parts and scores
+that pick on the held-out part. A new setting is kept only when the combined held-out
+score beats the current setting. The check discourages gains confined to small
+portions of the dataset, but can accept them when they recur across folds.
+Among improving candidates with equal scores, ReAnchor prefers the widest gap,
+which leaves the most room on either side. Each search step tries at most
+40 gap midpoints, thinning large lists to settings spaced evenly through
+the observed values. For Boolean outputs, it also tries threshold zero when
+P(True)=0 is observed, since that boundary is the only way to classify those
+answers as True. When two observed probabilities are adjacent floats, it tries
+the upper value as a threshold: `p >= threshold` separates them without a midpoint.
+
+## Requests and the cache
+
+The numeric settings are not part of the request. Repeated identical requests
+reuse cached answers. In a composed program, changing an upstream decision can
+change downstream inputs or which predictors run, producing new requests and
+additional backend calls. Native-output promotion also changes the request.
+
+`compile` requires caching by default to avoid repeating identical backend
+calls; it does not guarantee a fixed request count. Pass `require_cache=False`
+to run without this check. The cache check inspects statically bound or globally
+configured clients. For programs selecting clients inside `forward()`, disable
+the check and manage caching on those clients.
+
+## Native outputs on a generative LM
+
+A generative LM answers a native `bool` or `Literal` output with a single value.
+It does not report probabilities, so ReAnchor has nothing to fit. When an output
+has an entry in the predictor's `fields`, `Predict` asks the LM for
+probabilities instead. `Predict` then applies the threshold or weights to pick
+the value, and the output still returns a `bool` or a `Literal` member.
+
+The first pass with probabilities sends new requests, because the request changes.
+If the fitted behavior is rejected, restoring the original configuration returns
+an unconfigured native field to direct generation.
+
+```python
+dspy.configure(lm=dspy.LM("your-provider/your-model")) # Use your model's identifier.
+tuned = ReAnchor(metric).compile(dspy.Predict(Match), trainset=trainset)
+```
+
+A System One model such as Jev always returns probabilities. Enabling probability
+execution leaves its request unchanged; restoring a field restores its original
+decoding settings, not direct generation.
+
+## Errors
+
+ReAnchor stops at the first error from the program or the metric. On a
+generative LM, a malformed answer counts as an error. For example, `Predict`
+raises when a `Score` answer leaves out a probability for any level. Use a
+model that follows JSON schemas reliably, and set a client `timeout` that fits
+your backend's load.
+
+## Results
+
+`compile` returns a copy of the program and leaves your program unchanged. After
+`compile`, `report` holds:
+
+- `train_score_before` and `train_score`, the mean metric score on the training
+ examples before and after calibration.
+- `val_score_before` and `val_score`, the same scores on `valset` when you pass
+ one. ReAnchor never fits settings on `valset`.
+- `fitted`, one row per output with the fitted value, or the reason ReAnchor
+ skipped it. Each fitted row has an `observed` entry with the number of calls
+ and settings tried; threshold and cut reports also summarize observed values. Its
+ `fold_check` entry counts the search steps whose better training score passed
+ or failed the fold check. Per-field `train_score_original` is the score before
+ enabling probabilities, `train_score_at_start` is the probability-based baseline,
+ and `train_score` describes the retained behavior. Rejected configurations have
+ a `skipped` reason instead of a fitted `value`.
+
+Your metric may return a number or a `dspy.Prediction` with a `score`.
+
+When you set `log_dir`, ReAnchor writes `report.json` to that folder.
+
+The fitted settings are part of each predictor's `fields`, so the tuned program
+saves and loads like any other `Predict` program.
+
+::: dspy.experimental.ReAnchor
+ options:
+ members: [__init__, compile]
diff --git a/docs/docs/api/experimental/index.md b/docs/docs/api/experimental/index.md
index d152c6625..1c6a2f318 100644
--- a/docs/docs/api/experimental/index.md
+++ b/docs/docs/api/experimental/index.md
@@ -6,4 +6,5 @@ API reference for DSPy experimental. Select a page below.
- [Citations](Citations.md)
- [Decision types and System One models](DecisionTypes.md)
- [Document](Document.md)
+- [ReAnchor](ReAnchor.md)
<!-- END_API_INDEX -->
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
index 70e047012..b57e76dfa 100644
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -113,6 +113,7 @@ nav:
- Citations: api/experimental/Citations.md
- Decision types and System One models: api/experimental/DecisionTypes.md
- Document: api/experimental/Document.md
+ - ReAnchor: api/experimental/ReAnchor.md
- Models:
- api/models/index.md
- BaseLM: api/models/BaseLM.md
diff --git a/dspy/adapters/decision.py b/dspy/adapters/decision.py
index 5bafa93ba..b8dc80526 100644
--- a/dspy/adapters/decision.py
+++ b/dspy/adapters/decision.py
@@ -2,13 +2,28 @@
import copy
import json
+import threading
import warnings
+from contextlib import contextmanager
from functools import lru_cache
from pydantic import BaseModel, ConfigDict, Field, create_model
from dspy.adapters.decision_state import DecisionState
from dspy.adapters.types.decision import Choice, Noul, Probability, Score
+from dspy.dsp.utils.settings import settings
+
+
+@contextmanager
+def record_evidence():
+ """Collect decision evidence in this execution context and its DSPy workers.
+
+ Yields a list that receives one ``(predictor, output name, evidence)`` entry per decoded
+ output. ``predictor`` is the innermost calling module. Nested collectors are independent.
+ """
+ log = []
+ with settings.context(_decision_evidence=(log, threading.Lock())):
+ yield log
@lru_cache(maxsize=256)
@@ -60,7 +75,6 @@ class DecisionAdapter:
def _prepare(self, signature, demos, inputs, lm_kwargs):
from dspy.adapters.utils import get_field_description_string
- from dspy.dsp.utils.settings import settings
from dspy.predict.predict import serialize_object
if settings.send_stream is not None:
@@ -112,6 +126,12 @@ class DecisionAdapter:
raise ValueError(f"Invalid Score distribution for {name!r}.")
answer["probabilities"] = {int(k): v for k, v in answer["probabilities"].items()}
answers[name] = answer
+ recorder = settings.get("_decision_evidence")
+ if recorder is not None:
+ log, lock = recorder
+ caller = (settings.caller_modules or [None])[-1]
+ with lock:
+ log.extend((caller, name, copy.deepcopy(answer)) for name, answer in answers.items())
results.append({**completion, **self.state._decode(answers)})
return results
diff --git a/dspy/experimental/__init__.py b/dspy/experimental/__init__.py
index 76481e2d5..e99c24437 100644
--- a/dspy/experimental/__init__.py
+++ b/dspy/experimental/__init__.py
@@ -2,6 +2,7 @@ from dspy.adapters.types.citation import Citations
from dspy.adapters.types.decision import Choice, Noul, Score
from dspy.adapters.types.document import Document
from dspy.clients.typesafe import TypeSafe
+from dspy.teleprompt.reanchor import ReAnchor
__all__ = [
"Citations",
@@ -10,4 +11,5 @@ __all__ = [
"Noul",
"Score",
"TypeSafe",
+ "ReAnchor",
]
diff --git a/dspy/teleprompt/reanchor/__init__.py b/dspy/teleprompt/reanchor/__init__.py
new file mode 100644
index 000000000..34598adf4
--- /dev/null
+++ b/dspy/teleprompt/reanchor/__init__.py
@@ -0,0 +1,3 @@
+from dspy.teleprompt.reanchor.reanchor import ReAnchor
+
+__all__ = ["ReAnchor"]
diff --git a/dspy/teleprompt/reanchor/calibrate.py b/dspy/teleprompt/reanchor/calibrate.py
new file mode 100644
index 000000000..c27e934a1
--- /dev/null
+++ b/dspy/teleprompt/reanchor/calibrate.py
@@ -0,0 +1,354 @@
+"""Fit the numeric decision parameters of every Predict in a program against the metric.
+
+Each `Predict` holds per-output parameters in `fields` that reinterpret the backend's
+probabilities without adding request parameters. Each candidate reruns the program over the
+training set, reusing cached answers for identical requests. Changed upstream decisions can
+produce new downstream requests. The search runs directly against the metric, whatever shape
+the program's outputs have.
+
+Three kinds of parameter are fitted, all in the `Predict`'s `fields[field]` configuration:
+- `threshold`, a Boolean output's cut point on P(True).
+- `cuts`, a Score's boundaries on the mean level index. They pick the returned `.level` and leave
+ `.value` alone.
+- `weights`, a Choice's multiplier per option label, applied to the probabilities before the
+ option is picked. An option picked too often gets a multiplier below 1.
+
+The candidate settings come from the training set. One pass records the probabilities the output
+is decoded from on every call. A setting anywhere between two neighbouring observed values makes
+the same decisions, so the candidates are the midpoints of those gaps: between P(True) values for
+a threshold, between mean level indexes for a cut, and between the points where an option's pick
+flips for a weight. Among equal scores the candidate in the widest gap wins.
+
+A setting stays unless a candidate scores strictly better and the gain holds across folds. The
+fold check splits the training set into FOLDS parts. For each part, it picks a setting on the
+other parts and scores that pick on the held-out part. A candidate replaces the current setting
+only when those held-out scores beat the current setting's. This discourages gains confined to
+small portions of the dataset, but can accept them when they recur across folds.
+
+Calibration enables probability-based execution for every compatible output by adding an entry
+in `fields`. After fitting, it compares against the original behavior under the same fold check
+and restores the original field configuration unless the fitted behavior wins. The adapter
+handles probability requests for the backend used on each call.
+
+"""
+
+import copy
+import itertools
+import math
+import random
+from typing import Any, Callable
+
+import dspy
+from dspy.adapters.decision import record_evidence
+from dspy.adapters.decision_state import DecisionState
+from dspy.adapters.types.decision import Choice, Noul, decision_type
+from dspy.predict.predict import Predict
+from dspy.utils.parallelizer import ParallelExecutor
+
+MAX_CANDIDATES = 40 # candidate settings per parameter; more distinct values are thinned to quantiles
+WEIGHT_RANGE = (1e-3, 1e3) # the smallest and largest Choice multiplier tried
+FOLDS = 5 # training-set parts the fold check holds out in turn
+
+
+def decision_outputs(predict: Predict) -> dict[str, type]:
+ """The outputs a decision parameter can apply to, with their decision types."""
+ kinds = {name: decision_type(field) for name, field in predict.signature.output_fields.items()}
+ return {name: kind for name, kind in kinds.items() if kind is not None}
+
+
+def predictors(program) -> list[tuple[str, Predict]]:
+ """Every Predict with a decision output, by the name `named_parameters` gives it. A bare Predict is `self`."""
+ return [(name, p) for name, p in program.named_parameters() if isinstance(p, Predict) and decision_outputs(p)]
+
+
+def caches(predict: Predict) -> bool:
+ """Check caching for a statically bound or configured client."""
+ lm = predict.lm or dspy.settings.lm
+ if lm is None:
+ raise ValueError(
+ "ReAnchor cannot check caching without a bound or globally configured client. "
+ "For runtime client selection, pass require_cache=False."
+ )
+ enabled = predict.config.get("cache", getattr(lm, "cache", False))
+ return bool(enabled) and getattr(lm, "_cache_responses", True)
+
+
+def effective(predict: Predict, field: str) -> dict:
+ """The output's full configuration: its stored overrides on top of its type's defaults."""
+ fields = dict(predict.fields)
+ fields.setdefault(field, {})
+ return DecisionState(predict.signature, fields).fields[field]
+
+
+def metric_value(metric: Callable, example, pred) -> float:
+ """The metric's score as a float, from a number or from a prediction carrying `score`."""
+ verdict = metric(example, pred)
+ return verdict.score if hasattr(verdict, "score") else float(verdict)
+
+
+def scores(
+ program, examples: list, metric: Callable, num_threads: int | None = None, progress: bool = False
+) -> list[float]:
+ """The program's metric score on each example. Any program or metric error fails the pass."""
+
+ def one(example):
+ with dspy.context(trace=[]):
+ pred = program(**example.inputs())
+ return metric_value(metric, example, pred)
+
+ executor = ParallelExecutor(num_threads=num_threads, max_errors=1, disable_progress_bar=not progress)
+ values = executor.execute(one, examples)
+ if not all(math.isfinite(s) for s in values):
+ raise ValueError("ReAnchor requires finite metric values.")
+ return values
+
+
+def run(program, examples: list, metric: Callable, num_threads: int | None = None, progress: bool = False) -> float:
+ """The program's mean metric score over `examples`."""
+ values = scores(program, examples, metric, num_threads, progress)
+ return math.fsum(values) / len(values)
+
+
+def calibrate(
+ program,
+ trainset: list,
+ metric: Callable,
+ num_threads: int | None = None,
+) -> list[dict[str, Any]]:
+ """Fit every threshold, cut, and weight in place.
+
+ Returns one report row per output: the fitted value, or why it was skipped.
+ """
+ report = []
+
+ def score() -> list[float]:
+ return scores(program, trainset, metric, num_threads)
+
+ for name, predict in predictors(program):
+ for field, kind in decision_outputs(predict).items():
+ original = copy.deepcopy(predict.fields.get(field))
+ before = score()
+ predict.fields[field] = effective(predict, field)
+ evidence = _observe(program, predict, field, trainset, metric, num_threads)
+ row = {"predictor": name, "field": field}
+ parameter, fit = (
+ ("threshold", _fit_threshold)
+ if issubclass(kind, Noul)
+ else ("weights", _fit_weights)
+ if issubclass(kind, Choice)
+ else ("cuts", _fit_cuts)
+ )
+ base = score()
+ best, fitted, observed, check = fit(predict, field, kind, evidence, score, base)
+ row.update(parameter=parameter, value=best, train_score=_mean(fitted), train_score_at_start=_mean(base))
+ row.update(observed=observed, fold_check=check, train_score_original=_mean(before))
+ if _select(before, [(fitted, (), None)])[0] is None:
+ _restore(predict, field, original)
+ row.pop("value")
+ row.update(skipped="fitted behavior did not beat the original", train_score=_mean(before))
+ report.append(row)
+ return report
+
+
+def _restore(predict: Predict, field: str, original: dict | None) -> None:
+ """Put back the output's entry in `fields` as it was before calibration, or remove it."""
+ if original is None:
+ del predict.fields[field]
+ else:
+ predict.fields[field] = original
+
+
+def _observe(program, predict: Predict, field: str, trainset: list, metric: Callable, num_threads) -> list[dict]:
+ """The evidence the output is decoded from on each training call, from one pass at the current setting."""
+ with record_evidence() as log:
+ scores(program, trainset, metric, num_threads)
+ return [evidence for caller, name, evidence in log if caller is predict and name == field]
+
+
+def _tidy(x: float, lo: float, hi: float) -> float:
+ """The shortest decimal strictly between `lo` and `hi`, rounded from `x`."""
+ for digits in range(1, 12):
+ rounded = round(x, digits)
+ if lo < rounded < hi:
+ return rounded
+ return x
+
+
+def _gaps(values: list[float], lo: float, hi: float, geometric: bool = False) -> list[tuple[float, float]]:
+ """Candidate settings and the width of the gap each sits in.
+
+ The candidates are the midpoints between neighbouring distinct values, with `lo` and `hi`
+ closing the range. A setting anywhere inside one gap makes the same decisions as its midpoint,
+ so these cover every outcome. More than MAX_CANDIDATES gaps are thinned to evenly spaced
+ quantiles of the values. `geometric` takes midpoints and widths on a log scale.
+ """
+ inner = sorted({v for v in values if lo < v < hi})
+ if len(inner) > MAX_CANDIDATES - 1:
+ step = (len(inner) - 1) / (MAX_CANDIDATES - 2)
+ inner = sorted({inner[round(i * step)] for i in range(MAX_CANDIDATES - 1)})
+ points = [lo, *inner, hi]
+ gaps = []
+ for a, b in itertools.pairwise(points):
+ middle, width = (math.sqrt(a * b), math.log(b / a)) if geometric else ((a + b) / 2, b - a)
+ if a < middle < b: # Adjacent floats can round the midpoint onto a boundary.
+ gaps.append((_tidy(middle, a, b), width))
+ return gaps
+
+
+def _summary(values: list[float]) -> dict[str, Any]:
+ distinct = sorted(set(values))
+ if not distinct:
+ return {"calls": 0}
+ return {
+ "calls": len(values),
+ "distinct": len(distinct),
+ "min": round(distinct[0], 4),
+ "max": round(distinct[-1], 4),
+ }
+
+
+def _mean(values: list[float]) -> float:
+ return round(math.fsum(values) / len(values), 4)
+
+
+def _folds(n: int) -> list[list[int]]:
+ """Example indexes split into up to FOLDS parts, the same way on every run."""
+ order = list(range(n))
+ random.Random(0).shuffle(order)
+ k = min(FOLDS, n)
+ return [order[i::k] for i in range(k)]
+
+
+def _pick(start: list[float], tried: list[tuple], rows) -> tuple | None:
+ """The best entry of `tried` on `rows`, or None when none scores strictly better than `start` there.
+
+ Each entry is (per-example scores, tie-break key, setting). Equal totals go to the larger key.
+ """
+
+ def total(values):
+ return math.fsum(values[i] for i in rows)
+
+ best = max(tried, key=lambda t: (total(t[0]), t[1]), default=None)
+ return best if best is not None and total(best[0]) > total(start) else None
+
+
+def _select(start: list[float], tried: list[tuple]) -> tuple[tuple | None, bool]:
+ """The entry of `tried` to keep, or None to keep the start, and whether the fold check refused a better entry.
+
+ The entry that scores best on the whole training set is kept only when picking on all folds but
+ one, and scoring the pick on the held-out fold, beats the start on those same examples.
+ """
+ best = _pick(start, tried, range(len(start)))
+ if best is None:
+ return None, False
+ held, held_start = 0.0, 0.0
+ for fold in _folds(len(start)):
+ out = set(fold)
+ pick = _pick(start, tried, [i for i in range(len(start)) if i not in out])
+ held += math.fsum((pick[0] if pick else start)[i] for i in fold)
+ held_start += math.fsum(start[i] for i in fold)
+ return (best, False) if held > held_start else (None, True)
+
+
+def _fit_threshold(
+ predict: Predict, field: str, kind: type, evidence: list[dict], score: Callable, base: list[float]
+) -> tuple[float, list[float], dict, dict]:
+ """The threshold that scores best, between the observed probabilities.
+
+ The starting threshold stays unless a candidate scores strictly better and passes the fold
+ check. Among equal scores, the candidate in the widest gap wins, since it leaves the most room
+ on either side.
+ """
+ config = predict.fields[field]
+ start = config["threshold"]
+ probabilities = [e["noul"] for e in evidence]
+ tried = []
+ candidates = _gaps(probabilities, 0.0, 1.0)
+ for lower, upper in itertools.pairwise(sorted(set(probabilities))):
+ if math.nextafter(lower, upper) == upper:
+ # No interior float exists, but p >= upper still separates the pair.
+ candidates.append((upper, upper - lower))
+ if 0.0 in probabilities:
+ candidates.append((0.0, 0.0)) # P(True) >= threshold makes zero a distinct outcome.
+ for t, width in candidates:
+ config["threshold"] = t
+ tried.append((score(), (width, -abs(t - start), t), t))
+ kept, refused = _select(base, tried)
+ best, values = (kept[2], kept[0]) if kept else (start, base)
+ config["threshold"] = best
+ observed = {**_summary(probabilities), "candidates": len(tried)}
+ return best, values, observed, {"passed": int(kept is not None), "failed": int(refused)}
+
+
+def _fit_weights(
+ predict: Predict, field: str, kind: type, evidence: list[dict], score: Callable, base: list[float]
+) -> tuple[dict, list[float], dict, dict]:
+ """The Choice multipliers that score best on the training set.
+
+ Each option's multiplier moves in turn while the others hold. An option's pick on a call flips
+ where its weighted probability meets the strongest rival's, so the candidates sit between those
+ flip points. A candidate replaces the current multipliers only when it scores strictly better
+ and passes the fold check.
+ """
+ labels = list(kind.criteria())
+ config = predict.fields[field]
+ best = {label: config["weights"].get(label, 1.0) for label in labels}
+ best_scores, candidates, check = base, 0, {"passed": 0, "failed": 0}
+ for label in labels:
+ flips = []
+ for e in evidence:
+ p = e["probabilities"]
+ rival = max(best[other] * p[other] for other in labels if other != label)
+ if p[label] > 0 and rival > 0:
+ flips.append(rival / p[label])
+ low, high = WEIGHT_RANGE
+ if flips:
+ low, high = max(low, min(flips) / 4), min(high, max(flips) * 4)
+ tried = []
+ for w, width in _gaps(flips, low, high, geometric=True):
+ if w == best[label]:
+ continue
+ candidate = {**best, label: w}
+ config["weights"] = candidate
+ tried.append((score(), width, candidate))
+ candidates += len(tried)
+ kept, refused = _select(best_scores, tried)
+ check["passed"] += kept is not None
+ check["failed"] += refused
+ if kept:
+ best_scores, _, best = kept
+ config["weights"] = best
+ return best, best_scores, {"calls": len(evidence), "candidates": candidates}, check
+
+
+def _fit_cuts(
+ predict: Predict, field: str, kind: type, evidence: list[dict], score: Callable, base: list[float]
+) -> tuple[list[float], list[float], dict, dict]:
+ """The Score cuts that score best on the training set.
+
+ A call's level depends only on its mean level index. Each cut moves in turn between the
+ observed means that lie between its neighbours, and stays strictly between them. A candidate
+ replaces the current cuts only when it scores strictly better and passes the fold check.
+ """
+ config = predict.fields[field]
+ top = len(kind.criteria()) - 1
+ means = [sum(i * p for i, p in e["probabilities"].items()) / sum(e["probabilities"].values()) for e in evidence]
+ best, best_scores, candidates, check = list(config["cuts"]), base, 0, {"passed": 0, "failed": 0}
+ for i in range(len(best)):
+ below = best[i - 1] if i else 0
+ above = best[i + 1] if i + 1 < len(best) else top
+ tried = []
+ for c, width in _gaps(means, below, above):
+ if c == best[i]:
+ continue
+ candidate = [*best[:i], c, *best[i + 1 :]]
+ config["cuts"] = candidate
+ tried.append((score(), width, candidate))
+ candidates += len(tried)
+ kept, refused = _select(best_scores, tried)
+ check["passed"] += kept is not None
+ check["failed"] += refused
+ if kept:
+ best_scores, _, best = kept
+ config["cuts"] = best
+ return best, best_scores, {**_summary(means), "candidates": candidates}, check
diff --git a/dspy/teleprompt/reanchor/reanchor.py b/dspy/teleprompt/reanchor/reanchor.py
new file mode 100644
index 000000000..bb51e6b7e
--- /dev/null
+++ b/dspy/teleprompt/reanchor/reanchor.py
@@ -0,0 +1,87 @@
+"""Fit a program's decision thresholds, cuts, and weights against the metric."""
+
+import json
+import logging
+from pathlib import Path
+from typing import Any
+
+from dspy.teleprompt.reanchor.calibrate import caches, calibrate, predictors, run
+from dspy.teleprompt.teleprompt import Teleprompter
+from dspy.utils.annotation import experimental
+
+logger = logging.getLogger(__name__)
+
+
+@experimental
+class ReAnchor(Teleprompter):
+ """Calibrate a program's decisions: fit every threshold, Score cut, and Choice weight against the metric.
+
+ The program can be a single `Predict` or any `dspy.Module` holding predictors with decision
+ outputs, and the metric is the only supervision. Each parameter changes how `Predict` reads the
+ backend's probabilities without adding request parameters. Identical requests reuse cached
+ answers, but changed upstream decisions can produce new downstream requests.
+
+ ReAnchor enables probability-based execution for every compatible output through the predictor's
+ `fields`. It keeps each fitted configuration only when it beats the original behavior and passes
+ a fold check; otherwise it restores the original configuration, including absent field entries.
+
+ Args:
+ metric: Per-example metric to maximize, as in `dspy.Evaluate`. It may return a number, or a
+ `dspy.Prediction` with a `score`.
+ num_threads: Evaluation concurrency, as in `dspy.Evaluate`.
+ log_dir: When set, report.json is written here.
+ require_cache: When True, check caching on statically bound or globally configured clients.
+ Set it to False for runtime client selection or to allow uncached requests.
+
+ After `compile`, `report` holds the fitted parameters and the metric's mean before and after
+ calibration, on the training set and on the validation set when one is given.
+ """
+
+ def __init__(self, metric, *, num_threads=None, log_dir=None, require_cache=True):
+ super().__init__()
+ self.metric = metric
+ self.num_threads = num_threads
+ self.log_dir = Path(log_dir) if log_dir else None
+ self.require_cache = require_cache
+ self.report: dict[str, Any] = {}
+
+ def compile(self, student, *, trainset, valset=None):
+ """Return a calibrated copy; leave the student unchanged.
+
+ Args:
+ student: A `Predict`, or a module holding predictors with decision outputs.
+ trainset: Examples the parameters are fitted on.
+ valset: Examples scored before and after calibration for the report. Nothing is fitted on them.
+ """
+ if not trainset:
+ raise ValueError("trainset must contain at least one example.")
+ program = student.deepcopy()
+ found = predictors(program)
+ if not found:
+ raise ValueError("The student must contain at least one Predict with a decision output.")
+ uncached = [name for name, predict in found if not caches(predict)] if self.require_cache else []
+ if self.require_cache and uncached:
+ raise ValueError(
+ f"Predictors {uncached} do not cache responses, so every candidate setting would send new requests. "
+ "Enable the client's cache, or pass require_cache=False to calibrate anyway."
+ )
+
+ logger.info("answering %d training examples", len(trainset))
+ before = self._score(program, trainset, progress=True)
+ val_before = self._score(program, valset) if valset else None
+ logger.info("fitting thresholds, cuts, and weights")
+ fitted = calibrate(program, trainset, self.metric, num_threads=self.num_threads)
+ self.report = {"train_score_before": before, "train_score": self._score(program, trainset), "fitted": fitted}
+ logger.info("calibrated: train %s -> %s", before, self.report["train_score"])
+ if valset:
+ self.report.update(val_score_before=val_before, val_score=self._score(program, valset))
+ logger.info("validation: %s -> %s", val_before, self.report["val_score"])
+
+ if self.log_dir:
+ self.log_dir.mkdir(parents=True, exist_ok=True)
+ (self.log_dir / "report.json").write_text(json.dumps(self.report, indent=2, default=str), encoding="utf-8")
+ program._compiled = True
+ return program
+
+ def _score(self, program, examples: list, progress: bool = False) -> float:
+ return round(run(program, examples, self.metric, self.num_threads, progress), 4)
diff --git a/tests/teleprompt/reanchor/conftest.py b/tests/teleprompt/reanchor/conftest.py
new file mode 100644
index 000000000..c87be1ec2
--- /dev/null
+++ b/tests/teleprompt/reanchor/conftest.py
@@ -0,0 +1,16 @@
+import pytest
+
+import dspy
+from tests.teleprompt.reanchor.fakes import FakeClient
+
+
+def system_one():
+ """Configures a FakeClient as the LM for one test."""
+
+ def install(answer, cache=True):
+ client = FakeClient(answer, cache=cache)
+ dspy.configure(lm=client)
+ return client
+
+ return install
diff --git a/tests/teleprompt/reanchor/fakes.py b/tests/teleprompt/reanchor/fakes.py
new file mode 100644
index 000000000..6f325cb5e
--- /dev/null
+++ b/tests/teleprompt/reanchor/fakes.py
@@ -0,0 +1,68 @@
+"""Stand-in decision clients and generative LMs for the ReAnchor tests."""
+
+import copy
+import re
+import threading
+
+from dspy.clients.engines.dummy_engine import AsyncDummyEngine, DummyEngine
+from dspy.utils.dummies import DummyLM
+
+
+class FakeClient:
+ """A System One client. Answers each question with `answer(state, name, question)`, and records every request."""
+
+ supports_decision_requests = True
+
+ def __init__(self, answer, cache=True):
+ self.answer = answer
+ self.cache = cache
+ self.calls = []
+ self.history = []
+ self.callbacks = []
+
+ def __call__(self, state, questions):
+ self.calls.append(copy.deepcopy((state, questions)))
+ return {name: self.answer(state, name, q) for name, q in questions.items()}
+
+ async def acall(self, state, questions):
+ return self(state, questions)
+
+
+class _ComputedEngine(DummyEngine):
+ _lock = threading.Lock()
+
+ def _complete_messages(self, messages):
+ with self._lock:
+ self.owner.answers = {"": self.owner.answer(messages)}
+ return super()._complete_messages(messages)
+
+
+class ComputedLM(DummyLM):
+ """A generative LM whose output field values are `answer(inputs, evidence)`.
+
+ `inputs` maps each input field to its text in the final message, and `evidence` is True when
+ the request asks for probabilities instead of native values.
+ """
+
+ def __init__(self, answer, adapter=None):
+ super().__init__({}, adapter=adapter)
+ self.answer_fn = answer
+ self._engine_spec = _ComputedEngine(self)
+ self._async_engine_spec = AsyncDummyEngine(self._engine_spec)
+
+ def answer(self, messages):
+ inputs = dict(re.findall(r"\[\[ ## (\w+) ## \]\]\n(.*)", messages[-1]["content"]))
+ evidence = any("noul" in m["content"] or "probabilities" in m["content"] for m in messages)
+ return self.answer_fn(inputs, evidence)
+
+
+def noul(p: float) -> dict:
+ return {"noul": p}
+
+
+def score(probabilities: dict[int, float], confidence: float = 0.8) -> dict:
+ return {"score": 0.0, "confidence": confidence, "probabilities": probabilities}
+
+
+def choice(probabilities: dict[str, float], confidence: float = 0.7) -> dict:
+ return {"confidence": confidence, "probabilities": probabilities}
diff --git a/tests/teleprompt/reanchor/test_reanchor.py b/tests/teleprompt/reanchor/test_reanchor.py
new file mode 100644
index 000000000..e5e526fc1
--- /dev/null
+++ b/tests/teleprompt/reanchor/test_reanchor.py
@@ -0,0 +1,291 @@
+"""ReAnchor on a stubbed System One client and a stubbed generative LM."""
+
+import copy
+import json
+import logging
+
+import pytest
+
+import dspy
+from dspy.experimental import ReAnchor
+from tests.teleprompt.reanchor.fakes import ComputedLM, noul
+
+
+def leaning(state, name, q):
+ """Jev leans high, 0.9 on a same pair and 0.7 on a different one, and reads a tricky same pair
+ as different, so no threshold gets every pair right. The best thresholds lie between 0.7 and 0.9; ReAnchor picks the midpoint, 0.8."""
+ pair = state["inputs"]["pair"]
+ if pair.endswith("same-tricky"):
+ return noul(0.7)
+ return noul(0.9 if "same" in pair else 0.7)
+
+
+class Sig(dspy.Signature):
+ pair: str = dspy.InputField()
+ match: bool = dspy.OutputField(desc="Are the two the same?")
+
+
[email protected](autouse=True)
+def jev(system_one):
+ return system_one(leaning)
+
+
[email protected](autouse=True)
+def settings():
+ # Configured rather than set with `dspy.context`, which would pin the client a test installs later.
+ dspy.configure(adapter=dspy.JSONAdapter())
+
+
+def examples(prefix=""):
+ kinds = ["same"] * 6 + ["same-tricky"] * 4 + ["different"] * 8
+ return [dspy.Example(pair=prefix + k, match=k.startswith("same")).with_inputs("pair") for k in kinds]
+
+
+def metric(gold, pred, trace=None):
+ return float(pred.match == gold.match)
+
+
+def test_compile_fits_the_threshold_and_reports_the_training_scores():
+ optimizer = ReAnchor(metric, num_threads=2)
+ program = optimizer.compile(dspy.Predict(Sig), trainset=examples())
+ assert isinstance(program, dspy.Predict)
+ assert program.fields["match"]["threshold"] == 0.8
+ assert optimizer.report["train_score_before"] == 0.5556 and optimizer.report["train_score"] == 0.7778
+ assert optimizer.report["fitted"][0]["parameter"] == "threshold"
+ assert "val_score" not in optimizer.report
+
+
+def test_the_validation_set_is_scored_and_never_fitted_on():
+ optimizer = ReAnchor(metric, num_threads=2)
+ val = examples("v-")[:10] # same pairs only, where the default threshold is already right
+ program = optimizer.compile(dspy.Predict(Sig), trainset=examples(), valset=val)
+ assert program.fields["match"]["threshold"] == 0.8
+ assert optimizer.report["val_score_before"] == 1.0 and optimizer.report["val_score"] == 0.6
+
+
+def test_compile_leaves_the_student_unchanged_and_marks_the_program_compiled():
+ student = dspy.Predict(Sig)
+ before = copy.deepcopy(student.fields)
+ program = ReAnchor(metric, num_threads=2).compile(student, trainset=examples())
+ assert student.fields == before
+ assert program._compiled and program is not student
+
+
+def test_a_metric_returning_a_prediction_is_read_by_its_score():
+ graded = lambda gold, pred, trace=None: dspy.Prediction(score=metric(gold, pred), feedback="") # noqa: E731
+ program = ReAnchor(graded, num_threads=2).compile(dspy.Predict(Sig), trainset=examples())
+ assert program.fields["match"]["threshold"] == 0.8
+
+
+def test_a_module_holding_predictors_gets_each_one_calibrated():
+ class Wrapper(dspy.Module):
+ def __init__(self):
+ super().__init__()
+ self.judge = dspy.Predict(Sig)
+ self.judges = [dspy.Predict(Sig)]
+
+ def forward(self, pair):
+ return self.judges[0](pair=pair) if self.judge(pair=pair).match else dspy.Prediction(match=False)
+
+ program = ReAnchor(metric, num_threads=2).compile(Wrapper(), trainset=examples())
+ assert isinstance(program, Wrapper)
+ assert program.judge.fields["match"]["threshold"] == 0.8
+ assert program.judges[0].fields == {} # No further gain after calibrating the first judge.
+
+
+class Items(dspy.Module):
+ def __init__(self):
+ super().__init__()
+ self.judge = dspy.Predict(Sig)
+
+ def forward(self, pairs):
+ return dspy.Prediction(matches=[self.judge(pair=p).match for p in pairs])
+
+
+def test_a_predictor_called_once_per_item_is_calibrated_from_the_program_metric():
+ batch = ["same", "different", "different"]
+ trainset = [dspy.Example(pairs=batch, matches=[k.startswith("same") for k in batch]).with_inputs("pairs")] * 4
+
+ def items_metric(gold, pred, trace=None):
+ return sum(a == b for a, b in zip(pred.matches, gold.matches, strict=True)) / len(gold.matches)
+
+ program = ReAnchor(items_metric, num_threads=2).compile(Items(), trainset=trainset)
+ assert isinstance(program, Items) and program.judge.fields["match"]["threshold"] == 0.8
+
+
+def test_the_program_keeps_a_bound_client_and_callbacks(system_one):
+ client = system_one(leaning)
+ dspy.configure(lm=None)
+ student = dspy.Predict(Sig, callbacks=[])
+ student.set_lm(client)
+ program = ReAnchor(metric, num_threads=2).compile(student, trainset=examples())
+ assert program.lm is not None and program(pair="same").match is True
+
+
+def test_log_dir_holds_the_report(tmp_path):
+ ReAnchor(metric, num_threads=2, log_dir=tmp_path).compile(dspy.Predict(Sig), trainset=examples())
+ assert json.loads((tmp_path / "report.json").read_text())["train_score"] == 0.7778
+ assert {path.name for path in tmp_path.iterdir()} == {"report.json"}
+
+
+def test_compile_logs_each_stage():
+ lines = []
+ handler = logging.Handler()
+ handler.emit = lambda record: lines.append(record.getMessage())
+ log = logging.getLogger("dspy.teleprompt.reanchor.reanchor")
+ log.addHandler(handler)
+ try:
+ ReAnchor(metric, num_threads=2).compile(dspy.Predict(Sig), trainset=examples(), valset=examples("v-"))
+ finally:
+ log.removeHandler(handler)
+ assert lines == [
+ "answering 18 training examples",
+ "fitting thresholds, cuts, and weights",
+ "calibrated: train 0.5556 -> 0.7778",
+ "validation: 0.5556 -> 0.7778",
+ ]
+
+
+def test_empty_trainset_fails():
+ with pytest.raises(ValueError, match="trainset"):
+ ReAnchor(metric).compile(dspy.Predict(Sig), trainset=[])
+
+
[email protected]("keyword", ["teacher", "unsupported"])
+def test_unsupported_compile_arguments_are_not_silently_ignored(keyword):
+ with pytest.raises(TypeError, match="unexpected keyword argument"):
+ ReAnchor(metric).compile(dspy.Predict(Sig), trainset=examples(), **{keyword: None})
+
+
+def test_a_student_without_a_decision_output_fails():
+ with pytest.raises(ValueError, match="decision output"):
+ ReAnchor(metric).compile(dspy.Predict("pair -> answer: str"), trainset=examples())
+
+
+def test_an_uncached_client_fails_unless_the_cache_is_waived(system_one):
+ system_one(leaning, cache=False)
+ with pytest.raises(ValueError, match="require_cache=False"):
+ ReAnchor(metric).compile(dspy.Predict(Sig), trainset=examples())
+ program = ReAnchor(metric, num_threads=2, require_cache=False).compile(dspy.Predict(Sig), trainset=examples())
+ assert program.fields["match"]["threshold"] == 0.8
+
+
+def test_a_predictor_config_that_turns_the_cache_off_fails(system_one):
+ with pytest.raises(ValueError, match="require_cache=False"):
+ ReAnchor(metric).compile(dspy.Predict(Sig, cache=False), trainset=examples())
+
+
+def generative(inputs, evidence):
+ """The LM answers True on every pair when asked for a bool. Asked for probabilities, it leans
+ the same way as the System One client."""
+ if not evidence:
+ return {"match": True}
+ pair = inputs["pair"]
+ return {"match": noul(0.7 if pair.endswith("same-tricky") else 0.9 if "same" in pair else 0.7)}
+
+
+def test_a_native_bool_on_a_generative_lm_is_promoted_when_probabilities_score_better():
+ dspy.configure(lm=ComputedLM(generative, adapter=dspy.JSONAdapter()))
+ student = dspy.Predict(Sig)
+ assert student.fields == {}
+ optimizer = ReAnchor(metric, num_threads=2, require_cache=False)
+ program = optimizer.compile(student, trainset=examples())
+ assert student.fields == {}
+ assert program.fields == {"match": {"threshold": 0.8}}
+ row = optimizer.report["fitted"][0]
+ assert "skipped" not in row and row["train_score_original"] == 0.5556 and row["train_score"] == 0.7778
+
+
[email protected]("accurate_probabilities", [True, False])
+def test_a_native_bool_stays_native_when_probabilities_do_not_score_better(accurate_probabilities):
+ def right(inputs, evidence):
+ answer = inputs["pair"].startswith("same")
+ probability = (0.9 if answer else 0.1) if accurate_probabilities else 0.9
+ return {"match": noul(probability)} if evidence else {"match": answer}
+
+ dspy.configure(lm=ComputedLM(right, adapter=dspy.JSONAdapter()))
+ optimizer = ReAnchor(metric, num_threads=2, require_cache=False)
+ program = optimizer.compile(dspy.Predict(Sig), trainset=examples())
+ assert program.fields == {}
+ row = optimizer.report["fitted"][0]
+ assert row["skipped"] == "fitted behavior did not beat the original"
+ assert row["train_score"] == row["train_score_original"] == 1.0
+ assert row["train_score_at_start"] == (1.0 if accurate_probabilities else 0.5556)
+ assert "value" not in row
+
+
+def test_a_native_bool_stays_native_when_probabilities_win_only_one_example():
+ pairs = [f"same{i}" for i in range(20)] + [f"different{i}" for i in range(20)]
+
+ def one_miss(inputs, evidence):
+ answer = inputs["pair"].startswith("same")
+ if evidence:
+ return {"match": noul(0.9 if answer else 0.1)}
+ return {"match": answer and inputs["pair"] != "same0"}
+
+ dspy.configure(lm=ComputedLM(one_miss, adapter=dspy.JSONAdapter()))
+ train = [dspy.Example(pair=p, match=p.startswith("same")).with_inputs("pair") for p in pairs]
+ optimizer = ReAnchor(metric, num_threads=2, require_cache=False)
+ program = optimizer.compile(dspy.Predict(Sig), trainset=train)
+ assert program.fields == {}
+ assert optimizer.report["fitted"][0]["skipped"] == "fitted behavior did not beat the original"
+
+
[email protected]("binding", ["call", "context"])
[email protected]("backend", ["jev", "lm"])
+def test_runtime_clients_can_calibrate_without_static_cache_checks(jev, binding, backend):
+ client = jev if backend == "jev" else ComputedLM(generative, adapter=dspy.JSONAdapter())
+
+ class Routed(dspy.Module):
+ def __init__(self):
+ super().__init__()
+ self.judge = dspy.Predict(Sig)
+
+ def forward(self, pair):
+ if binding == "call":
+ return self.judge(pair=pair, lm=client)
+ with dspy.context(lm=client):
+ return self.judge(pair=pair)
+
+ with dspy.context(lm=None):
+ with pytest.raises(ValueError, match="require_cache=False"):
+ ReAnchor(metric).compile(Routed(), trainset=examples())
+ optimizer = ReAnchor(metric, num_threads=2, require_cache=False)
+ program = optimizer.compile(Routed(), trainset=examples())
+ assert program.judge.fields == {"match": {"threshold": 0.8}}
+ assert program(pair="different").match is False
+ assert optimizer.report["train_score"] == 0.7778
+
+
[email protected]("original", [None, {}, {"criteria": {"true": {"what": "same item"}}}])
+def test_rejected_jev_configuration_is_restored_exactly(original):
+ student = dspy.Predict(Sig)
+ if original is not None:
+ student.fields["match"] = copy.deepcopy(original)
+ # All True is already perfect; fitting must not leave materialized defaults behind.
+ train = [dspy.Example(pair="same", match=True).with_inputs("pair") for _ in range(10)]
+ optimizer = ReAnchor(metric, num_threads=2)
+ program = optimizer.compile(student, trainset=train)
+ assert program.fields == student.fields
+ row = optimizer.report["fitted"][0]
+ assert row["skipped"] == "fitted behavior did not beat the original"
+ assert row["train_score_original"] == row["train_score"] == 1.0
+ assert "value" not in row
+
+
+def test_a_partial_field_entry_is_filled_from_the_type_defaults(system_one):
+ student = dspy.Predict(Sig)
+ student.set_criteria("match", {"true": "The same item.", "false": "Two different items."})
+ program = ReAnchor(metric, num_threads=2).compile(student, trainset=examples())
+ assert program.fields["match"] == {
+ "criteria": {"true": "The same item.", "false": "Two different items."},
+ "threshold": 0.8,
+ }
+
+
+def test_the_calibrated_program_saves_and_loads(tmp_path):
+ program = ReAnchor(metric, num_threads=2).compile(dspy.Predict(Sig), trainset=examples())
+ program.save(tmp_path / "program.json")
+ restored = dspy.Predict(Sig)
+ restored.load(tmp_path / "program.json")
+ assert restored.fields == {"match": {"threshold": 0.8}}
diff --git a/tests/teleprompt/reanchor/test_reanchor_calibrate.py b/tests/teleprompt/reanchor/test_reanchor_calibrate.py
new file mode 100644
index 000000000..85149b184
--- /dev/null
+++ b/tests/teleprompt/reanchor/test_reanchor_calibrate.py
@@ -0,0 +1,399 @@
+"""Calibration fits each predictor's thresholds, cuts, and weights against the metric, in place."""
+
+import math
+from typing import Annotated, Literal
+
+import pytest
+
+import dspy
+from dspy.experimental import Choice, Score
+from dspy.teleprompt.reanchor.calibrate import calibrate, predictors
+from tests.teleprompt.reanchor.fakes import choice, noul, score
+
+Rating = Score["low", "mid", "high"]
+
+
+class Match(dspy.Signature):
+ pair: str = dspy.InputField()
+ match: bool = dspy.OutputField(desc="Are the two the same?")
+
+
+def test_a_leaning_noul_gets_a_threshold_between_its_piles(system_one):
+ system_one(lambda state, name, q: noul(0.9 if state["inputs"]["pair"] == "same" else 0.7))
+ program = dspy.Predict(Match)
+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 4]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)
+ assert program.fields["match"]["threshold"] == 0.8
+ assert report[0]["train_score"] == 1.0
+
+
[email protected]("lower", [0.0, 0.7, math.nextafter(0.7, 1.0), math.nextafter(1.0, 0.0)])
+def test_adjacent_probabilities_can_be_separated_by_a_boolean_threshold(system_one, lower):
+ upper = math.nextafter(lower, 1.0)
+ system_one(lambda state, name, q: noul(upper if state["inputs"]["pair"] == "same" else lower))
+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 10]
+ optimizer = dspy.experimental.ReAnchor(lambda g, p: float(p.match == g.match), num_threads=2)
+ program = optimizer.compile(dspy.Predict(Match), trainset=train)
+ assert program.fields["match"]["threshold"] == upper
+ assert program(pair="same").match is True
+ assert program(pair="different").match is False
+ assert optimizer.report["train_score_before"] == 0.5
+ assert optimizer.report["train_score"] == 1.0
+
+
[email protected]("target", [True, False])
+def test_zero_probability_can_be_classified_on_either_side(system_one, target):
+ system_one(lambda state, name, q: noul(0.0))
+ program = dspy.Predict(Match)
+ train = [dspy.Example(pair=str(i), match=target).with_inputs("pair") for i in range(10)]
+ report = calibrate(program, train, lambda g, p: float(p.match == g.match), num_threads=2)
+ assert program.fields == ({"match": {"threshold": 0.0}} if target else {})
+ assert report[0]["train_score"] == 1.0
+
+
+def test_a_starting_threshold_stays_unless_a_candidate_scores_strictly_better(system_one):
+ # 0.73 already splits 0.74 from 0.72, so the candidate in that gap only ties.
+ system_one(lambda state, name, q: noul(0.74 if state["inputs"]["pair"] == "same" else 0.72))
+ program = dspy.Predict(Match)
+ program.fields["match"] = {"threshold": 0.73}
+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 4]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)
+ assert program.fields["match"]["threshold"] == 0.73
+ assert report[0]["train_score_at_start"] == 1.0 and report[0]["train_score"] == 1.0
+
+
+class Kind(dspy.Signature):
+ item: str = dspy.InputField()
+ kind: Literal["x", "y"] = dspy.OutputField(desc="Which kind?")
+
+
+def test_choice_multipliers_move_an_overpicked_option(system_one):
+ # Jev leans to "x": 0.6 on an "x" item and 0.55 on a "y" item.
+ system_one(
+ lambda state, name, q: choice(
+ {"x": 0.6, "y": 0.4} if state["inputs"]["item"] == "x" else {"x": 0.55, "y": 0.45}
+ )
+ )
+ program = dspy.Predict(Kind)
+ train = [dspy.Example(item=k, kind=k).with_inputs("item") for k in ["x", "y"] * 4]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.kind == g.kind), num_threads=2)
+ assert report[0]["train_score"] == 1.0
+ w = program.fields["kind"]["weights"]
+ assert 0.55 * w["x"] < 0.45 * w["y"] and 0.6 * w["x"] > 0.4 * w["y"]
+
+
+# Jev leans high: a low item still puts half its mass on the top level (mean index 1.3).
+LEANING = {"lo": {0: 0.2, 1: 0.3, 2: 0.5}, "hi": {0: 0.0, 1: 0.2, 2: 0.8}}
+
+
+class RateLevel(dspy.Signature):
+ item: str = dspy.InputField()
+ rating: Rating = dspy.OutputField(desc="How good is it?")
+
+
+def test_score_cuts_move_to_fit_a_metric_on_the_level(system_one):
+ system_one(lambda state, name, q: score(LEANING[state["inputs"]["item"]]))
+ program = dspy.Predict(RateLevel)
+ train = [dspy.Example(item=k, rating={"lo": 0, "hi": 2}[k]).with_inputs("item") for k in ["lo", "hi"] * 3]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.rating.level == g.rating), num_threads=2)
+ cuts = program.fields["rating"]["cuts"]
+ assert report[0]["parameter"] == "cuts"
+ assert report[0]["train_score_at_start"] == 0.5 and report[0]["train_score"] == 1.0
+ assert 1.3 < cuts[0] < cuts[1] <= 1.8
+
+
+def test_score_cuts_stay_at_their_defaults_when_the_metric_ignores_the_level(system_one):
+ system_one(lambda state, name, q: score(LEANING[state["inputs"]["item"]]))
+ program = dspy.Predict(RateLevel)
+ train = [dspy.Example(item=k, rating={"lo": 0, "hi": 2}[k]).with_inputs("item") for k in ["lo", "hi"]]
+ report = calibrate(program, train, lambda g, p, trace=None: 1 - abs(p.rating.value - g.rating) / 2, num_threads=2)
+ assert program.fields == {}
+ assert report[0]["train_score"] == report[0]["train_score_at_start"]
+
+
+def test_score_search_does_not_collapse_cuts_at_adjacent_floats(system_one):
+ # The midpoint between 0.7 and this observed mean rounds back to 0.7.
+ probability = 0.7000000000000001
+ system_one(lambda state, name, q: score({0: 1 - probability, 1: probability, 2: 0.0}))
+ program = dspy.Predict(RateLevel)
+ program.fields["rating"] = {"cuts": [0.7, 1.5]}
+ train = [dspy.Example(item=str(i), rating=1).with_inputs("item") for i in range(6)]
+ report = calibrate(program, train, lambda g, p: float(p.rating.level == g.rating), num_threads=2)
+ assert program.fields["rating"] == {"cuts": [0.7, 1.5]}
+ assert report[0]["train_score"] == 1.0
+
+
+class Route(dspy.Signature):
+ ticket: str = dspy.InputField()
+ queue: Annotated[Literal["billing", "tech"], Choice[("billing", "Payments"), ("tech", "Bugs")]] = dspy.OutputField(
+ desc="Which queue?"
+ )
+
+
+def test_a_native_literal_with_choice_criteria_gets_weights_and_stays_native(system_one):
+ system_one(lambda state, name, q: choice({"billing": 0.6, "tech": 0.4}))
+ program = dspy.Predict(Route)
+ train = [dspy.Example(ticket=str(i), queue="tech" if i % 3 else "billing").with_inputs("ticket") for i in range(6)]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.queue == g.queue), num_threads=2)
+ assert report[0]["parameter"] == "weights" and report[0]["train_score"] == round(4 / 6, 4)
+ assert program(ticket="0").queue == "tech"
+
+
+def test_predictors_finds_nested_and_listed_predictors():
+ class Wrapper(dspy.Module):
+ def __init__(self):
+ super().__init__()
+ self.judge = dspy.Predict(Match)
+ self.judges = [dspy.Predict(Kind)]
+ self.writer = dspy.Predict("pair -> note")
+
+ assert [name for name, _ in predictors(Wrapper())] == ["judge", "judges[0]"]
+ assert [name for name, _ in predictors(dspy.Predict(Match))] == ["self"]
+
+
[email protected]("threads", [1, 4])
+def test_calibration_leaves_the_request_unchanged(system_one, threads):
+ client = system_one(lambda state, name, q: noul(0.9 if state["inputs"]["pair"] == "same" else 0.7))
+ program = dspy.Predict(Match)
+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"]]
+ calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=threads)
+ assert (
+ len({repr(c) for c in client.calls}) == 2
+ ) # one distinct request per example, however many settings were tried
+
+
+def test_a_metric_error_fails_the_pass(system_one):
+ system_one(lambda state, name, q: noul(0.9))
+
+ def broken(example, prediction):
+ raise RuntimeError("metric failed")
+
+ train = [dspy.Example(pair="same", match=True).with_inputs("pair")]
+ with pytest.raises(Exception, match="Execution cancelled"):
+ calibrate(dspy.Predict(Match), train, broken, num_threads=1)
+
+
[email protected]("value", [float("nan"), float("inf")])
+def test_a_nonfinite_metric_fails(system_one, value):
+ system_one(lambda state, name, q: noul(0.9))
+ train = [dspy.Example(pair="same", match=True).with_inputs("pair")]
+ with pytest.raises(ValueError, match="finite metric"):
+ calibrate(dspy.Predict(Match), train, lambda g, p, trace=None: value, num_threads=1)
+
+
+class TwoFlags(dspy.Signature):
+ pair: str = dspy.InputField()
+ match: bool = dspy.OutputField(desc="Are the two the same?")
+ noise: bool = dspy.OutputField(desc="Is it noisy?")
+ kind: Literal["x", "y"] = dspy.OutputField(desc="Which kind?")
+
+
+def two_flags(state, name, q):
+ if name == "kind":
+ return choice({"x": 0.6, "y": 0.4})
+ return noul(0.9 if state["inputs"]["pair"] == "same" else 0.7)
+
+
+def test_outputs_the_metric_does_not_read_keep_their_numeric_settings(system_one):
+ client = system_one(two_flags)
+ program = dspy.Predict(TwoFlags)
+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 4]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)
+ rows = {r["field"]: r for r in report}
+ assert rows["noise"]["train_score"] == rows["noise"]["train_score_at_start"]
+ assert rows["kind"]["train_score"] == rows["kind"]["train_score_at_start"]
+ assert "skipped" not in rows["match"] and program.fields["match"]["threshold"] == 0.8
+ assert "noise" not in program.fields
+ assert "kind" not in program.fields
+ assert len({repr(c) for c in client.calls}) == 2
+
+
+def test_balanced_labels_are_not_mistaken_for_an_ignored_output(system_one):
+ # All True and all False both score 0.5 on average; the per-example scores still differ.
+ system_one(lambda state, name, q: noul(0.9 if state["inputs"]["pair"] == "same" else 0.7))
+ program = dspy.Predict(Match)
+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 2]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)
+ assert "skipped" not in report[0] and program.fields["match"]["threshold"] == 0.8
+
+
+def test_equal_scores_at_extremes_do_not_hide_a_useful_threshold(system_one):
+ system_one(lambda state, name, q: noul(0.4 if state["inputs"]["pair"] == "same" else 0.2))
+
+ class Batch(dspy.Module):
+ def __init__(self):
+ super().__init__()
+ self.judge = dspy.Predict(Match)
+
+ def forward(self, pairs):
+ return dspy.Prediction(matches=[self.judge(pair=p).match for p in pairs])
+
+ program = Batch()
+ train = [dspy.Example(pairs=["same", "different"], matches=[True, False]).with_inputs("pairs") for _ in range(10)]
+ # Default, all-True, and all-False decisions all fail exact match. Only the interior split wins.
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.matches == g.matches), num_threads=2)
+ assert program.judge.fields["match"]["threshold"] == pytest.approx(0.3)
+ assert report[0]["train_score_at_start"] == 0.0
+ assert report[0]["train_score"] == 1.0
+
+
+def test_probabilities_bunched_near_one_get_a_threshold_between_them(system_one):
+ # Every P(True) is 0.98 or 1.0, above any fixed grid; the gap between them splits the labels.
+ system_one(lambda state, name, q: noul(1.0 if state["inputs"]["pair"] == "same" else 0.98))
+ program = dspy.Predict(Match)
+ train = [dspy.Example(pair=k, match=k == "same").with_inputs("pair") for k in ["same", "different"] * 4]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)
+ assert program.fields["match"]["threshold"] == 0.99
+ assert report[0]["train_score"] == 1.0
+ assert report[0]["observed"] == {"calls": 8, "distinct": 2, "min": 0.98, "max": 1.0, "candidates": 2}
+
+
+def test_a_choice_multiplier_can_leave_the_range_a_fixed_grid_would_try(system_one):
+ # A "y" item still puts 0.95 on "x"; separating the items needs a "y" multiplier above 19.
+ system_one(
+ lambda state, name, q: choice(
+ {"x": 0.99, "y": 0.01} if state["inputs"]["item"] == "x" else {"x": 0.95, "y": 0.05}
+ )
+ )
+ program = dspy.Predict(Kind)
+ train = [dspy.Example(item=k, kind=k).with_inputs("item") for k in ["x", "y"] * 4]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.kind == g.kind), num_threads=2)
+ w = program.fields["kind"]["weights"]
+ assert report[0]["train_score"] == 1.0
+ assert 0.95 * w["x"] < 0.05 * w["y"] and 0.99 * w["x"] > 0.01 * w["y"]
+
+
+def test_many_distinct_probabilities_are_thinned_to_the_candidate_cap(system_one):
+ from dspy.teleprompt.reanchor.calibrate import MAX_CANDIDATES
+
+ system_one(lambda state, name, q: noul(int(state["inputs"]["pair"]) / 200))
+ program = dspy.Predict(Match)
+ train = [dspy.Example(pair=str(i), match=i >= 120).with_inputs("pair") for i in range(200)]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=4)
+ assert report[0]["observed"]["distinct"] == 200
+ assert report[0]["observed"]["candidates"] <= MAX_CANDIDATES + 1 # Gap midpoints plus the zero endpoint.
+ assert 0.55 < program.fields["match"]["threshold"] <= 0.6 and report[0]["train_score"] >= 0.97
+
+
+def test_evidence_is_recorded_for_the_predictor_that_decoded_it(system_one):
+ from dspy.adapters.decision import record_evidence
+
+ system_one(lambda state, name, q: noul(0.9) if name == "match" else choice({"x": 0.6, "y": 0.4}))
+
+ class Two(dspy.Module):
+ def __init__(self):
+ super().__init__()
+ self.judge = dspy.Predict(Match)
+ self.kind = dspy.Predict(Kind)
+
+ def forward(self, items):
+ return [self.judge(pair=i).match for i in items], self.kind(item="a").kind
+
+ program = Two()
+ with record_evidence() as log:
+ program(items=["a", "b", "c"])
+ program(items=["d"])
+ assert [(caller is program.judge, name) for caller, name, _ in log].count((True, "match")) == 3
+ caller, name, evidence = log[-1]
+ assert caller is program.kind and name == "kind" and evidence["probabilities"] == {"x": 0.6, "y": 0.4}
+ assert len(log) == 4
+
+
+def test_concurrent_evidence_collectors_are_isolated_and_reach_dspy_workers(system_one):
+ from concurrent.futures import ThreadPoolExecutor
+ from threading import Barrier
+
+ from dspy.adapters.decision import record_evidence
+ from dspy.utils.parallelizer import ParallelExecutor
+
+ system_one(lambda state, name, q: noul(0.2 if state["inputs"]["pair"] == "a" else 0.8))
+ barrier = Barrier(2)
+
+ def collect(pair):
+ predict = dspy.Predict(Match)
+ with record_evidence() as log:
+ barrier.wait(timeout=10)
+ ParallelExecutor(num_threads=2, disable_progress_bar=True).execute(lambda _: predict(pair=pair), range(2))
+ barrier.wait(timeout=10)
+ return predict, log
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ results = list(executor.map(collect, ["a", "b"]))
+ for (predict, log), probability in zip(results, [0.2, 0.8], strict=True):
+ assert len(log) == 2
+ assert all(
+ caller is predict and name == "match" and evidence == {"noul": probability}
+ for caller, name, evidence in log
+ )
+
+
+def test_nested_evidence_collector_restores_outer_context_after_error(system_one):
+ from dspy.adapters.decision import record_evidence
+
+ system_one(lambda state, name, q: noul(float(state["inputs"]["pair"])))
+ predict = dspy.Predict(Match)
+ with record_evidence() as outer:
+ predict(pair="0.1")
+ with pytest.raises(RuntimeError, match="stop"):
+ with record_evidence() as inner:
+ predict(pair="0.2")
+ raise RuntimeError("stop")
+ predict(pair="0.3")
+ predict(pair="0.4")
+ assert [evidence["noul"] for _, _, evidence in outer] == [0.1, 0.3]
+ assert [evidence["noul"] for _, _, evidence in inner] == [0.2]
+ assert dspy.settings.get("_decision_evidence") is None
+
+
+def lone_outlier(same: float, different: float, odd: float, count: int = 1):
+ """20 same pairs, 19 different pairs, and `count` of the same pairs replaced by `odd` ones."""
+ pairs = (
+ [f"same{i}" for i in range(20 - count)] + [f"odd{i}" for i in range(count)] + [f"diff{i}" for i in range(19)]
+ )
+
+ def answer(state, name, q):
+ pair = state["inputs"]["pair"]
+ return noul(odd if pair.startswith("odd") else same if pair.startswith("same") else different)
+
+ train = [dspy.Example(pair=p, match=not p.startswith("diff")).with_inputs("pair") for p in pairs]
+ return answer, train
+
+
+def test_a_threshold_stays_when_its_whole_gain_is_one_example(system_one):
+ answer, train = lone_outlier(0.9, 0.1, 0.3)
+ system_one(answer)
+ program = dspy.Predict(Match)
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)
+ assert program.fields == {}
+ assert report[0]["fold_check"] == {"passed": 0, "failed": 1}
+
+
+def test_a_threshold_moves_when_its_gain_recurs_across_the_training_set(system_one):
+ answer, train = lone_outlier(0.9, 0.1, 0.3, count=6)
+ system_one(answer)
+ program = dspy.Predict(Match)
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.match == g.match), num_threads=2)
+ assert program.fields["match"]["threshold"] == 0.2
+ assert report[0]["fold_check"] == {"passed": 1, "failed": 0}
+
+
+def test_score_cuts_stay_when_their_whole_gain_is_one_example(system_one):
+ levels = {"lo": {0: 0.8, 1: 0.1, 2: 0.1}, "hi": {0: 0.1, 1: 0.0, 2: 0.9}, "odd": {0: 0.5, 1: 0.3, 2: 0.2}}
+ system_one(lambda state, name, q: score(levels[state["inputs"]["item"].rstrip("0123456789")]))
+ program = dspy.Predict(RateLevel)
+ items = [f"lo{i}" for i in range(19)] + ["odd0"] + [f"hi{i}" for i in range(20)]
+ train = [dspy.Example(item=k, rating=2 if k.startswith("hi") else 0).with_inputs("item") for k in items]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.rating.level == g.rating), num_threads=2)
+ assert program.fields == {}
+ assert report[0]["fold_check"]["failed"] >= 1
+
+
+def test_choice_weights_stay_when_their_whole_gain_is_one_example(system_one):
+ leans = {"x": {"x": 0.8, "y": 0.2}, "y": {"x": 0.2, "y": 0.8}, "odd": {"x": 0.55, "y": 0.45}}
+ system_one(lambda state, name, q: choice(leans[state["inputs"]["item"].rstrip("0123456789")]))
+ program = dspy.Predict(Kind)
+ items = [f"x{i}" for i in range(20)] + [f"y{i}" for i in range(19)] + ["odd0"]
+ train = [dspy.Example(item=k, kind="x" if k.startswith("x") else "y").with_inputs("item") for k in items]
+ report = calibrate(program, train, lambda g, p, trace=None: float(p.kind == g.kind), num_threads=2)
+ assert program.fields == {}
+ assert report[0]["fold_check"]["failed"] >= 1