Skip to content

An AI-powered search engine with a generative UI

10.3 KB
Raw Blame

Evals

Measured checks on model behaviour, kept separate from __tests__ because they call real models, cost money, and produce rates rather than pass/fail.

Layers

Layer Where Runs Needs keys
Evaluator unit tests evals/lib/__tests__ every PR, via bun run test no
Experiments evals/run-*.ts on prompt and model changes, and on demand yes

The first layer exists because an evaluator is code too. analyzeRelatedQuestions is tested against recorded spec blocks so a broken evaluator cannot quietly report a healthy rate. Those tests are part of the normal suite and run on fork PRs, where secrets are unavailable.

related-questions

bun run eval:related-questions                              # deployed adaptive model, 1 trial per item
bun run eval:related-questions -- --mode quick              # the quick search mode instead
bun run eval:related-questions -- --trials 3                # 3 generations per item
bun run eval:related-questions -- --model openai:some-model # candidate model
bun run eval:related-questions -- --items fuji-routes       # one item, for debugging
bun run eval:related-questions -- --help

Answers 22 fixed queries (10 substantive, 12 trivial) against pinned search results and reports:

  • emissionRate: substantive answers that carried a related questions block
  • falsePositiveRate: trivial answers that carried one anyway
  • wellFormedRate: emitted blocks that satisfy the spec contract (one block, three buttons, submitQuery, button text identical to the query)

Emission and well-formedness are separate metrics on purpose. A model that stops emitting and a model that emits a broken block are different failures with different fixes.

Search modes

--mode selects which deployed configuration is replayed. The two differ in more than wording: quick mode runs a different system prompt, a smaller tool set, and its own providerOptions, so a result measured on one mode says nothing about the other. The flag defaults to adaptive, which is only a default for a bare local run; CI runs both modes as a matrix, because a prompt or model change can move one and leave the other alone. That is the whole reason the modes are separable here.

Both modes are gated against the same thresholds.json entry. There is no per-mode threshold because nothing yet needs one; add one when a mode's healthy range is genuinely different rather than to make a run go green.

Why it is built this way

  • Search results are fixed. Live search would put retrieval variance into a measurement about the prompt and the model.
  • The task mirrors production. Same prompt builder, same providerOptions from config/models/cloud.json, same toModelOutput trimming. Reasoning effort has moved emission rates before, so a run without provider options would measure something the product does not do.
  • BRAVE_SEARCH_API_KEY is pinned by the runner. The adaptive prompt branches on whether a general search provider is configured, so an unset key would silently evaluate a different prompt. No request reaches Brave; the search tool is a fixture.
  • Each trial is its own item. Emission is probabilistic, so one generation per query measures almost nothing. --trials repeats each query while keeping one item equal to one generation in Langfuse.

What counts as trivial

An item is trivial only when the fixture holds exactly one unambiguous value answering it and no comparison or procedure is implied. "How tall is Mount Fuji" qualifies. "What is the typical energy density of an LFP pack" does not, because the fixture puts LFP and NMC side by side and the answer naturally becomes a comparison.

The distinction matters because the label is set by the shape of the question while the model decides by the shape of its own answer. A loosely trivial question draws out a structured answer, and the block that follows it is arguably correct, which inflates falsePositiveRate with disagreement rather than defects. Questions whose answers expand belong in the substantive set.

source-cards

bun run eval:source-cards                       # adaptive, 1 trial per item
bun run eval:source-cards -- --mode quick --trials 3

Answers 16 fixed queries (8 listing, 8 other) and reports:

  • emissionRate: listing answers ("find", "recommend", "where can I get") that presented the items as linked cards (Card with href inside a Grid)
  • misuseRate: other answers (explanation, steps, comparison, single fact, writing) that carried cards anyway
  • wellFormedRate: emitted blocks with 2 to 8 cards, each with a title Text and a unique web href
  • groundedRate: cards whose URL is a fixture search result URL, verbatim
  • duplicatedRate: answers that also repeated the cards as a markdown list

Fixture URLs use real-looking hosts. With .example hosts the adaptive model correctly judged the results to be placeholders and declined to present them, which measured the fixture instead of the prompt.

draft-edits

bun run eval:draft-edits                     # synthetic chats, plus local production seeds if present
bun run eval:draft-edits -- --trials 3
bun run eval:draft-edits -- --no-synthetic   # production seeds only

Replays one turn in a chat that already holds a draft. Each item carries the chat so far and goes through the same history preparation as the chat route (draft compaction, then <current_draft> attached to the latest user message), with its own mode's prompt, tools, and providerOptions. The real editDraft tool runs against the rebuilt draft, so a call is recorded exactly as production would apply it. Reports:

  • hijackRate: unrelated requests (a question, a keyword list) where editDraft was called at all
  • newBlockRate: requests for a new piece on another topic answered with a new complete draft block and no editDraft call
  • editSuccessRate: requests to change the draft that changed it, either through an applied edit or a new complete block

The committed dataset is synthetic. Production turns, where the hijack was first observed, hold users' own text, so they are built into evals/_local/draft-edits-seeds.json, which is gitignored and loaded only when present. Output for production items is never printed, only ids and verdicts.

Thresholds

thresholds.json gates CI. A null value means the metric is reported but not enforced, which is the state a new metric starts in.

minSamples is the smallest denominator a threshold may be enforced on, and the sampled rates set it at 50, which the five trials CI runs reach and a local run does not. That is not caution for its own sake: identical code measured 1.000 and 0.833 on emissionRate across two runs of thirty substantive samples, so a three-trial run cannot carry a ceiling or a floor. completionRate declares no minimum because it counts items that actually failed rather than sampling a probability.

completionRate guards the measurement itself: the experiment runner drops an item whose task threw, which shrinks the denominator of every other rate. A run that lost more than a tenth of its items is not a valid measurement.

Rates are read at the run level, never per item. At three trials an individual query swings by a third from one run to the next, so a single query moving is noise; raise --trials before drawing a conclusion about one of them. Whole runs move too: the same code measured 0.28 and 0.36 on falsePositiveRate in consecutive runs, which is why CI runs five trials and why a before/after comparison needs more than one run on each side.

Set a baseline before enforcing anything. Run the eval a few times on the current production model, then commit the numbers. Committing them rather than comparing against the previous Langfuse run is deliberate: a threshold that gets loosened should appear in a diff with an author and a reason.

A violation throws RegressionError, which the Langfuse experiment GitHub Action reads to fail the job.

Be honest about what a single gated run proves. emissionRate separates cleanly: the failure it exists to catch is a model swap that empties the block entirely, and fifty samples settle that. falsePositiveRate does not. Sampled three times before the trivial-skip fix it read 0.200, 0.278 and 0.361, and twice after it read 0.083 and 0.111, so the distributions overlap at the tails and no ceiling separates them on one run. That gate catches a blowup. Deciding that a change moved the metric still takes several runs on each side.

Datasets

evals/datasets/*.json holds hand-written, synthetic data. It is committed, so a prompt change and the data it is measured against show up in the same review.

Anything derived from production traces (thumbs-down turns, errored turns) must not be committed here. This repository is public. Put that data in a Langfuse managed dataset and fetch it at run time:

const dataset = await langfuse.dataset.get('morphic-thumbs-down-regression')

experiment.run accepts local items and Langfuse dataset items interchangeably, so the task and the evaluators do not change when the source does.

Credentials

LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_BASE_URL, plus the API key for the provider under test. The runner loads .env.local.

Keys are required: the experiment runner uploads item scores through the Langfuse client whether or not spans are exported, so a keyless run fails partway rather than degrading to a local-only run. The CI job skips itself with a warning when they are absent, so a repository without them does not collect a red mark on every PR that touches evals/.

Experiment traces are tagged by the SDK with the sdk-experiment environment, so they can be excluded from production metrics without a separate project.

Known quirk

When items are skipped, the Input and Expected lines in the printed item list are misaligned: the Langfuse formatter pairs itemResults[i] with originalData[i] after the failed entries have been filtered out. The scores, the outputs, and the trace links are unaffected, because each result carries its own item. The runner prints a warning whenever this applies.