The Open Framework for Building Agents
---
title: "Assertions"
description: "Scoped methods, value assertions, the matcher mini-language, and gate vs soft severity."
---
Assertions are how an eval grades what its `test(t)` function produced. Each one records a result and returns a chainable handle. The runner reads every recorded result to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are three deterministic surfaces: scoped methods, `t.check` for grading a specific value, and `t.score` for recording a measurement you computed yourself. For model-graded assertions, see [Judge](./judge).
## Scoped assertions
Scoped assertions take no explicit value and gate by default. Assertions on `t` inspect the whole run after `test` finishes. Session assertions snapshot that session when called, and turn assertions inspect one immutable response. A scope is **parked** when it paused on unanswered human-in-the-loop (HITL) input.
| Assertion | Asserts |
| ----------------------------------------------------- | ----------------------------------------------------------------------------- |
| `t.succeeded()` | The run did not fail and did not park on unanswered HITL input |
| `t.parked()` | The run cleanly parked on HITL input |
| `t.messageIncludes(token)` | Joined assistant text contains `token` (string or RegExp) |
| `turn.outputEquals(value)` / `.outputMatches(schema)` | Deep equality or Standard Schema validation of turn/session structured output |
| `t.calledTool(name, opts?)` | A matching tool call completed (`input`, `output`, `status`, `count`) |
| `t.loadedSkill(skill, opts?)` | A load of `skill` completed (`output`, `status`, `count`) |
| `t.notCalledTool(name)` | No request for `name` in any lifecycle state |
| `t.toolOrder([...names])` | Tool requests appear in order |
| `t.usedNoTools()` | No tool calls at all; skill loads don't count |
| `t.maxToolCalls(n)` | At most `n` tool calls; skill loads don't count |
| `t.noFailedActions()` | No tool, subagent, or skill action reported a failure |
| `t.calledSubagent(name, opts?)` | A subagent delegation happened (identity, remote, output, status, count) |
| `t.event(type, opts?)` / `t.notEvent(type, opts?)` | Typed event presence, data, and count matching |
| `t.eventOrder([...matchers])` | Matching event groups occur in order |
| `t.eventsSatisfy(label, predicate)` | Escape hatch: any predicate over the typed event stream |
`succeeded()` accepts both a closed session and a healthy session left open for the next user message; it rejects protocol failures and unanswered HITL. `parked()` requires a clean HITL park. Structured output assertions live on turns and independent sessions, where the output is unambiguous (see the [output schema guide](../guides/client/output-schema)).
```ts
await t.send("What is the weather in Brooklyn?");
t.succeeded();
t.calledTool("get_weather");
```
The same vocabulary narrows naturally in multi-turn and externally-created session evals:
```ts
const first = await t.send("Call get_weather for Brooklyn");
first.calledTool("get_weather", { count: 1 });
const attached = await t.target.attachSession(sessionId);
attached.succeeded();
attached.messageIncludes("Sunny");
```
`t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other, never both in the same run.
A call through [`eve__tool`](../concepts/built-in-tools#eve__search-eve__tool-and-eve__skill) is reported under the name and input of the entry it reaches, such as `linear__list_issues`, so assert on that entry; `eve__search` calls count as tool calls in `t.usedNoTools()` and `t.maxToolCalls(n)`, and skill loads don't. The tool-name assertions and `turn.requireToolCall` throw for `eve__tool` and `eve__skill`, whose calls are reported as their entries and as skill loads, and for a retired name such as `load_skill` unless the root agent declares a tool by that name or has a dynamic tool resolver, and say what to assert instead.
## Value assertions with `t.check`
`t.check(value, assertion)` grades an explicit value against a builder from `eve/evals/expect`. The value can be a turn’s `.message`, parsed JSON, or any local you computed:
```ts
import { includes, equals, matches, satisfies, similarity } from "eve/evals/expect";
t.check(turn.message, includes(/sunny/i)); // substring or RegExp (gate)
t.check(parsed, equals({ city: "Brooklyn" })); // deep structural equality (gate)
t.check(parsed, matches(WeatherSchema)); // Standard Schema, e.g. Zod (gate)
t.check(turn.message, similarity("Sunny, 72F")); // fuzzy 0–1 Levenshtein (soft)
t.check(
latencyMs,
satisfies((value) => value < 1_000, "latency under one second"),
);
```
| Builder | Scores | Default |
| ---------------------- | ------------------------------------------------------- | ------- |
| `includes(value)` | coerced string contains a substring or matches a RegExp | gate |
| `equals(value)` | deep structural equality | gate |
| `matches(schema)` | validates against a Standard Schema | gate |
| `similarity(expected)` | normalized Levenshtein similarity, 1 = identical | soft |
| `satisfies(fn, label)` | custom boolean predicate | gate |
For example, a failed labeled equality check prints:
```text
✗ equals [status] (0% < 100%): expected {"status":"active"}; received {"status":"disabled"}
```
Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground. For nuanced grading, reach for the [judge](./judge).
## Scores with `t.score`
`t.score(evaluation)` records a score from 0 to 1. The evaluation is the number or `{ score, message?, metadata? }`, or a promise of either. Chain `.label(key)` to name it; reporters and artifacts expose the label as `key`. Without a label the score reports as `score`, and several unlabeled scores share that name.
```ts
const reply = turn.message ?? "";
t.score({
score: Math.max(0, 1 - reply.length / 400),
metadata: { length: reply.length },
}).label("reply-brevity");
t.score(gradeFaithfulness(reply, reference)).label("faithfulness"); // async
```
On its own a score has no verdict and does not affect the eval's outcome. Chain a rule when it should:
```ts
t.score(gradeFaithfulness(reply, reference)).label("faithfulness").gate(0.9); // fails below 0.9
t.score(brevity).label("reply-brevity").atLeast(0.5); // marks the eval `scored`, fatal under --strict
```
The score is reported as measured. A `0.82` gated at `0.9` is reported as `score: 0.82`, `threshold: 0.9`, `passed: false`. An evaluator that throws or returns a score outside 0 to 1 records an errored outcome with the error message and no score, and fails the eval.
Every [reporter](./reporters#custom-reporters) receives the entries in `onEvalComplete` and can read them by `key`:
```ts
onEvalComplete(result) {
for (const { key, score, threshold, passed } of result.assertions) {
if (key === undefined) continue;
store.write({ evalId: result.id, key, score, threshold, passed });
}
}
```
The same entries land in the [run artifacts](./running#artifacts) under `.eve/evals/`, in `summary.json` and one line per eval in `results.jsonl`:
```json
{
"name": "score [reply-brevity]",
"key": "reply-brevity",
"score": 0.84,
"severity": "soft",
"errored": false
}
```
`threshold` and `passed` are present only when a rule was chained.
## The matcher mini-language
`t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, remoteUrl, output, status, count }`. Calls match `status: "completed"` by default. Tool lifecycle checks also accept `"pending"`, `"failed"`, and `"rejected"`. Subagent checks accept `"working"`, `"completed"`, `"failed"`, and `"cancelled"`. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls.
A subagent call is a task call: it stays `working` until its `task.settled` event arrives, then takes that event's `status` (`completed`, `failed`, or `cancelled`) and, when completed, its `output`. A completed call leaves the child session available for later calls with the same `taskId`.
The runner counts each `task.started` with `kind: "agent"` as a subagent call and takes its status from `task.settled`. `childSessionId` and `remoteUrl` come from the task's `agent.started`, so they are absent for a call that failed before its session opened. A proxied input request alone does not change the subagent call's status; assert that request with `requireInputRequest`.
Matcher values accept a literal (objects partial-deep-match), a RegExp, or a predicate function that returns a boolean:
```ts
t.calledTool("bash", { input: { command: /^pwd/ }, count: 1 });
t.calledTool("echo", { count: (count) => count >= 2 });
t.calledTool("echo", { output: (value) => String(value).includes(marker) });
parked.calledTool("guarded", { status: "pending", count: 1 });
t.calledTool("guarded", { output: /approved/, count: 1 });
t.calledSubagent("weather", {
remoteUrl: (value) => value === process.env.WEATHER_AGENT_URL,
output: /72F/,
});
```
`requireInputRequest` uses the same matcher language for `input`, `prompt`, and `display`. Its `optionIds` matcher receives option ids in request order; a literal array must match that complete ordered list exactly:
```ts
const request = session.requireInputRequest({
toolName: "confirm_deploy",
optionIds: ["approve", "cancel"],
});
```
## Run state and derived facts
Beyond the raw `session.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, lifecycle status), subagent calls, and HITL input requests. A turn that leaves the session open for a next message is the normal end state of a successful turn; parking on unanswered HITL input is tracked separately.
Typed event matching covers presence, absence, numeric or predicate counts, partial event data, and ordering:
```ts
turn.notEvent("result.completed");
turn.eventOrder([
{ type: "task.started", data: { name: "researcher" }, count: 2 },
{ type: "task.settled", data: { status: "completed" }, count: 2 },
]);
```
When a protocol invariant needs cross-event correlation, `eventsSatisfy` remains the escape hatch:
```ts
t.eventsSatisfy("assistant reply includes the marker", (events) =>
events.some((e) => e.type === "message.completed" && e.data.message?.includes(marker)),
);
```
### Assert and score reasoning
When the provider exposes reasoning, eve captures each completed block as a `reasoning.completed` event. Assert on its text with `turn.event(...)`, or join the blocks and pass them to a judge:
```ts
turn.event("reasoning.completed", {
data: { reasoning: /checks? the units/i },
});
const reasoning = turn.events
.filter((event) => event.type === "reasoning.completed")
.map((event) => event.data.reasoning)
.join("\n\n");
t.judge("The reasoning checks its assumptions before concluding.", {
on: reasoning,
}).atLeast(0.8);
```
A provider that does not expose reasoning emits no `reasoning.completed` event, so the `turn.event` gate fails and the judge grades an empty string. Reasoning the provider keeps hidden cannot be graded. The default judge input and `session.transcript` exclude reasoning, which is why the example passes it with `on`.
## Required preconditions
Recorded assertions never throw and are not awaitable. When later control flow depends on a value, use `await t.require(value, assertion)`. It records a gate, returns the original value when it passes, and stops the test body without adding a duplicate execution error when it fails:
```ts
await t.require(
sessionIds,
satisfies((ids) => ids.length > 0, "dispatch started a session"),
);
await t.target.attachSession(sessionIds[0]!);
```
Use the matching `require*` lookups when dependent code needs protocol data:
```ts
const call = turn.requireToolCall("search");
const request = session.requireInputRequest({ toolName: "guarded" });
```
## Severity
Every assertion returns a chainable handle. Severity rides on the assertion, so there is no separate thresholds map to keep in sync.
- `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
- `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold it records no verdict and never fails.
- `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).
- `.label(key)` adds a stable name when an eval records several assertions from the same family. Reporters and artifacts expose it as `key`. Braintrust and Datadog keep one score per name, so `eve eval` warns when soft assertions in one eval share a name with different scores.
The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge(...)` assertion are soft. Annotate only when you deviate:
```ts
t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate
t.check(turn.message, similarity("Sunny")).label("forecast wording").atLeast(0.8);
t.check(turn.message, includes("error")).soft(); // track without failing the build
```
## What to read next
- [Judge](./judge): LLM-graded assertions with thresholds
- [Cases](./cases): where assertions attach
- [Running evals](./running): how verdicts map to exit codes