Skip to content

armgit / letta

Public Imported from GitHub
0

Platform for stateful agents: AI with advanced memory that can learn and self-improve over time.

feat: integrate Letta Evals documentation (#5665)

Cameron Pfiffer237a50255235
diff --git a/fern/docs.yml b/fern/docs.yml
index 37e97adda..bcaf277bd 100644
--- a/fern/docs.yml
+++ b/fern/docs.yml
@@ -50,6 +50,9 @@ tabs:
leaderboard:
display-name: Leaderboard
skip-slug: true
+ evals:
+ display-name: Evals
+ slug: evals
landing-page:
page: home
path: pages/index.mdx
@@ -723,6 +726,76 @@ navigation:
- page: Contributing Results
path: pages/leaderboard/contributing.mdx
+ - tab: evals
+ layout:
+ - link: Chat on Discord
+ icon: fa-brands fa-discord
+ href: https://discord.gg/letta
+ - link: Developer Forum
+ icon: fa-sharp fa-light fa-comments
+ href: https://forum.letta.com
+ - link: DeepLearning.AI Course
+ icon: fa-sharp fa-light fa-building-columns
+ href: https://www.deeplearning.ai/short-courses/llms-as-operating-systems-agent-memory/?utm_campaign=memgpt-launch&utm_content=331638345&utm_medium=social&utm_source=docs&hss_channel=tw-992153930095251456
+ - section: Get Started
+ contents:
+ - page: Overview
+ path: pages/evals/overview.mdx
+ - page: Getting Started
+ path: pages/evals/getting-started.mdx
+ - section: Core Concepts
+ contents:
+ - page: Concepts Overview
+ path: pages/evals/concepts/overview.mdx
+ - page: Suites
+ path: pages/evals/concepts/suites.mdx
+ - page: Datasets
+ path: pages/evals/concepts/datasets.mdx
+ - page: Targets
+ path: pages/evals/concepts/targets.mdx
+ - page: Graders
+ path: pages/evals/concepts/graders.mdx
+ - page: Extractors
+ path: pages/evals/concepts/extractors.mdx
+ - page: Gates
+ path: pages/evals/concepts/gates.mdx
+ - section: Configuration
+ contents:
+ - page: Suite YAML Reference
+ path: pages/evals/configuration/suite-yaml.mdx
+ - section: Graders
+ contents:
+ - page: Tool Graders
+ path: pages/evals/graders/tool-graders.mdx
+ - page: Rubric Graders
+ path: pages/evals/graders/rubric-graders.mdx
+ - page: Multi-Metric Grading
+ path: pages/evals/graders/multi-metric.mdx
+ - section: Extractors
+ contents:
+ - page: Built-in Extractors
+ path: pages/evals/extractors/builtin.mdx
+ - page: Custom Extractors
+ path: pages/evals/extractors/custom.mdx
+ - section: Advanced
+ contents:
+ - page: Custom Graders
+ path: pages/evals/advanced/custom-graders.mdx
+ - page: Multi-Turn Conversations
+ path: pages/evals/advanced/multi-turn-conversations.mdx
+ - section: CLI Reference
+ contents:
+ - page: Commands
+ path: pages/evals/cli/commands.mdx
+ - section: Results & Metrics
+ contents:
+ - page: Understanding Results
+ path: pages/evals/results/overview.mdx
+ - section: Troubleshooting
+ contents:
+ - page: Common Issues
+ path: pages/evals/troubleshooting.mdx
+
# - tab: cookbooks
# layout:
# - section: Cookbooks
diff --git a/fern/pages/evals/advanced/custom-graders.mdx b/fern/pages/evals/advanced/custom-graders.mdx
new file mode 100644
index 000000000..bce7b8938
--- /dev/null
+++ b/fern/pages/evals/advanced/custom-graders.mdx
@@ -0,0 +1,66 @@
+# Custom Graders
+
+Write your own grading functions to implement custom evaluation logic.
+
+<Note>
+Custom graders let you implement domain-specific evaluation, parse complex formats, and apply custom scoring algorithms.
+</Note>
+
+## Basic Structure
+
+```python
+from letta_evals.decorators import grader
+from letta_evals.models import GradeResult, Sample
+
+@grader
+def my_custom_grader(sample: Sample, submission: str) -> GradeResult:
+ """Custom grading logic."""
+
+ # Your evaluation logic
+ score = calculate_score(submission, sample.ground_truth)
+
+ # Ensure score is between 0.0 and 1.0
+ score = max(0.0, min(1.0, score))
+
+ return GradeResult(
+ score=score,
+ rationale=f"Score based on custom logic: {score}"
+ )
+```
+
+## Example: JSON Validation
+
+```python
+import json
+from letta_evals.decorators import grader
+from letta_evals.models import GradeResult, Sample
+
+@grader
+def valid_json(sample: Sample, submission: str) -> GradeResult:
+ """Check if submission is valid JSON."""
+ try:
+ json.loads(submission)
+ return GradeResult(score=1.0, rationale="Valid JSON")
+ except json.JSONDecodeError as e:
+ return GradeResult(score=0.0, rationale=f"Invalid JSON: {e}")
+```
+
+## Registration
+
+Custom graders are automatically registered when you import them in your suite's setup script or custom evaluators file.
+
+## Configuration
+
+```yaml
+graders:
+ my_metric:
+ kind: tool
+ function: my_custom_grader # Your function name
+ extractor: last_assistant
+```
+
+## Next Steps
+
+- [Tool Graders](/evals/graders/tool-graders) - Built-in grading functions
+- [Graders Concept](/evals/core-concepts/graders) - Understanding graders
+- [Example Custom Graders](https://github.com/letta-ai/letta-evals/tree/main/examples) - See examples in the letta-evals repo
diff --git a/fern/pages/evals/advanced/multi-turn-conversations.mdx b/fern/pages/evals/advanced/multi-turn-conversations.mdx
new file mode 100644
index 000000000..269761ef1
--- /dev/null
+++ b/fern/pages/evals/advanced/multi-turn-conversations.mdx
@@ -0,0 +1,94 @@
+# Multi-Turn Conversations
+
+Multi-turn conversations allow you to test how agents handle context across multiple exchanges.
+
+<Note>
+This is essential for stateful agents where behavior depends on conversation history.
+</Note>
+
+## Why Use Multi-Turn?
+
+Multi-turn conversations enable testing that single-turn prompts cannot:
+
+- **Memory storage**: Verify agents persist information to memory blocks
+- **Tool call sequences**: Test multi-step workflows
+- **Context retention**: Ensure agents remember details from earlier
+- **State evolution**: Track how agent state changes across interactions
+- **Conversational coherence**: Test if agents maintain context appropriately
+
+## Format
+
+### Single-Turn (Default)
+
+```jsonl
+{"input": "What is the capital of France?", "ground_truth": "Paris"}
+```
+
+### Multi-Turn
+
+```jsonl
+{"input": ["My name is Alice", "What's my name?"], "ground_truth": "Alice"}
+```
+
+The agent processes each input in sequence, with state carrying over between turns.
+
+## Example 1: Memory Recall Testing
+
+Test if the agent remembers information across turns:
+
+```jsonl
+{"input": ["Remember that my favorite color is blue", "What's my favorite color?"], "ground_truth": "blue"}
+```
+
+Suite configuration:
+```yaml
+graders:
+ response_check:
+ kind: tool
+ function: contains
+ extractor: last_assistant # Check the agent's response
+```
+
+## Example 2: Memory Correction Testing
+
+Test if the agent correctly updates memory when users correct themselves:
+
+```jsonl
+{"input": ["Please remember that I like bananas.", "Actually, sorry, I meant I like apples."], "ground_truth": "apples"}
+```
+
+Suite configuration:
+```yaml
+graders:
+ memory_check:
+ kind: tool
+ function: contains
+ extractor: memory_block
+ extractor_config:
+ block_label: human # Check the actual memory block, not just the response
+```
+
+<Note>
+**Key difference:** The `memory_block` extractor verifies the agent actually stored the corrected information in memory, not just that it responded correctly. This tests real memory persistence.
+</Note>
+
+## When to Test Memory Blocks vs. Responses
+
+**Use `last_assistant` or `all_assistant` extractors when:**
+- Testing what the agent says in conversation
+- Verifying response content and phrasing
+- Checking conversational coherence
+
+**Use `memory_block` extractor when:**
+- Verifying information was actually stored in memory
+- Testing memory updates and corrections
+- Validating persistent state changes
+- Ensuring the agent's internal state is correct
+
+See the [multiturn-memory-block-extractor example](https://github.com/letta-ai/letta-evals/tree/main/examples/multiturn-memory-block-extractor) for a complete working implementation.
+
+## Next Steps
+
+- [Datasets](/evals/core-concepts/datasets) - Creating test datasets
+- [Extractors](/evals/core-concepts/extractors) - Extracting from trajectories
+- [Targets](/evals/core-concepts/targets) - Agent lifecycle and testing behavior
diff --git a/fern/pages/evals/cli/commands.mdx b/fern/pages/evals/cli/commands.mdx
new file mode 100644
index 000000000..0b8eee31c
--- /dev/null
+++ b/fern/pages/evals/cli/commands.mdx
@@ -0,0 +1,342 @@
+# CLI Commands
+
+The **letta-evals** command-line interface lets you run evaluations, validate configurations, and inspect available components.
+
+<Note>
+**Quick overview:**
+- **`run`** - Execute an evaluation suite (most common)
+- **`validate`** - Check suite configuration without running
+- **`list-extractors`** - Show available extractors
+- **`list-graders`** - Show available grader functions
+- **Exit codes** - 0 for pass, 1 for fail (perfect for CI/CD)
+</Note>
+
+**Typical workflow:**
+1. Validate your suite: `letta-evals validate suite.yaml`
+2. Run evaluation: `letta-evals run suite.yaml --output results/`
+3. Check exit code: `echo $?` (0 = passed, 1 = failed)
+
+## run
+
+Run an evaluation suite.
+
+```bash
+letta-evals run <suite.yaml> [options]
+```
+
+### Arguments
+
+- `suite.yaml`: Path to the suite configuration file (required)
+
+### Options
+
+#### --output, -o
+Save results to a directory.
+
+```bash
+letta-evals run suite.yaml --output results/
+```
+
+Creates:
+- `results/header.json`: Evaluation metadata
+- `results/summary.json`: Aggregate metrics and configuration
+- `results/results.jsonl`: Per-sample results (one JSON per line)
+
+#### --quiet, -q
+Quiet mode - only show pass/fail result.
+
+```bash
+letta-evals run suite.yaml --quiet
+```
+
+Output:
+```
+✓ PASSED
+```
+
+#### --max-concurrent
+Maximum concurrent sample evaluations. **Default**: 15
+
+```bash
+letta-evals run suite.yaml --max-concurrent 10
+```
+
+Higher values = faster evaluation but more resource usage.
+
+#### --api-key
+Letta API key (overrides LETTA_API_KEY environment variable).
+
+```bash
+letta-evals run suite.yaml --api-key your-key
+```
+
+#### --base-url
+Letta server base URL (overrides suite config and environment variable).
+
+```bash
+letta-evals run suite.yaml --base-url http://localhost:8283
+```
+
+#### --project-id
+Letta project ID for cloud deployments.
+
+```bash
+letta-evals run suite.yaml --project-id proj_abc123
+```
+
+#### --cached, -c
+Path to cached results (JSONL) for re-grading trajectories without re-running the agent.
+
+```bash
+letta-evals run suite.yaml --cached previous_results.jsonl
+```
+
+Use this to test different graders on the same agent trajectories.
+
+#### --num-runs
+Run the evaluation multiple times to measure consistency. **Default**: 1
+
+```bash
+letta-evals run suite.yaml --num-runs 10
+```
+
+**Output with multiple runs:**
+- Each run creates a separate `run_N/` directory with individual results
+- An `aggregate_stats.json` file contains statistics across all runs (mean, standard deviation, pass rate)
+
+### Examples
+
+Basic run:
+```bash
+letta-evals run suite.yaml # Run evaluation, show results in terminal
+```
+
+Save results:
+```bash
+letta-evals run suite.yaml --output evaluation-results/ # Save to directory
+```
+
+Letta Cloud:
+```bash
+letta-evals run suite.yaml \
+ --base-url https://api.letta.com \
+ --api-key $LETTA_API_KEY \
+ --project-id proj_abc123
+```
+
+Quiet CI mode:
+```bash
+letta-evals run suite.yaml --quiet
+if [ $? -eq 0 ]; then
+ echo "Evaluation passed"
+else
+ echo "Evaluation failed"
+ exit 1
+fi
+```
+
+### Exit Codes
+
+- `0`: Evaluation passed (gate criteria met)
+- `1`: Evaluation failed (gate criteria not met or error)
+
+## validate
+
+Validate a suite configuration without running it.
+
+```bash
+letta-evals validate <suite.yaml>
+```
+
+Checks:
+- YAML syntax is valid
+- Required fields are present
+- Paths exist
+- Configuration is consistent
+- Grader/extractor combinations are valid
+
+Output on success:
+```
+✓ Suite configuration is valid
+```
+
+Output on error:
+```
+✗ Validation failed:
+ - Agent file not found: agent.af
+ - Grader 'my_metric' references unknown function
+```
+
+## list-extractors
+
+List all available extractors.
+
+```bash
+letta-evals list-extractors
+```
+
+Output:
+```
+Available extractors:
+ last_assistant - Extract the last assistant message
+ first_assistant - Extract the first assistant message
+ all_assistant - Concatenate all assistant messages
+ pattern - Extract content matching regex
+ tool_arguments - Extract tool call arguments
+ tool_output - Extract tool return value
+ after_marker - Extract content after a marker
+ memory_block - Extract from memory block (requires agent_state)
+```
+
+## list-graders
+
+List all available grader functions.
+
+```bash
+letta-evals list-graders
+```
+
+Output:
+```
+Available graders:
+ exact_match - Exact string match with ground_truth
+ contains - Check if contains ground_truth
+ regex_match - Match regex pattern
+ ascii_printable_only - Validate ASCII-only content
+```
+
+## help
+
+Show help information.
+
+```bash
+letta-evals --help
+```
+
+Show help for a specific command:
+
+```bash
+letta-evals run --help
+letta-evals validate --help
+```
+
+## Environment Variables
+
+### LETTA_API_KEY
+API key for Letta authentication.
+
+```bash
+export LETTA_API_KEY=your-key-here
+```
+
+### LETTA_BASE_URL
+Letta server base URL.
+
+```bash
+export LETTA_BASE_URL=http://localhost:8283
+```
+
+### LETTA_PROJECT_ID
+Letta project ID (for cloud).
+
+```bash
+export LETTA_PROJECT_ID=proj_abc123
+```
+
+### OPENAI_API_KEY
+OpenAI API key (for rubric graders).
+
+```bash
+export OPENAI_API_KEY=your-openai-key
+```
+
+## Configuration Priority
+
+Configuration values are resolved in this order (highest to lowest priority):
+
+1. CLI arguments (`--api-key`, `--base-url`, `--project-id`)
+2. Suite YAML configuration
+3. Environment variables
+
+## Using in CI/CD
+
+### GitHub Actions
+
+```yaml
+name: Run Evals
+on: [push]
+
+jobs:
+ evaluate:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v2
+
+ - name: Install dependencies
+ run: pip install letta-evals
+
+ - name: Run evaluation
+ env:
+ LETTA_API_KEY: ${{ secrets.LETTA_API_KEY }}
+ OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+ run: |
+ letta-evals run suite.yaml --quiet --output results/
+
+ - name: Upload results
+ uses: actions/upload-artifact@v2
+ with:
+ name: eval-results
+ path: results/
+```
+
+### GitLab CI
+
+```yaml
+evaluate:
+ script:
+ - pip install letta-evals
+ - letta-evals run suite.yaml --quiet --output results/
+ artifacts:
+ paths:
+ - results/
+ variables:
+ LETTA_API_KEY: $LETTA_API_KEY
+ OPENAI_API_KEY: $OPENAI_API_KEY
+```
+
+## Debugging
+
+### Common Issues
+
+<Warning>
+**"Agent file not found"**
+
+```bash
+# Check file exists relative to suite YAML location
+ls -la path/to/agent.af
+```
+</Warning>
+
+<Warning>
+**"Connection refused"**
+
+```bash
+# Verify Letta server is running
+curl http://localhost:8283/v1/health
+```
+</Warning>
+
+<Warning>
+**"Invalid API key"**
+
+```bash
+# Check environment variable is set
+echo $LETTA_API_KEY
+```
+</Warning>
+
+## Next Steps
+
+- [Understanding Results](/evals/results-metrics/understanding-results) - Interpreting evaluation output
+- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete configuration options
+- [Getting Started](/evals/get-started/getting-started) - Complete tutorial with examples
diff --git a/fern/pages/evals/concepts/datasets.mdx b/fern/pages/evals/concepts/datasets.mdx
new file mode 100644
index 000000000..70c031839
--- /dev/null
+++ b/fern/pages/evals/concepts/datasets.mdx
@@ -0,0 +1,425 @@
+# Datasets
+
+**Datasets** are the test cases that define what your agent will be evaluated on. Each sample in your dataset represents one evaluation scenario.
+
+<Note>
+**Quick overview:**
+- **Two formats**: JSONL (flexible, powerful) or CSV (simple, spreadsheet-friendly)
+- **Required field**: `input` - the prompt(s) to send to the agent
+- **Common fields**: `ground_truth` (expected answer), `tags` (for filtering), `metadata` (extra info)
+- **Advanced fields**: `agent_args` (customize agent per sample), `rubric_vars` (per-sample rubric context)
+- **Multi-turn support**: Send multiple messages in sequence using arrays
+</Note>
+
+**Typical workflow:**
+1. Create a JSONL or CSV file with test cases
+2. Reference it in your suite YAML: `dataset: test_cases.jsonl`
+3. Run evaluation - each sample is tested independently
+4. Results show per-sample and aggregate scores
+
+Datasets can be created in two formats: **JSONL** or **CSV**. Choose based on your team's workflow and complexity needs.
+
+## Dataset Formats
+
+### JSONL Format
+
+Each line is a JSON object representing one test case:
+
+```jsonl
+{"input": "What's the capital of France?", "ground_truth": "Paris"}
+{"input": "Calculate 2+2", "ground_truth": "4"}
+{"input": "What color is the sky?", "ground_truth": "blue"}
+```
+
+**Best for:**
+- Complex data structures (nested objects, arrays)
+- Multi-turn conversations
+- Advanced features (agent_args, rubric_vars)
+- Teams comfortable with JSON/code
+- Version control (clean line-by-line diffs)
+
+### CSV Format
+
+Standard CSV with headers:
+
+```csv
+input,ground_truth
+"What's the capital of France?","Paris"
+"Calculate 2+2","4"
+"What color is the sky?","blue"
+```
+
+**Best for:**
+- Simple question-answer pairs
+- Teams that prefer spreadsheets (Excel, Google Sheets)
+- Non-technical collaborators creating test cases
+- Quick dataset creation and editing
+- Easy sharing with non-developers
+
+## Quick Reference
+
+| Field | Required | Type | Purpose |
+|-------|----------|------|---------|
+| `input` | ✅ | string or array | Prompt(s) to send to agent |
+| `ground_truth` | ❌ | string | Expected answer (for tool graders) |
+| `tags` | ❌ | array of strings | For filtering samples |
+| `agent_args` | ❌ | object | Per-sample agent customization |
+| `rubric_vars` | ❌ | object | Per-sample rubric variables |
+| `metadata` | ❌ | object | Arbitrary extra data |
+| `id` | ❌ | integer | Sample ID (auto-assigned if omitted) |
+
+## Field Reference
+
+### Required Fields
+
+#### input
+The prompt(s) to send to the agent. Can be a string or array of strings:
+
+Single message:
+```json
+{"input": "Hello, who are you?"}
+```
+
+Multi-turn conversation:
+```json
+{"input": ["Hello", "What's your name?", "Tell me about yourself"]}
+```
+
+### Optional Fields
+
+#### ground_truth
+The expected answer or content to check against. Required for most tool graders (exact_match, contains, etc.):
+
+```json
+{"input": "What is 2+2?", "ground_truth": "4"}
+```
+
+#### metadata
+Arbitrary additional data about the sample:
+
+```json
+{
+ "input": "What is photosynthesis?",
+ "ground_truth": "process where plants convert light into energy",
+ "metadata": {
+ "category": "biology",
+ "difficulty": "medium"
+ }
+}
+```
+
+#### tags
+List of tags for filtering samples:
+
+```json
+{"input": "Solve x^2 = 16", "ground_truth": "4", "tags": ["math", "algebra"]}
+```
+
+Filter by tags in your suite:
+```yaml
+sample_tags: [math] # Only samples tagged "math" will be evaluated
+```
+
+#### agent_args
+
+Custom arguments passed to programmatic agent creation when using `agent_script`. Allows per-sample agent customization.
+
+JSONL:
+```json
+{
+ "input": "What items do we have?",
+ "agent_args": {
+ "item": {"sku": "SKU-123", "name": "Widget A", "price": 19.99}
+ }
+}
+```
+
+CSV:
+```csv
+input,agent_args
+"What items do we have?","{""item"": {""sku"": ""SKU-123"", ""name"": ""Widget A"", ""price"": 19.99}}"
+```
+
+Your agent factory function can access these values via `sample.agent_args` to customize agent configuration.
+
+See [Targets - agent_script](/evals/core-concepts/targets#agent_script) for details on programmatic agent creation.
+
+#### rubric_vars
+
+Variables to inject into rubric templates when using rubric graders. This allows you to provide per-sample context or examples to the LLM judge.
+
+**Example:** Evaluating code quality against a reference implementation.
+
+JSONL:
+```jsonl
+{"input": "Write a function to calculate fibonacci numbers", "rubric_vars": {"reference_code": "def fib(n):\n if n <= 1: return n\n return fib(n-1) + fib(n-2)", "required_features": "recursion, base case"}}
+```
+
+CSV:
+```csv
+input,rubric_vars
+"Write a function to calculate fibonacci numbers","{""reference_code"": ""def fib(n):\n if n <= 1: return n\n return fib(n-1) + fib(n-2)"", ""required_features"": ""recursion, base case""}"
+```
+
+In your rubric template file, reference variables with `{variable_name}`:
+
+**rubric.txt:**
+```
+Evaluate the submitted code against this reference implementation:
+
+{reference_code}
+
+Required features: {required_features}
+
+Score on correctness (0.6) and code quality (0.4).
+```
+
+When the rubric grader runs, variables are replaced with values from `rubric_vars`:
+
+**Final formatted prompt sent to LLM:**
+```
+Evaluate the submitted code against this reference implementation:
+
+def fib(n):
+ if n <= 1: return n
+ return fib(n-1) + fib(n-2)
+
+Required features: recursion, base case
+
+Score on correctness (0.6) and code quality (0.4).
+```
+
+This lets you customize evaluation criteria per sample using the same rubric template.
+
+See [Rubric Graders](/evals/graders/rubric-graders) for details on rubric templates.
+
+#### id
+Sample ID is automatically assigned (0-based index) if not provided. You can override:
+
+```json
+{"id": 42, "input": "Test case 42"}
+```
+
+## Complete Example
+
+```jsonl
+{"id": 1, "input": "What is the capital of France?", "ground_truth": "Paris", "tags": ["geography", "easy"], "metadata": {"region": "Europe"}}
+{"id": 2, "input": "Calculate the square root of 144", "ground_truth": "12", "tags": ["math", "medium"]}
+{"id": 3, "input": ["Hello", "What can you help me with?"], "tags": ["conversation"]}
+```
+
+## Dataset Best Practices
+
+### 1. Clear Ground Truth
+
+Make ground truth specific enough to grade but flexible enough to match valid responses:
+
+<Tip>
+Good:
+```json
+{"input": "What's the largest planet?", "ground_truth": "Jupiter"}
+```
+</Tip>
+
+<Warning>
+Too strict (might miss valid answers):
+```json
+{"input": "What's the largest planet?", "ground_truth": "Jupiter is the largest planet in our solar system."}
+```
+</Warning>
+
+### 2. Diverse Test Cases
+
+Include edge cases and variations:
+
+```jsonl
+{"input": "What is 2+2?", "ground_truth": "4", "tags": ["math", "easy"]}
+{"input": "What is 0.1 + 0.2?", "ground_truth": "0.3", "tags": ["math", "floating_point"]}
+{"input": "What is 999999999 + 1?", "ground_truth": "1000000000", "tags": ["math", "large_numbers"]}
+```
+
+### 3. Use Tags for Organization
+
+Organize samples by type, difficulty, or feature:
+
+```json
+{"tags": ["tool_usage", "search"]}
+{"tags": ["memory", "recall"]}
+{"tags": ["reasoning", "multi_step"]}
+```
+
+### 4. Multi-Turn Conversations
+
+Test conversational context and memory updates:
+
+```jsonl
+{"input": ["My name is Alice", "What's my name?"], "ground_truth": "Alice", "tags": ["memory", "recall"]}
+{"input": ["Please remember that I like bananas.", "Actually, sorry, I meant I like apples."], "ground_truth": "apples", "tags": ["memory", "correction"]}
+{"input": ["I work at Google", "Update my workplace to Microsoft", "Where do I work?"], "ground_truth": "Microsoft", "tags": ["memory", "multi_step"]}
+```
+
+<Tip>
+**Testing memory corrections:** Use multi-turn inputs to test if agents properly update memory when users correct themselves. Combine with the `memory_block` extractor to verify the final memory state, not just the response.
+</Tip>
+
+### 5. No Ground Truth for LLM Judges
+
+If using rubric graders, ground truth is optional:
+
+```jsonl
+{"input": "Write a creative story about a robot", "tags": ["creative"]}
+{"input": "Explain quantum computing simply", "tags": ["explanation"]}
+```
+
+The LLM judge evaluates based on the rubric, not ground truth.
+
+## Loading Datasets
+
+Datasets are automatically loaded by the runner:
+
+```yaml
+dataset: path/to/dataset.jsonl # Path to your test cases (JSONL or CSV)
+```
+
+Paths are relative to the suite YAML file location.
+
+## Dataset Filtering
+
+### Limit Sample Count
+
+```yaml
+max_samples: 10 # Only evaluate first 10 samples (useful for testing)
+```
+
+### Filter by Tags
+
+```yaml
+sample_tags: [math, medium] # Only samples with ALL these tags
+```
+
+## Creating Datasets Programmatically
+
+You can generate datasets with Python:
+
+```python
+import json
+
+samples = []
+for i in range(100):
+ samples.append({
+ "input": f"What is {i} + {i}?",
+ "ground_truth": str(i + i),
+ "tags": ["math", "addition"]
+ })
+
+with open("dataset.jsonl", "w") as f:
+ for sample in samples:
+ f.write(json.dumps(sample) + "\n")
+```
+
+## Dataset Format Validation
+
+The runner validates:
+- Each line is valid JSON
+- Required fields are present
+- Field types are correct
+
+Validation errors will be reported with line numbers.
+
+## Examples by Use Case
+
+### Question Answering
+
+JSONL:
+```jsonl
+{"input": "What is the capital of France?", "ground_truth": "Paris"}
+{"input": "Who wrote Romeo and Juliet?", "ground_truth": "Shakespeare"}
+```
+
+CSV:
+```csv
+input,ground_truth
+"What is the capital of France?","Paris"
+"Who wrote Romeo and Juliet?","Shakespeare"
+```
+
+### Tool Usage Testing
+
+JSONL:
+```jsonl
+{"input": "Search for information about pandas", "ground_truth": "search"}
+{"input": "Calculate 15 * 23", "ground_truth": "calculator"}
+```
+
+CSV:
+```csv
+input,ground_truth
+"Search for information about pandas","search"
+"Calculate 15 * 23","calculator"
+```
+
+Ground truth = expected tool name.
+
+### Memory Testing (Multi-turn)
+
+JSONL:
+```jsonl
+{"input": ["Remember that my favorite color is blue", "What's my favorite color?"], "ground_truth": "blue"}
+{"input": ["I live in Tokyo", "Where do I live?"], "ground_truth": "Tokyo"}
+```
+
+CSV (using JSON array strings):
+```csv
+input,ground_truth
+"[""Remember that my favorite color is blue"", ""What's my favorite color?""]","blue"
+"[""I live in Tokyo"", ""Where do I live?""]","Tokyo"
+```
+
+### Code Generation
+
+JSONL:
+```jsonl
+{"input": "Write a function to reverse a string in Python"}
+{"input": "Create a SQL query to find users older than 21"}
+```
+
+CSV:
+```csv
+input
+"Write a function to reverse a string in Python"
+"Create a SQL query to find users older than 21"
+```
+
+Use rubric graders to evaluate code quality.
+
+## CSV Advanced Features
+
+CSV supports all the same features as JSONL by encoding complex data as JSON strings in cells:
+
+**Multi-turn conversations** (requires escaped JSON array string):
+```csv
+input,ground_truth
+"[""Hello"", ""What's your name?""]","Alice"
+```
+
+**Agent arguments** (requires escaped JSON object string):
+```csv
+input,agent_args
+"What items do we have?","{""initial_inventory"": [""apple"", ""banana""]}"
+```
+
+**Rubric variables** (requires escaped JSON object string):
+```csv
+input,rubric_vars
+"Write a story","{""max_length"": 500, ""genre"": ""sci-fi""}"
+```
+
+<Note>
+**Note:** Complex data structures require JSON encoding in CSV. If you're frequently using these advanced features, JSONL may be easier to read and maintain.
+</Note>
+
+## Next Steps
+
+- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete configuration options including filtering
+- [Graders](/evals/core-concepts/graders) - How to evaluate agent responses
+- [Multi-Turn Conversations](/evals/advanced/multi-turn-conversations) - Testing conversational flows
diff --git a/fern/pages/evals/concepts/extractors.mdx b/fern/pages/evals/concepts/extractors.mdx
new file mode 100644
index 000000000..247e7904c
--- /dev/null
+++ b/fern/pages/evals/concepts/extractors.mdx
@@ -0,0 +1,374 @@
+# Extractors
+
+**Extractors** select what content to evaluate from an agent's response. They navigate the conversation trajectory and extract the specific piece you want to grade.
+
+<Note>
+**Quick overview:**
+- **Purpose**: Agent responses are complex (messages, tool calls, memory) - extractors isolate what to grade
+- **Built-in options**: last_assistant, tool_arguments, memory_block, pattern, and more
+- **Flexible**: Different graders can use different extractors in the same suite
+- **Automatic**: No setup needed - just specify in your grader config
+</Note>
+
+**Common patterns:**
+- `last_assistant` - Most common, gets the agent's final message (90% of use cases)
+- `tool_arguments` - Verify agent called the right tool with correct args
+- `memory_block` - Check if agent updated memory correctly
+- `pattern` - Extract structured data with regex
+
+Extractors determine what part of the agent's response gets graded. They pull out specific content from the conversation trajectory.
+
+## Why Extractors?
+
+An agent's response is complex - it includes assistant messages, tool calls, tool returns, memory updates, etc. Extractors let you focus on exactly what you want to evaluate.
+
+**The evaluation flow:**
+```
+Agent Response → Extractor → Submission Text → Grader → Score
+```
+
+For example:
+```
+Full trajectory:
+ UserMessage: "What's the capital of France?"
+ ToolCallMessage: search(query="capital of france")
+ ToolReturnMessage: "Paris is the capital..."
+ AssistantMessage: "The capital of France is Paris."
+
+↓ extractor: last_assistant ↓
+
+Extracted: "The capital of France is Paris."
+
+↓ grader: contains (ground_truth="Paris") ↓
+
+Score: 1.0
+```
+
+## Trajectory Structure
+
+A trajectory is a list of turns, where each turn is a list of Letta messages:
+
+```python
+[
+ [UserMessage(...), AssistantMessage(...), ToolCallMessage(...), ToolReturnMessage(...)], # Turn 1
+ [AssistantMessage(...)] # Turn 2
+]
+```
+
+Extractors navigate this structure to pull out the submission text.
+
+## Built-in Extractors
+
+### last_assistant
+
+Extracts the last assistant message content.
+
+```yaml
+graders:
+ quality:
+ kind: tool
+ function: contains
+ extractor: last_assistant # Extract final agent message
+```
+
+Most common extractor - gets the agent's final response.
+
+### first_assistant
+
+Extracts the first assistant message content.
+
+```yaml
+graders:
+ initial_response:
+ kind: tool
+ function: contains
+ extractor: first_assistant # Extract first agent message
+```
+
+Useful for testing immediate responses before tool usage.
+
+### all_assistant
+
+Concatenates all assistant messages with a separator.
+
+```yaml
+graders:
+ complete_response:
+ kind: rubric
+ prompt_path: rubric.txt
+ extractor: all_assistant # Concatenate all agent messages
+ extractor_config:
+ separator: "\n\n" # Join messages with double newline
+```
+
+Use when you need the full conversation context.
+
+### last_turn
+
+Extracts all assistant messages from the last turn only.
+
+```yaml
+graders:
+ final_turn:
+ kind: tool
+ function: contains
+ extractor: last_turn # Messages from final turn only
+ extractor_config:
+ separator: " " # Join with spaces
+```
+
+Useful when the agent makes multiple statements in the final turn.
+
+### pattern
+
+Extracts content matching a regex pattern from assistant messages.
+
+```yaml
+graders:
+ extract_number:
+ kind: tool
+ function: exact_match
+ extractor: pattern # Extract using regex
+ extractor_config:
+ pattern: 'Result: (\d+)' # Regex pattern to match
+ group: 1 # Extract capture group 1
+ search_all: false # Only find first match
+```
+
+Example: Extract "42" from "The answer is Result: 42"
+
+### tool_arguments
+
+Extracts arguments from a specific tool call.
+
+```yaml
+graders:
+ search_query:
+ kind: tool
+ function: contains
+ extractor: tool_arguments # Extract tool call arguments
+ extractor_config:
+ tool_name: search # Which tool to extract from
+```
+
+Returns the JSON arguments as a string.
+
+Example: If agent calls `search(query="pandas", limit=10)`, extracts:
+```json
+{"query": "pandas", "limit": 10}
+```
+
+### tool_output
+
+Extracts the return value from a specific tool call.
+
+```yaml
+graders:
+ search_results:
+ kind: tool
+ function: contains
+ extractor: tool_output # Extract tool return value
+ extractor_config:
+ tool_name: search # Which tool's output to extract
+```
+
+Finds the tool call and its corresponding return message.
+
+### after_marker
+
+Extracts content after a specific marker string.
+
+```yaml
+graders:
+ answer_section:
+ kind: tool
+ function: contains
+ extractor: after_marker # Extract content after marker
+ extractor_config:
+ marker: "ANSWER:" # Marker string to find
+ include_marker: false # Don't include "ANSWER:" in output
+```
+
+Example: From "Here's my analysis... ANSWER: Paris", extracts "Paris"
+
+### memory_block
+
+Extracts content from a specific memory block (requires agent_state).
+
+```yaml
+graders:
+ human_memory:
+ kind: tool
+ function: exact_match
+ extractor: memory_block # Extract from agent memory
+ extractor_config:
+ block_label: human # Which memory block to extract
+```
+
+<Warning>
+**Important**: This extractor requires the agent's final state, which adds overhead. The runner automatically fetches agent_state when this extractor is used.
+</Warning>
+
+Example use case: Verify the agent correctly updated its memory about the user.
+
+## Extractor Configuration
+
+Some extractors accept additional configuration via `extractor_config`:
+
+```yaml
+graders:
+ my_metric:
+ kind: tool
+ function: contains
+ extractor: pattern # Use pattern extractor
+ extractor_config: # Configuration for this extractor
+ pattern: 'Answer: (.*)' # Regex pattern
+ group: 1 # Extract capture group 1
+```
+
+## Choosing an Extractor
+
+| Use Case | Recommended Extractor |
+|----------|---------------------|
+| Final agent response | `last_assistant` |
+| First response before tools | `first_assistant` |
+| Complete conversation | `all_assistant` |
+| Specific format extraction | `pattern` |
+| Tool usage validation | `tool_arguments` |
+| Tool result checking | `tool_output` |
+| Memory validation | `memory_block` |
+| Structured output | `after_marker` |
+
+## Content Flattening
+
+Assistant messages can contain multiple content parts. Extractors automatically flatten complex content to plain text.
+
+## Empty Extraction
+
+If an extractor finds no matching content, it returns an empty string `""`. This typically results in a score of 0.0 from the grader.
+
+## Custom Extractors
+
+You can write custom extractors. See [Custom Extractors](/evals/extractors/custom-extractors) for details.
+
+Example:
+
+```python
+from letta_evals.decorators import extractor
+from letta_client import LettaMessageUnion
+
+@extractor
+def my_extractor(trajectory: List[List[LettaMessageUnion]], config: dict) -> str:
+ # Custom extraction logic
+ return extracted_text
+```
+
+Register by importing in your suite's setup script or custom evaluators file.
+
+## Multi-Metric Extraction
+
+Different graders can use different extractors:
+
+```yaml
+graders:
+ response_quality: # Evaluate final message quality
+ kind: rubric
+ prompt_path: quality.txt
+ extractor: last_assistant # Extract final response
+
+ tool_usage: # Check tool was called correctly
+ kind: tool
+ function: exact_match
+ extractor: tool_arguments # Extract tool args
+ extractor_config:
+ tool_name: search # From search tool
+
+ memory_update: # Verify memory updated
+ kind: tool
+ function: contains
+ extractor: memory_block # Extract from memory
+ extractor_config:
+ block_label: human # Human memory block
+```
+
+Each grader independently extracts and evaluates different aspects.
+
+## Listing Extractors
+
+See all available extractors:
+
+```bash
+letta-evals list-extractors
+```
+
+## Examples
+
+### Extract Final Answer
+
+```yaml
+extractor: last_assistant # Get final agent message
+```
+
+Agent: "Let me search... *uses tool* ... The answer is Paris."
+Extracted: "The answer is Paris."
+
+### Extract Tool Arguments
+
+```yaml
+extractor: tool_arguments # Get tool call args
+extractor_config:
+ tool_name: search # From search tool
+```
+
+Agent calls: `search(query="pandas", limit=5)`
+Extracted: `{"query": "pandas", "limit": 5}`
+
+### Extract Pattern
+
+```yaml
+extractor: pattern # Extract with regex
+extractor_config:
+ pattern: 'RESULT: (\w+)' # Match pattern
+ group: 1 # Extract capture group 1
+```
+
+Agent: "After calculation... RESULT: SUCCESS"
+Extracted: "SUCCESS"
+
+### Extract Memory
+
+```yaml
+extractor: memory_block # Extract from agent memory
+extractor_config:
+ block_label: human # Human memory block
+```
+
+Agent updates memory block "human" to: "User's name is Alice"
+Extracted: "User's name is Alice"
+
+## Troubleshooting
+
+<Warning>
+**Extractor returns empty string**
+
+**Problem**: Grader always gives score 0.0 because extractor finds nothing.
+
+**Common causes**:
+- **Wrong extractor**: Using `first_assistant` but agent doesn't respond until after tool use → use `last_assistant`
+- **Wrong tool name**: `tool_arguments` with `tool_name: "search"` but agent calls `"web_search"` → check actual tool name
+- **Wrong memory block**: `memory_block` with `block_label: "user"` but block is actually labeled `"human"` → check block labels
+- **Pattern doesn't match**: `pattern: "Answer: (.*)"` but agent says "The answer is..." → adjust regex
+</Warning>
+
+<Tip>
+**Debug tips**:
+1. Check the trajectory in results JSON to see actual agent output
+2. Use `last_assistant` first to see what's there
+3. Verify tool names with `letta-evals list-extractors`
+</Tip>
+
+## Next Steps
+
+- [Built-in Extractors Reference](/evals/extractors/built-in-extractors) - Complete extractor documentation
+- [Custom Extractors Guide](/evals/extractors/custom-extractors) - Write your own extractors
+- [Graders](/evals/core-concepts/graders) - How to use extractors with graders
diff --git a/fern/pages/evals/concepts/gates.mdx b/fern/pages/evals/concepts/gates.mdx
new file mode 100644
index 000000000..29d5f31d2
--- /dev/null
+++ b/fern/pages/evals/concepts/gates.mdx
@@ -0,0 +1,384 @@
+# Gates
+
+**Gates** are the pass/fail criteria for your evaluation. They determine whether your agent meets the required performance threshold by checking aggregate metrics.
+
+<Note>
+**Quick overview:**
+- **Single decision**: One gate per suite determines pass/fail
+- **Two metrics**: `avg_score` (average of all scores) or `accuracy` (percentage passing threshold)
+- **Flexible operators**: `>=`, `>`, `<=`, `<`, `==` for threshold comparison
+- **Customizable pass criteria**: Define what counts as "passing" for accuracy calculations
+- **Exit codes**: Suite exits 0 for pass, 1 for fail
+</Note>
+
+**Common patterns:**
+- Average score must be 80%+: `avg_score >= 0.8`
+- 90%+ of samples must pass: `accuracy >= 0.9`
+- Custom threshold: Define per-sample pass criteria with `pass_value`
+
+Gates define the pass/fail criteria for your evaluation. They check if aggregate metrics meet a threshold.
+
+## Basic Structure
+
+```yaml
+gate:
+ metric_key: accuracy # Which grader to evaluate
+ metric: avg_score # Use average score (default)
+ op: gte # Greater than or equal
+ value: 0.8 # 80% threshold
+```
+
+## Why Use Gates?
+
+Gates provide **automated pass/fail decisions** for your evaluations, which is essential for:
+
+**CI/CD Integration**: Gates let you block deployments if agent performance drops:
+```bash
+letta-evals run suite.yaml
+# Exit code 0 = pass (continue deployment)
+# Exit code 1 = fail (block deployment)
+```
+
+**Regression Testing**: Set a baseline threshold and ensure new changes don't degrade performance:
+```yaml
+gate:
+ metric: avg_score
+ op: gte
+ value: 0.85 # Must maintain 85%+ to pass
+```
+
+**Quality Enforcement**: Require agents meet minimum standards before production:
+```yaml
+gate:
+ metric: accuracy
+ op: gte
+ value: 0.95 # 95% of test cases must pass
+```
+
+### What Happens When Gates Fail?
+
+When a gate condition is not met:
+
+1. **Console output** shows failure message:
+ ```text
+ ✗ FAILED (0.72/1.00 avg, 72.0% pass rate)
+ Gate check failed: avg_score (0.72) not >= 0.80
+ ```
+
+2. **Exit code** is 1 (non-zero indicates failure):
+ ```bash
+ letta-evals run suite.yaml
+ echo $? # Prints 1 if gate failed
+ ```
+
+3. **Results JSON** includes `gate_passed: false`:
+ ```json
+ {
+ "gate_passed": false,
+ "gate_check": {
+ "metric": "avg_score",
+ "value": 0.72,
+ "threshold": 0.80,
+ "operator": "gte",
+ "passed": false
+ },
+ "metrics": { ... }
+ }
+ ```
+
+4. **All other data is preserved** - you still get full results, scores, and trajectories even when gating fails
+
+<Tip>
+**Common use case in CI**:
+
+```bash
+#!/bin/bash
+letta-evals run suite.yaml --output results.json
+
+if [ $? -ne 0 ]; then
+ echo "❌ Agent evaluation failed - blocking merge"
+ exit 1
+else
+ echo "✅ Agent evaluation passed - safe to merge"
+fi
+```
+</Tip>
+
+## Required Fields
+
+### metric_key
+
+Which grader to evaluate. Must match a key in your `graders` section:
+
+```yaml
+graders:
+ accuracy: # Grader name
+ kind: tool
+ function: exact_match
+ extractor: last_assistant
+
+gate:
+ metric_key: accuracy # Must match grader name above
+ op: gte # >=
+ value: 0.8 # 80% threshold
+```
+
+If you only have one grader, `metric_key` can be omitted - it will default to your single grader.
+
+### metric
+
+Which aggregate statistic to compare. Two options:
+
+#### avg_score
+
+Average score across all samples (0.0 to 1.0):
+
+```yaml
+gate:
+ metric_key: quality # Check quality grader
+ metric: avg_score # Use average of all scores
+ op: gte # >=
+ value: 0.7 # Must average 70%+
+```
+
+Example: If scores are [0.8, 0.9, 0.6], avg_score = 0.77
+
+#### accuracy
+
+Pass rate as a percentage (0.0 to 1.0):
+
+```yaml
+gate:
+ metric_key: accuracy # Check accuracy grader
+ metric: accuracy # Use pass rate, not average
+ op: gte # >=
+ value: 0.8 # 80% of samples must pass
+```
+
+By default, samples with score `>= 1.0` are considered "passing".
+
+You can customize the per-sample threshold with `pass_op` and `pass_value` (see below).
+
+<Note>
+**Note**: The default `metric` is `avg_score`, so you can omit it if that's what you want:
+
+```yaml
+gate:
+ metric_key: quality # Check quality grader
+ op: gte # >=
+ value: 0.7 # 70% threshold (defaults to avg_score)
+```
+</Note>
+
+### op
+
+Comparison operator:
+
+- `gte`: Greater than or equal (`>=`)
+- `gt`: Greater than (`>`)
+- `lte`: Less than or equal (`<=`)
+- `lt`: Less than (`<`)
+- `eq`: Equal (`==`)
+
+Most common: `gte` (at least X)
+
+### value
+
+Threshold value for comparison:
+
+- For `avg_score`: 0.0 to 1.0
+- For `accuracy`: 0.0 to 1.0 (representing percentage)
+
+```yaml
+gate:
+ metric: avg_score # Average score
+ op: gte # >=
+ value: 0.75 # 75% threshold
+```
+
+```yaml
+gate:
+ metric: accuracy # Pass rate
+ op: gte # >=
+ value: 0.9 # 90% must pass
+```
+
+## Optional Fields
+
+### pass_op and pass_value
+
+Customize when individual samples are considered "passing" (used for accuracy calculation):
+
+```yaml
+gate:
+ metric_key: quality # Check quality grader
+ metric: accuracy # Use pass rate
+ op: gte # >=
+ value: 0.8 # 80% must pass
+ pass_op: gte # Sample passes if >=
+ pass_value: 0.7 # This threshold (70%)
+```
+
+Default behavior:
+- If `metric` is `avg_score`: samples pass if score `>=` the gate value
+- If `metric` is `accuracy`: samples pass if score `>= 1.0` (perfect)
+
+## Examples
+
+### Require 80% Average Score
+
+```yaml
+gate:
+ metric_key: quality # Check quality grader
+ metric: avg_score # Use average
+ op: gte # >=
+ value: 0.8 # 80% average
+```
+
+Passes if the average score across all samples is `>= 0.8`
+
+### Require 90% Pass Rate (Perfect Scores)
+
+```yaml
+gate:
+ metric_key: accuracy # Check accuracy grader
+ metric: accuracy # Use pass rate
+ op: gte # >=
+ value: 0.9 # 90% must pass (default: score >= 1.0 to pass)
+```
+
+Passes if 90% of samples have score = 1.0
+
+### Require 75% Pass Rate (Score `>= 0.7`)
+
+```yaml
+gate:
+ metric_key: quality # Check quality grader
+ metric: accuracy # Use pass rate
+ op: gte # >=
+ value: 0.75 # 75% must pass
+ pass_op: gte # Sample passes if >=
+ pass_value: 0.7 # 70% threshold per sample
+```
+
+Passes if 75% of samples have score `>= 0.7`
+
+### Maximum Error Rate
+
+```yaml
+gate:
+ metric_key: quality # Check quality grader
+ metric: accuracy # Use pass rate
+ op: gte # >=
+ value: 0.95 # 95% must pass (allows 5% failures)
+ pass_op: gt # Sample passes if >
+ pass_value: 0.0 # 0.0 (any non-zero score)
+```
+
+Allows up to 5% failures.
+
+### Exact Pass Rate
+
+```yaml
+gate:
+ metric_key: quality # Check quality grader
+ metric: accuracy # Use pass rate
+ op: eq # Exactly equal
+ value: 1.0 # 100% (all samples must pass)
+```
+
+All samples must pass.
+
+## Multi-Metric Gating
+
+When you have multiple graders, you can only gate on one metric:
+
+```yaml
+graders:
+ accuracy: # First metric
+ kind: tool
+ function: exact_match
+ extractor: last_assistant
+
+ completeness: # Second metric
+ kind: rubric
+ prompt_path: completeness.txt
+ model: gpt-4o-mini
+ extractor: last_assistant
+
+gate:
+ metric_key: accuracy # Only gate on accuracy (completeness still computed)
+ metric: avg_score # Use average
+ op: gte # >=
+ value: 0.8 # 80% threshold
+```
+
+The evaluation passes/fails based on the gated metric, but results include scores for all metrics.
+
+## Understanding avg_score vs accuracy
+
+### avg_score
+- Arithmetic mean of all scores
+- Sensitive to partial credit
+- Good for continuous evaluation
+
+Example:
+- Scores: [1.0, 0.8, 0.6]
+- avg_score = (1.0 + 0.8 + 0.6) / 3 = 0.8
+
+### accuracy
+- Percentage of samples meeting a threshold
+- Binary pass/fail per sample
+- Good for strict requirements
+
+Example:
+- Scores: [1.0, 0.8, 0.6]
+- pass_value: 0.7
+- Passing: [1.0, 0.8] = 2 out of 3
+- accuracy = 2/3 = 0.667 (66.7%)
+
+## Errors and Attempted Samples
+
+If a sample fails (error during evaluation), it:
+- Gets a score of 0.0
+- Counts toward `total` but not `total_attempted`
+- Included in `avg_score_total` but not `avg_score_attempted`
+
+You can gate on either:
+- `avg_score_total`: Includes errors as 0.0
+- `avg_score_attempted`: Excludes errors (only successfully attempted samples)
+
+<Note>
+**Note**: The `metric` field currently only supports `avg_score` and `accuracy`. By default, gates use `avg_score_attempted`.
+</Note>
+
+## Gate Results
+
+After evaluation, you'll see:
+
+```text
+✓ PASSED (2.25/3.00 avg, 75.0% pass rate)
+```
+
+or
+
+```text
+✗ FAILED (1.80/3.00 avg, 60.0% pass rate)
+```
+
+The evaluation exit code reflects the gate result:
+- 0: Passed
+- 1: Failed
+
+## Advanced Gating
+
+For complex gating logic (e.g., "pass if accuracy `>= 80%` OR avg_score `>= 0.9`"), you'll need to:
+1. Run evaluation with one gate
+2. Examine the results JSON
+3. Apply custom logic in a post-processing script
+
+## Next Steps
+
+- [Understanding Results](/evals/results-metrics/understanding-results) - Interpreting evaluation output
+- [Multi-Metric Evaluation](/evals/graders/multi-metric-grading) - Using multiple graders
+- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete gate configuration
diff --git a/fern/pages/evals/concepts/graders.mdx b/fern/pages/evals/concepts/graders.mdx
new file mode 100644
index 000000000..bc0e498e0
--- /dev/null
+++ b/fern/pages/evals/concepts/graders.mdx
@@ -0,0 +1,330 @@
+# Graders
+
+**Graders** are the scoring functions that evaluate agent responses. They take the extracted submission (from an extractor) and assign a score between 0.0 (complete failure) and 1.0 (perfect success).
+
+<Note>
+**Quick overview:**
+- **Two types**: Tool graders (deterministic Python functions) and Rubric graders (LLM-as-judge)
+- **Built-in functions**: exact_match, contains, regex_match, ascii_printable_only
+- **Custom graders**: Write your own grading logic
+- **Multi-metric**: Combine multiple graders in one suite
+- **Flexible extraction**: Each grader can use a different extractor
+</Note>
+
+**When to use each:**
+- **Tool graders**: Fast, deterministic, free - perfect for exact matching, patterns, tool validation
+- **Rubric graders**: Flexible, subjective, costs API calls - ideal for quality, creativity, nuanced evaluation
+
+Graders evaluate agent responses and assign scores between 0.0 (complete failure) and 1.0 (perfect success).
+
+## Grader Types
+
+There are two types of graders:
+
+### Tool Graders
+
+Python functions that programmatically compare the submission to ground truth or apply deterministic checks.
+
+```yaml
+graders:
+ accuracy:
+ kind: tool # Deterministic grading
+ function: exact_match # Built-in grading function
+ extractor: last_assistant # Use final agent response
+```
+
+Best for:
+- Exact matching
+- Pattern checking
+- Tool call validation
+- Deterministic criteria
+
+### Rubric Graders
+
+LLM-as-judge evaluation using custom prompts and criteria. Can use either direct LLM API calls or a Letta agent as the judge.
+
+**Standard rubric grading (LLM API):**
+```yaml
+graders:
+ quality:
+ kind: rubric # LLM-as-judge
+ prompt_path: rubric.txt # Custom evaluation criteria
+ model: gpt-4o-mini # Judge model
+ extractor: last_assistant # What to evaluate
+```
+
+**Agent-as-judge (Letta agent):**
+```yaml
+graders:
+ agent_judge:
+ kind: rubric # Still "rubric" kind
+ agent_file: judge.af # Judge agent with submit_grade tool
+ prompt_path: rubric.txt # Evaluation criteria
+ extractor: last_assistant # What to evaluate
+```
+
+Best for:
+- Subjective quality assessment
+- Open-ended responses
+- Nuanced evaluation
+- Complex criteria
+- Judges that need tools (when using agent-as-judge)
+
+## Built-in Tool Graders
+
+### exact_match
+
+Checks if submission exactly matches ground truth (case-sensitive, whitespace-trimmed).
+
+```yaml
+graders:
+ accuracy:
+ kind: tool
+ function: exact_match # Case-sensitive, whitespace-trimmed
+ extractor: last_assistant # Extract final response
+```
+
+Requires: `ground_truth` in dataset
+
+Score: 1.0 if exact match, 0.0 otherwise
+
+### contains
+
+Checks if submission contains ground truth (case-insensitive).
+
+```yaml
+graders:
+ contains_answer:
+ kind: tool
+ function: contains # Case-insensitive substring match
+ extractor: last_assistant # Search in final response
+```
+
+Requires: `ground_truth` in dataset
+
+Score: 1.0 if found, 0.0 otherwise
+
+### regex_match
+
+Checks if submission matches a regex pattern in ground truth.
+
+```yaml
+graders:
+ pattern:
+ kind: tool
+ function: regex_match # Pattern matching
+ extractor: last_assistant # Check final response
+```
+
+Dataset sample:
+```json
+{"input": "Generate a UUID", "ground_truth": "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"}
+```
+
+Score: 1.0 if pattern matches, 0.0 otherwise
+
+### ascii_printable_only
+
+Validates that all characters are printable ASCII (useful for ASCII art, formatted output).
+
+```yaml
+graders:
+ ascii_check:
+ kind: tool
+ function: ascii_printable_only # Validate ASCII characters
+ extractor: last_assistant # Check final response
+```
+
+Does not require ground truth.
+
+Score: 1.0 if all characters are printable ASCII, 0.0 if any non-printable characters found
+
+## Rubric Graders
+
+Rubric graders use an LLM to evaluate responses based on custom criteria.
+
+### Basic Configuration
+
+```yaml
+graders:
+ quality:
+ kind: rubric # LLM-as-judge
+ prompt_path: quality_rubric.txt # Evaluation criteria
+ model: gpt-4o-mini # Judge model
+ temperature: 0.0 # Deterministic
+ extractor: last_assistant # What to evaluate
+```
+
+### Rubric Prompt Format
+
+Your rubric file should describe the evaluation criteria. Use placeholders:
+
+- `{input}`: The original input from the dataset
+- `{submission}`: The extracted agent response
+- `{ground_truth}`: Ground truth from dataset (if available)
+
+Example `quality_rubric.txt`:
+```
+Evaluate the response for:
+1. Accuracy: Does it correctly answer the question?
+2. Completeness: Is the answer thorough?
+3. Clarity: Is it well-explained?
+
+Input: {input}
+Expected: {ground_truth}
+Response: {submission}
+
+Score from 0.0 to 1.0 where:
+- 1.0: Perfect response
+- 0.75: Good with minor issues
+- 0.5: Acceptable but incomplete
+- 0.25: Poor quality
+- 0.0: Completely wrong
+```
+
+### Inline Prompt
+
+Instead of a file, you can include the prompt inline:
+
+```yaml
+graders:
+ quality:
+ kind: rubric # LLM-as-judge
+ prompt: | # Inline prompt instead of file
+ Evaluate the creativity and originality of the response.
+ Score 1.0 for highly creative, 0.0 for generic or unoriginal.
+ model: gpt-4o-mini # Judge model
+ extractor: last_assistant # What to evaluate
+```
+
+### Model Configuration
+
+```yaml
+graders:
+ quality:
+ kind: rubric
+ prompt_path: rubric.txt # Evaluation criteria
+ model: gpt-4o-mini # Judge model
+ temperature: 0.0 # Deterministic (0.0-2.0)
+ provider: openai # LLM provider (default: openai)
+ max_retries: 5 # API retry attempts
+ timeout: 120.0 # Request timeout in seconds
+```
+
+Supported providers:
+- `openai` (default)
+
+Models:
+- Any OpenAI-compatible model
+- Special handling for reasoning models (o1, o3) - temperature automatically adjusted to 1.0
+
+### Structured Output
+
+Rubric graders use JSON mode to get structured responses:
+
+```json
+{
+ "score": 0.85,
+ "rationale": "The response is accurate and complete but could be more concise."
+}
+```
+
+The score is validated to be between 0.0 and 1.0.
+
+## Multi-Metric Configuration
+
+Evaluate multiple aspects in one suite:
+
+```yaml
+graders:
+ accuracy: # Tool grader for factual correctness
+ kind: tool
+ function: contains
+ extractor: last_assistant
+
+ completeness: # Rubric grader for thoroughness
+ kind: rubric
+ prompt_path: completeness_rubric.txt
+ model: gpt-4o-mini
+ extractor: last_assistant
+
+ tool_usage: # Tool grader for tool call validation
+ kind: tool
+ function: exact_match
+ extractor: tool_arguments # Extract tool call args
+ extractor_config:
+ tool_name: search # Which tool to check
+```
+
+Each grader can use a different extractor.
+
+## Extractor Configuration
+
+Every grader must specify an `extractor` to select what to grade:
+
+```yaml
+graders:
+ my_metric:
+ kind: tool
+ function: contains # Grading function
+ extractor: last_assistant # What to extract and grade
+```
+
+Some extractors need additional configuration:
+
+```yaml
+graders:
+ tool_check:
+ kind: tool
+ function: contains # Check if ground truth in tool args
+ extractor: tool_arguments # Extract tool call arguments
+ extractor_config: # Configuration for this extractor
+ tool_name: search # Which tool to extract from
+```
+
+See [Extractors](/evals/core-concepts/extractors) for all available extractors.
+
+## Custom Graders
+
+You can write custom grading functions. See [Custom Graders](/evals/advanced/custom-graders) for details.
+
+## Grader Selection Guide
+
+| Use Case | Recommended Grader |
+|----------|-------------------|
+| Exact answer matching | `exact_match` |
+| Keyword checking | `contains` |
+| Pattern validation | `regex_match` |
+| Tool call validation | `exact_match` with `tool_arguments` extractor |
+| Quality assessment | Rubric grader |
+| Creativity evaluation | Rubric grader |
+| Format checking | Custom tool grader |
+| Multi-criteria evaluation | Multiple graders |
+
+## Score Interpretation
+
+All scores are between 0.0 and 1.0:
+
+- **1.0**: Perfect - meets all criteria
+- **0.75-0.99**: Good - minor issues
+- **0.5-0.74**: Acceptable - notable gaps
+- **0.25-0.49**: Poor - major problems
+- **0.0-0.24**: Failed - did not meet criteria
+
+Tool graders typically return binary scores (0.0 or 1.0), while rubric graders can return any value in the range.
+
+## Error Handling
+
+If grading fails (e.g., network error, invalid format):
+- Score is set to 0.0
+- Rationale includes error message
+- Metadata includes error details
+
+This ensures evaluations can continue even with individual failures.
+
+## Next Steps
+
+- [Tool Graders](/evals/graders/tool-graders) - Built-in and custom functions
+- [Rubric Graders](/evals/graders/rubric-graders) - LLM-as-judge details
+- [Multi-Metric Evaluation](/evals/graders/multi-metric-grading) - Using multiple graders
+- [Extractors](/evals/core-concepts/extractors) - Selecting what to grade
diff --git a/fern/pages/evals/concepts/overview.mdx b/fern/pages/evals/concepts/overview.mdx
new file mode 100644
index 000000000..9a83c0fac
--- /dev/null
+++ b/fern/pages/evals/concepts/overview.mdx
@@ -0,0 +1,207 @@
+# Core Concepts
+
+Understanding how Letta Evals works and what makes it different.
+
+<Note>
+**Just want to run an eval?** Skip to [Getting Started](/evals/get-started/getting-started) for a hands-on quickstart.
+</Note>
+
+## Built for Stateful Agents
+
+Letta Evals is a testing framework specifically designed for agents that maintain state. Unlike traditional eval frameworks built for simple input-output models, Letta Evals understands that agents:
+
+- Maintain memory across conversations
+- Use tools and external functions
+- Evolve their behavior based on interactions
+- Have persistent context and state
+
+This means you can test aspects of your agent that other frameworks can't: memory updates, multi-turn conversations, tool usage patterns, and state evolution over time.
+
+## The Evaluation Flow
+
+Every evaluation follows this flow:
+
+**Dataset → Target (Agent) → Extractor → Grader → Gate → Result**
+
+1. **Dataset**: Your test cases (questions, scenarios, expected outputs)
+2. **Target**: The agent being evaluated
+3. **Extractor**: Pulls out the relevant information from the agent's response
+4. **Grader**: Scores the extracted information
+5. **Gate**: Pass/fail criteria for the overall evaluation
+6. **Result**: Metrics, scores, and detailed results
+
+### What You Can Test
+
+With Letta Evals, you can test aspects of agents that traditional frameworks can't:
+
+- **Memory updates**: Did the agent correctly remember the user's name?
+- **Multi-turn conversations**: Can the agent maintain context across multiple exchanges?
+- **Tool usage**: Does the agent call the right tools with the right arguments?
+- **State evolution**: How does the agent's internal state change over time?
+
+<Note>
+**Example: Testing Memory Updates**
+
+```yaml
+graders:
+ memory_check:
+ kind: tool # Deterministic grading
+ function: contains # Check if ground_truth in extracted content
+ extractor: memory_block # Extract from agent memory (not just response!)
+ extractor_config:
+ block_label: human # Which memory block to check
+```
+
+Dataset:
+```jsonl
+{"input": "Please remember that I like bananas.", "ground_truth": "bananas"}
+```
+
+This doesn't just check if the agent responded correctly - it verifies the agent actually stored "bananas" in its memory block. Traditional eval frameworks can't inspect agent state like this.
+</Note>
+
+## Why Evals Matter
+
+AI agents are complex systems that can behave unpredictably. Without systematic evaluation, you can't:
+- **Know if changes improve or break your agent** - Did that prompt tweak help or hurt?
+- **Prevent regressions** - Catch when "fixes" break existing functionality
+- **Compare approaches objectively** - Which model works better for your use case?
+- **Build confidence before deployment** - Ensure quality before shipping to users
+- **Track improvement over time** - Measure progress as you iterate
+
+Manual testing doesn't scale. Evals let you test hundreds of scenarios in minutes.
+
+## What Evals Are Useful For
+
+### 1. Development & Iteration
+- Test prompt changes instantly across your entire test suite
+- Experiment with different models and compare results
+- Validate that new features work as expected
+
+### 2. Quality Assurance
+- Prevent regressions when modifying agent behavior
+- Ensure agents handle edge cases correctly
+- Verify tool usage and memory updates
+
+### 3. Model Selection
+- Compare GPT-4 vs Claude vs other models on your specific use case
+- Test different model configurations (temperature, system prompts, etc.)
+- Find the right cost/performance tradeoff
+
+### 4. Benchmarking
+- Measure agent performance on standard tasks
+- Track improvements over time
+- Share reproducible results with your team
+
+### 5. Production Readiness
+- Validate agents meet quality thresholds before deployment
+- Run continuous evaluation in CI/CD pipelines
+- Monitor production agent quality
+
+## How Letta Evals Works
+
+Letta Evals is built around a few key concepts that work together to create a flexible evaluation framework.
+
+## Key Components
+
+### Suite
+
+An **evaluation suite** is a complete test configuration defined in a YAML file. It ties together:
+- Which dataset to use
+- Which agent to test
+- How to grade responses
+- What criteria determine pass/fail
+
+Think of a suite as a reusable test specification.
+
+### Dataset
+
+A **dataset** is a JSONL file where each line represents one test case. Each sample has:
+- An input (what to ask the agent)
+- Optional ground truth (the expected answer)
+- Optional metadata (tags, custom fields)
+
+### Target
+
+The **target** is what you're evaluating. Currently, this is a Letta agent, specified by:
+- An agent file (.af)
+- An existing agent ID
+- A Python script that creates agents programmatically
+
+### Trajectory
+
+A **trajectory** is the complete conversation history from one test case. It's a list of turns, where each turn contains a list of Letta messages (assistant messages, tool calls, tool returns, etc.).
+
+### Extractor
+
+An **extractor** determines what part of the trajectory to evaluate. For example:
+- The last thing the agent said
+- All tool calls made
+- Content from agent memory
+- Text matching a pattern
+
+### Grader
+
+A **grader** scores how well the agent performed. There are two types:
+- **Tool graders**: Python functions that compare submission to ground truth
+- **Rubric graders**: LLM judges that evaluate based on custom criteria
+
+### Gate
+
+A **gate** is the pass/fail threshold for your evaluation. It compares aggregate metrics (like average score or pass rate) against a target value.
+
+## Multi-Metric Evaluation
+
+You can define multiple graders in one suite to evaluate different aspects:
+
+```yaml
+graders:
+ accuracy: # Check if answer is correct
+ kind: tool
+ function: exact_match
+ extractor: last_assistant # Use final response
+
+ tool_usage: # Check if agent called the right tool
+ kind: tool
+ function: contains
+ extractor: tool_arguments # Extract tool call args
+ extractor_config:
+ tool_name: search # From search tool
+```
+
+The gate can check any of these metrics:
+
+```yaml
+gate:
+ metric_key: accuracy # Gate on accuracy (tool_usage still computed)
+ op: gte # >=
+ value: 0.8 # 80% threshold
+```
+
+## Score Normalization
+
+All scores are normalized to the range [0.0, 1.0]:
+- 0.0 = complete failure
+- 1.0 = perfect success
+- Values in between = partial credit
+
+This allows different grader types to be compared and combined.
+
+## Aggregate Metrics
+
+Individual sample scores are aggregated in two ways:
+
+1. **Average Score**: Mean of all scores (0.0 to 1.0)
+2. **Accuracy/Pass Rate**: Percentage of samples passing a threshold
+
+You can gate on either metric type.
+
+## Next Steps
+
+Dive deeper into each concept:
+- [Suites](/evals/core-concepts/suites) - Suite configuration in detail
+- [Datasets](/evals/core-concepts/datasets) - Creating effective test datasets
+- [Targets](/evals/core-concepts/targets) - Agent configuration options
+- [Graders](/evals/core-concepts/graders) - Understanding grader types
+- [Extractors](/evals/core-concepts/extractors) - Extraction strategies
+- [Gates](/evals/core-concepts/gates) - Setting pass/fail criteria
diff --git a/fern/pages/evals/concepts/suites.mdx b/fern/pages/evals/concepts/suites.mdx
new file mode 100644
index 000000000..132d46c35
--- /dev/null
+++ b/fern/pages/evals/concepts/suites.mdx
@@ -0,0 +1,275 @@
+# Suites
+
+A **suite** is a YAML configuration file that defines a complete evaluation specification. It's the central piece that ties together your dataset, target agent, grading criteria, and pass/fail thresholds.
+
+<Note>
+**Quick overview:**
+- **Single file defines everything**: Dataset, agent, graders, and success criteria all in one YAML
+- **Reusable and shareable**: Version control your evaluation specs alongside your code
+- **Multi-metric support**: Evaluate multiple aspects (accuracy, quality, tool usage) in one run
+- **Multi-model testing**: Run the same suite across different LLM models
+- **Flexible filtering**: Test subsets using tags or sample limits
+</Note>
+
+**Typical workflow:**
+1. Create a suite YAML defining what and how to test
+2. Run `letta-evals run suite.yaml`
+3. Review results showing scores for each metric
+4. Suite passes or fails based on gate criteria
+
+An evaluation suite is a YAML configuration file that defines a complete test specification.
+
+## Basic Structure
+
+```yaml
+name: my-evaluation # Suite identifier
+description: Optional description of what this tests # Human-readable explanation
+dataset: path/to/dataset.jsonl # Test cases
+
+target: # What agent to evaluate
+ kind: agent
+ agent_file: agent.af # Agent to test
+ base_url: http://localhost:8283 # Letta server
+
+graders: # How to evaluate responses
+ my_metric:
+ kind: tool # Deterministic grading
+ function: exact_match # Grading function
+ extractor: last_assistant # What to extract from agent response
+
+gate: # Pass/fail criteria
+ metric_key: my_metric # Which metric to check
+ op: gte # Greater than or equal
+ value: 0.8 # 80% threshold
+```
+
+## Required Fields
+
+### name
+The name of your evaluation suite. Used in output and results.
+
+```yaml
+name: question-answering-eval
+```
+
+### dataset
+Path to the JSONL or CSV dataset file. Can be relative (to the suite YAML) or absolute.
+
+```yaml
+dataset: ./datasets/qa.jsonl # Relative to suite YAML location
+```
+
+### target
+Specifies what agent to evaluate. See [Targets](/evals/core-concepts/targets) for details.
+
+### graders
+One or more graders to evaluate agent performance. See [Graders](/evals/core-concepts/graders) for details.
+
+### gate
+Pass/fail criteria. See [Gates](/evals/core-concepts/gates) for details.
+
+## Optional Fields
+
+### description
+A human-readable description of what this suite tests:
+
+```yaml
+description: Tests the agent's ability to answer factual questions accurately
+```
+
+### max_samples
+Limit the number of samples to evaluate (useful for quick tests):
+
+```yaml
+max_samples: 10 # Only evaluate first 10 samples
+```
+
+### sample_tags
+Filter samples by tags (only evaluate samples with these tags):
+
+```yaml
+sample_tags: [math, easy] # Only samples tagged with "math" AND "easy"
+```
+
+Dataset samples can include tags:
+```jsonl
+{"input": "What is 2+2?", "ground_truth": "4", "tags": ["math", "easy"]}
+```
+
+### num_runs
+Number of times to run the entire evaluation suite (useful for testing non-deterministic behavior):
+
+```yaml
+num_runs: 5 # Run the evaluation 5 times
+```
+
+Default: 1
+
+### setup_script
+Path to a Python script with a setup function to run before evaluation:
+
+```yaml
+setup_script: setup.py:prepare_environment # script.py:function_name
+```
+
+The setup function should have this signature:
+```python
+def prepare_environment(suite: SuiteSpec) -> None:
+ # Setup code here
+ pass
+```
+
+## Path Resolution
+
+Paths in the suite YAML are resolved relative to the YAML file location:
+
+```
+project/
+├── suite.yaml
+├── dataset.jsonl
+└── agents/
+ └── my_agent.af
+```
+
+```yaml
+# In suite.yaml
+dataset: dataset.jsonl # Resolves to project/dataset.jsonl
+target:
+ agent_file: agents/my_agent.af # Resolves to project/agents/my_agent.af
+```
+
+Absolute paths are used as-is.
+
+## Multi-Grader Suites
+
+You can evaluate multiple metrics in one suite:
+
+```yaml
+graders:
+ accuracy: # Check if answer is correct
+ kind: tool
+ function: exact_match
+ extractor: last_assistant
+
+ completeness: # LLM judges response quality
+ kind: rubric
+ prompt_path: rubrics/completeness.txt
+ model: gpt-4o-mini
+ extractor: last_assistant
+
+ tool_usage: # Verify correct tool was called
+ kind: tool
+ function: contains
+ extractor: tool_arguments # Extract tool call arguments
+```
+
+The gate can check any of these metrics:
+
+```yaml
+gate:
+ metric_key: accuracy # Gate on accuracy metric (others still computed)
+ op: gte # Greater than or equal
+ value: 0.9 # 90% threshold
+```
+
+Results will include scores for all graders, even if you only gate on one.
+
+## Examples
+
+### Simple Tool Grader Suite
+
+```yaml
+name: basic-qa # Suite name
+dataset: questions.jsonl # Test questions
+
+target:
+ kind: agent
+ agent_file: qa_agent.af # Pre-configured agent
+ base_url: http://localhost:8283 # Local server
+
+graders:
+ accuracy: # Single metric
+ kind: tool # Deterministic grading
+ function: contains # Check if ground truth is in response
+ extractor: last_assistant # Use final agent message
+
+gate:
+ metric_key: accuracy # Gate on this metric
+ op: gte # Must be >=
+ value: 0.75 # 75% to pass
+```
+
+### Rubric Grader Suite
+
+```yaml
+name: quality-eval # Quality evaluation
+dataset: prompts.jsonl # Test prompts
+
+target:
+ kind: agent
+ agent_id: existing-agent-123 # Use existing agent
+ base_url: https://api.letta.com # Letta Cloud
+
+graders:
+ quality: # LLM-as-judge metric
+ kind: rubric # Subjective evaluation
+ prompt_path: quality_rubric.txt # Rubric template
+ model: gpt-4o-mini # Judge model
+ temperature: 0.0 # Deterministic
+ extractor: last_assistant # Evaluate final response
+
+gate:
+ metric_key: quality # Gate on this metric
+ metric: avg_score # Use average score
+ op: gte # Must be >=
+ value: 0.7 # 70% to pass
+```
+
+### Multi-Model Suite
+
+Test the same agent configuration across different models:
+
+```yaml
+name: model-comparison # Compare model performance
+dataset: test.jsonl # Same test for all models
+
+target:
+ kind: agent
+ agent_file: agent.af # Same agent configuration
+ base_url: http://localhost:8283 # Local server
+ model_configs: [gpt-4o-mini, claude-3-5-sonnet] # Test both models
+
+graders:
+ accuracy: # Single metric for comparison
+ kind: tool
+ function: exact_match
+ extractor: last_assistant
+
+gate:
+ metric_key: accuracy # Both models must pass this
+ op: gte # Must be >=
+ value: 0.8 # 80% threshold
+```
+
+Results will show per-model metrics.
+
+## Validation
+
+Validate your suite configuration before running:
+
+```bash
+letta-evals validate suite.yaml
+```
+
+This checks:
+- Required fields are present
+- Paths exist
+- Configuration is valid
+- Grader/extractor combinations are compatible
+
+## Next Steps
+
+- [Dataset Configuration](/evals/core-concepts/datasets)
+- [Target Configuration](/evals/core-concepts/targets)
+- [Grader Configuration](/evals/core-concepts/graders)
+- [Gate Configuration](/evals/core-concepts/gates)
diff --git a/fern/pages/evals/concepts/targets.mdx b/fern/pages/evals/concepts/targets.mdx
new file mode 100644
index 000000000..f5dfb1d23
--- /dev/null
+++ b/fern/pages/evals/concepts/targets.mdx
@@ -0,0 +1,329 @@
+# Targets
+
+A **target** is the agent you're evaluating. In Letta Evals, the target configuration determines how agents are created, accessed, and tested.
+
+<Note>
+**Quick overview:**
+- **Three ways to specify agents**: agent file (`.af`), existing agent ID, or programmatic creation script
+- **Critical distinction**: `agent_file`/`agent_script` create fresh agents per sample (isolated tests), while `agent_id` uses one agent for all samples (stateful conversation)
+- **Multi-model support**: Test the same agent configuration across different LLM models
+- **Flexible connection**: Connect to local Letta servers or Letta Cloud
+</Note>
+
+**When to use each approach:**
+- `agent_file` - Pre-configured agents saved as `.af` files (most common)
+- `agent_id` - Testing existing agents or multi-turn conversations with state
+- `agent_script` - Dynamic agent creation with per-sample customization
+
+The target configuration specifies how to create or access the agent for evaluation.
+
+## Target Configuration
+
+All targets have a `kind` field (currently only `agent` is supported):
+
+```yaml
+target:
+ kind: agent # Currently only "agent" is supported
+ # ... agent-specific configuration
+```
+
+## Agent Sources
+
+You must specify exactly ONE of these:
+
+### agent_file
+
+Path to a `.af` (Agent File) to upload:
+
+```yaml
+target:
+ kind: agent
+ agent_file: path/to/agent.af # Path to .af file
+ base_url: http://localhost:8283 # Letta server URL
+```
+
+The agent file will be uploaded to the Letta server and a new agent created for the evaluation.
+
+### agent_id
+
+ID of an existing agent on the server:
+
+```yaml
+target:
+ kind: agent
+ agent_id: agent-123-abc # ID of existing agent
+ base_url: http://localhost:8283 # Letta server URL
+```
+
+<Warning>
+**Modifies agent in-place:** Using `agent_id` will modify your agent's state, memory, and message history during evaluation. The same agent instance is used for all samples, processing them sequentially. **Do not use production agents or agents you don't want to modify.** Use `agent_file` or `agent_script` for reproducible, isolated testing.
+</Warning>
+
+### agent_script
+
+Path to a Python script with an agent factory function for programmatic agent creation:
+
+```yaml
+target:
+ kind: agent
+ agent_script: create_agent.py:create_inventory_agent # script.py:function_name
+ base_url: http://localhost:8283 # Letta server URL
+```
+
+Format: `path/to/script.py:function_name`
+
+The function must be decorated with `@agent_factory` and have the signature `async (client: AsyncLetta, sample: Sample) -> str`:
+
+```python
+from letta_client import AsyncLetta, CreateBlock
+from letta_evals.decorators import agent_factory
+from letta_evals.models import Sample
+
+@agent_factory
+async def create_inventory_agent(client: AsyncLetta, sample: Sample) -> str:
+ """Create and return agent ID for this sample."""
+ # Access custom arguments from the dataset
+ item = sample.agent_args.get("item", {})
+
+ # Create agent with sample-specific configuration
+ agent = await client.agents.create(
+ name="inventory-assistant",
+ memory_blocks=[
+ CreateBlock(
+ label="item_context",
+ value=f"Item: {item.get('name', 'Unknown')}"
+ )
+ ],
+ agent_type="letta_v1_agent",
+ model="openai/gpt-4.1-mini",
+ embedding="openai/text-embedding-3-small",
+ )
+
+ return agent.id
+```
+
+**Key features:**
+- Creates a fresh agent for each sample
+- Can customize agents using `sample.agent_args` from the dataset
+- Allows testing agent creation logic itself
+- Useful when you don't have pre-saved agent files
+
+**When to use:**
+- Testing agent creation workflows
+- Dynamic per-sample agent configuration
+- Agents that need sample-specific memory or tools
+- Programmatic agent testing
+
+## Connection Configuration
+
+### base_url
+
+Letta server URL:
+
+```yaml
+target:
+ base_url: http://localhost:8283 # Local Letta server
+ # or
+ base_url: https://api.letta.com # Letta Cloud
+```
+
+Default: `http://localhost:8283`
+
+### api_key
+
+API key for authentication (required for Letta Cloud):
+
+```yaml
+target:
+ api_key: your-api-key-here # Required for Letta Cloud
+```
+
+Or set via environment variable:
+```bash
+export LETTA_API_KEY=your-api-key-here
+```
+
+### project_id
+
+Letta project ID (for Letta Cloud):
+
+```yaml
+target:
+ project_id: proj_abc123 # Letta Cloud project
+```
+
+Or set via environment variable:
+```bash
+export LETTA_PROJECT_ID=proj_abc123
+```
+
+### timeout
+
+Request timeout in seconds:
+
+```yaml
+target:
+ timeout: 300.0 # Request timeout (5 minutes)
+```
+
+Default: 300 seconds
+
+## Multi-Model Evaluation
+
+Test the same agent across different models:
+
+### model_configs
+
+List of model configuration names from JSON files:
+
+```yaml
+target:
+ kind: agent
+ agent_file: agent.af
+ model_configs: [gpt-4o-mini, claude-3-5-sonnet] # Test with both models
+```
+
+The evaluation will run once for each model config. Model configs are JSON files in `letta_evals/llm_model_configs/`.
+
+### model_handles
+
+List of model handles (cloud-compatible identifiers):
+
+```yaml
+target:
+ kind: agent
+ agent_file: agent.af
+ model_handles: ["openai/gpt-4o-mini", "anthropic/claude-3-5-sonnet"] # Cloud model identifiers
+```
+
+Use this for Letta Cloud deployments.
+
+<Warning>
+**Note**: You cannot specify both `model_configs` and `model_handles`.
+</Warning>
+
+## Complete Examples
+
+### Local Development
+
+```yaml
+target:
+ kind: agent
+ agent_file: ./agents/my_agent.af # Pre-configured agent
+ base_url: http://localhost:8283 # Local server
+```
+
+### Letta Cloud
+
+```yaml
+target:
+ kind: agent
+ agent_id: agent-cloud-123 # Existing cloud agent
+ base_url: https://api.letta.com # Letta Cloud
+ api_key: ${LETTA_API_KEY} # From environment variable
+ project_id: proj_abc # Your project ID
+```
+
+### Multi-Model Testing
+
+```yaml
+target:
+ kind: agent
+ agent_file: agent.af # Same agent configuration
+ base_url: http://localhost:8283 # Local server
+ model_configs: [gpt-4o-mini, gpt-4o, claude-3-5-sonnet] # Test 3 models
+```
+
+Results will include per-model metrics:
+```
+Model: gpt-4o-mini - Avg: 0.85, Pass: 85.0%
+Model: gpt-4o - Avg: 0.92, Pass: 92.0%
+Model: claude-3-5-sonnet - Avg: 0.88, Pass: 88.0%
+```
+
+### Programmatic Agent Creation
+
+```yaml
+target:
+ kind: agent
+ agent_script: setup.py:CustomAgentFactory # Programmatic creation
+ base_url: http://localhost:8283 # Local server
+```
+
+## Environment Variable Precedence
+
+Configuration values are resolved in this order (highest priority first):
+
+1. CLI arguments (`--api-key`, `--base-url`, `--project-id`)
+2. Suite YAML configuration
+3. Environment variables (`LETTA_API_KEY`, `LETTA_BASE_URL`, `LETTA_PROJECT_ID`)
+
+## Agent Lifecycle and Testing Behavior
+
+The way your agent is specified fundamentally changes how the evaluation runs:
+
+### With agent_file or agent_script: Independent Testing
+
+**Agent lifecycle:**
+1. A fresh agent instance is created for each sample
+2. Agent processes the sample input(s)
+3. Agent remains on the server after the sample completes
+
+**Testing behavior:** Each sample is an independent, isolated test. Agent state (memory, message history) does not carry over between samples. This enables parallel execution and ensures reproducible results.
+
+**Use cases:**
+- Testing how the agent responds to various independent inputs
+- Ensuring consistent behavior across different scenarios
+- Regression testing where each case should be isolated
+- Evaluating agent responses without prior context
+
+<Note>
+**Example:** If you have 10 test cases, 10 separate agent instances will be created (one per test case), and they can run in parallel.
+</Note>
+
+### With agent_id: Sequential Script Testing
+
+**Agent lifecycle:**
+1. The same agent instance is used for all samples
+2. Agent processes each sample in sequence
+3. Agent state persists throughout the entire evaluation
+
+**Testing behavior:** The dataset becomes a conversation script where each sample builds on previous ones. Agent memory and message history accumulate, and earlier interactions affect later responses. Samples must execute sequentially.
+
+**Use cases:**
+- Testing multi-turn conversations with context
+- Evaluating how agent memory evolves over time
+- Simulating a single user session with multiple interactions
+- Testing scenarios where context should accumulate
+
+<Note>
+**Example:** If you have 10 test cases, they all run against the same agent instance in order, with state carrying over between each test.
+</Note>
+
+### Critical Differences
+
+| Aspect | agent_file / agent_script | agent_id |
+|--------|---------------------------|----------|
+| **Agent instances** | New agent per sample | Same agent for all samples |
+| **State isolation** | Fully isolated | State carries over |
+| **Execution** | Can run in parallel | Must run sequentially |
+| **Memory** | Fresh for each sample | Accumulates across samples |
+| **Use case** | Independent test cases | Conversation scripts |
+| **Reproducibility** | Highly reproducible | Depends on execution order |
+
+<Tip>
+**Best practice:** Use `agent_file` or `agent_script` for most evaluations to ensure reproducible, isolated tests. Use `agent_id` only when you specifically need to test how agent state evolves across multiple interactions.
+</Tip>
+
+## Validation
+
+The runner validates:
+- Exactly one of `agent_file`, `agent_id`, or `agent_script` is specified
+- Agent files have `.af` extension
+- Agent script paths are valid
+
+## Next Steps
+
+- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete target configuration options
+- [Datasets](/evals/core-concepts/datasets) - Using agent_args for sample-specific configuration
+- [Getting Started](/evals/get-started/getting-started) - Complete tutorial with target examples
diff --git a/fern/pages/evals/configuration/suite-yaml.mdx b/fern/pages/evals/configuration/suite-yaml.mdx
new file mode 100644
index 000000000..440e44cfb
--- /dev/null
+++ b/fern/pages/evals/configuration/suite-yaml.mdx
@@ -0,0 +1,427 @@
+# Suite YAML Reference
+
+Complete reference for suite configuration files.
+
+A **suite** is a YAML file that defines an evaluation: what agent to test, what dataset to use, how to grade responses, and what criteria determine pass/fail. This is your evaluation specification.
+
+<Note>
+**Quick overview:**
+- **name**: Identifier for your evaluation
+- **dataset**: JSONL file with test cases
+- **target**: Which agent to evaluate (via file, ID, or script)
+- **graders**: How to score responses (tool or rubric graders)
+- **gate**: Pass/fail criteria
+</Note>
+
+See [Getting Started](/evals/get-started/getting-started) for a tutorial, or [Core Concepts](/evals/core-concepts/suites) for conceptual overview.
+
+## File Structure
+
+```yaml
+name: string (required)
+description: string (optional)
+dataset: path (required)
+max_samples: integer (optional)
+sample_tags: array (optional)
+num_runs: integer (optional)
+setup_script: string (optional)
+
+target: object (required)
+ kind: "agent"
+ base_url: string
+ api_key: string
+ timeout: float
+ project_id: string
+ agent_id: string (one of: agent_id, agent_file, agent_script)
+ agent_file: path
+ agent_script: string
+ model_configs: array
+ model_handles: array
+
+graders: object (required)
+ <metric_key>: object
+ kind: "tool" | "rubric"
+ display_name: string
+ extractor: string
+ extractor_config: object
+ # Tool grader fields
+ function: string
+ # Rubric grader fields (LLM API)
+ prompt: string
+ prompt_path: path
+ model: string
+ temperature: float
+ provider: string
+ max_retries: integer
+ timeout: float
+ rubric_vars: array
+ # Rubric grader fields (agent-as-judge)
+ agent_file: path
+ judge_tool_name: string
+
+gate: object (required)
+ metric_key: string
+ metric: "avg_score" | "accuracy"
+ op: "gte" | "gt" | "lte" | "lt" | "eq"
+ value: float
+ pass_op: "gte" | "gt" | "lte" | "lt" | "eq"
+ pass_value: float
+```
+
+## Top-Level Fields
+
+### name (required)
+
+Suite name, used in output and results.
+
+**Type**: string
+
+```yaml
+name: question-answering-eval
+```
+
+### description (optional)
+
+Human-readable description of what the suite tests.
+
+**Type**: string
+
+```yaml
+description: Tests agent's ability to answer factual questions accurately
+```
+
+### dataset (required)
+
+Path to JSONL dataset file. Relative paths are resolved from the suite YAML location.
+
+**Type**: path (string)
+
+```yaml
+dataset: ./datasets/qa.jsonl
+dataset: /absolute/path/to/dataset.jsonl
+```
+
+### max_samples (optional)
+
+Limit the number of samples to evaluate. Useful for quick tests.
+
+**Type**: integer | **Default**: All samples
+
+```yaml
+max_samples: 10 # Only evaluate first 10 samples
+```
+
+### sample_tags (optional)
+
+Filter samples by tags. Only samples with ALL specified tags are evaluated.
+
+**Type**: array of strings
+
+```yaml
+sample_tags: [math, easy] # Only samples tagged with both
+```
+
+### num_runs (optional)
+
+Number of times to run the evaluation suite.
+
+**Type**: integer | **Default**: 1
+
+```yaml
+num_runs: 5 # Run the evaluation 5 times
+```
+
+### setup_script (optional)
+
+Path to Python script with setup function.
+
+**Type**: string (format: `path/to/script.py:function_name`)
+
+```yaml
+setup_script: setup.py:prepare_environment
+```
+
+## target (required)
+
+Configuration for the agent being evaluated.
+
+### kind (required)
+
+Type of target. Currently only `"agent"` is supported.
+
+```yaml
+target:
+ kind: agent
+```
+
+### base_url (optional)
+
+Letta server URL. **Default**: `http://localhost:8283`
+
+```yaml
+target:
+ base_url: http://localhost:8283
+ # or
+ base_url: https://api.letta.com
+```
+
+### api_key (optional)
+
+API key for Letta authentication. Can also be set via `LETTA_API_KEY` environment variable.
+
+```yaml
+target:
+ api_key: your-api-key-here
+```
+
+### timeout (optional)
+
+Request timeout in seconds. **Default**: 300.0
+
+```yaml
+target:
+ timeout: 600.0 # 10 minutes
+```
+
+### Agent Source (required, pick one)
+
+Exactly one of these must be specified:
+
+#### agent_id
+
+ID of existing agent on the server.
+
+```yaml
+target:
+ agent_id: agent-123-abc
+```
+
+#### agent_file
+
+Path to `.af` agent file.
+
+```yaml
+target:
+ agent_file: ./agents/my_agent.af
+```
+
+#### agent_script
+
+Path to Python script with agent factory.
+
+```yaml
+target:
+ agent_script: factory.py:MyAgentFactory
+```
+
+See [Targets](/evals/core-concepts/targets) for details on agent sources.
+
+### model_configs (optional)
+
+List of model configuration names to test. Cannot be used with `model_handles`.
+
+```yaml
+target:
+ model_configs: [gpt-4o-mini, claude-3-5-sonnet]
+```
+
+### model_handles (optional)
+
+List of model handles for cloud deployments. Cannot be used with `model_configs`.
+
+```yaml
+target:
+ model_handles: ["openai/gpt-4o-mini", "anthropic/claude-3-5-sonnet"]
+```
+
+## graders (required)
+
+One or more graders, each with a unique key.
+
+### kind (required)
+
+Grader type: `"tool"` or `"rubric"`.
+
+```yaml
+graders:
+ my_metric:
+ kind: tool
+```
+
+### extractor (required)
+
+Name of the extractor to use.
+
+```yaml
+graders:
+ my_metric:
+ extractor: last_assistant
+```
+
+### Tool Grader Fields
+
+#### function (required for tool graders)
+
+Name of the grading function.
+
+```yaml
+graders:
+ accuracy:
+ kind: tool
+ function: exact_match
+```
+
+### Rubric Grader Fields
+
+#### prompt or prompt_path (required)
+
+Inline rubric prompt or path to rubric file.
+
+```yaml
+graders:
+ quality:
+ kind: rubric
+ prompt: |
+ Evaluate response quality from 0.0 to 1.0.
+```
+
+#### model (optional)
+
+LLM model for judging. **Default**: `gpt-4o-mini`
+
+```yaml
+graders:
+ quality:
+ kind: rubric
+ model: gpt-4o
+```
+
+#### temperature (optional)
+
+Temperature for LLM generation. **Default**: 0.0
+
+```yaml
+graders:
+ quality:
+ kind: rubric
+ temperature: 0.0
+```
+
+#### agent_file (agent-as-judge)
+
+Path to `.af` agent file to use as judge.
+
+```yaml
+graders:
+ agent_judge:
+ kind: rubric
+ agent_file: judge.af
+ prompt_path: rubric.txt
+```
+
+## gate (required)
+
+Pass/fail criteria for the evaluation.
+
+### metric_key (optional)
+
+Which grader to evaluate. If only one grader, this can be omitted.
+
+```yaml
+gate:
+ metric_key: accuracy
+```
+
+### metric (optional)
+
+Which aggregate to compare: `avg_score` or `accuracy`. **Default**: `avg_score`
+
+```yaml
+gate:
+ metric: avg_score
+```
+
+### op (required)
+
+Comparison operator: `gte`, `gt`, `lte`, `lt`, `eq`
+
+```yaml
+gate:
+ op: gte # Greater than or equal
+```
+
+### value (required)
+
+Threshold value for comparison (0.0 to 1.0).
+
+```yaml
+gate:
+ value: 0.8 # Require >= 0.8
+```
+
+## Complete Examples
+
+### Minimal Suite
+
+```yaml
+name: basic-eval
+dataset: dataset.jsonl
+
+target:
+ kind: agent
+ agent_file: agent.af
+
+graders:
+ accuracy:
+ kind: tool
+ function: exact_match
+ extractor: last_assistant
+
+gate:
+ op: gte
+ value: 0.8
+```
+
+### Multi-Metric Suite
+
+```yaml
+name: comprehensive-eval
+description: Tests accuracy and quality
+dataset: test_data.jsonl
+
+target:
+ kind: agent
+ agent_file: agent.af
+
+graders:
+ accuracy:
+ kind: tool
+ function: contains
+ extractor: last_assistant
+
+ quality:
+ kind: rubric
+ prompt_path: rubrics/quality.txt
+ model: gpt-4o-mini
+ extractor: last_assistant
+
+gate:
+ metric_key: accuracy
+ op: gte
+ value: 0.85
+```
+
+## Validation
+
+Validate your suite before running:
+
+```bash
+letta-evals validate suite.yaml
+```
+
+## Next Steps
+
+- [Targets](/evals/core-concepts/targets) - Understanding agent sources and configuration
+- [Graders](/evals/core-concepts/graders) - Tool graders vs rubric graders
+- [Extractors](/evals/core-concepts/extractors) - What to extract from agent responses
+- [Gates](/evals/core-concepts/gates) - Setting pass/fail criteria
diff --git a/fern/pages/evals/extractors/builtin.mdx b/fern/pages/evals/extractors/builtin.mdx
new file mode 100644
index 000000000..b8be5d486
--- /dev/null
+++ b/fern/pages/evals/extractors/builtin.mdx
@@ -0,0 +1,96 @@
+# Built-in Extractors
+
+Letta Evals provides a set of built-in extractors that cover the most common extraction needs.
+
+<Note>
+**What are extractors?** Extractors determine what part of an agent's response gets evaluated. They take the full conversation trajectory and extract just the piece you want to grade.
+</Note>
+
+## Common Extractors
+
+### last_assistant
+
+Extracts the last assistant message content.
+
+```yaml
+extractor: last_assistant # Most common - gets final response
+```
+
+### first_assistant
+
+Extracts the first assistant message content.
+
+```yaml
+extractor: first_assistant
+```
+
+### all_assistant
+
+Concatenates all assistant messages with a separator.
+
+```yaml
+extractor: all_assistant
+extractor_config:
+ separator: "\n\n" # Join messages with double newline
+```
+
+### pattern
+
+Extracts content matching a regex pattern.
+
+```yaml
+extractor: pattern
+extractor_config:
+ pattern: 'Result: (\d+)' # Regex pattern to match
+ group: 1 # Extract capture group 1
+```
+
+### tool_arguments
+
+Extracts arguments from a specific tool call.
+
+```yaml
+extractor: tool_arguments
+extractor_config:
+ tool_name: search # Which tool to extract from
+```
+
+### tool_output
+
+Extracts the return value from a specific tool call.
+
+```yaml
+extractor: tool_output
+extractor_config:
+ tool_name: search
+```
+
+### memory_block
+
+Extracts content from a specific memory block.
+
+```yaml
+extractor: memory_block
+extractor_config:
+ block_label: human # Which memory block to extract
+```
+
+<Warning>
+**Important**: This extractor requires the agent's final state, which adds overhead.
+</Warning>
+
+### after_marker
+
+Extracts content after a specific marker string.
+
+```yaml
+extractor: after_marker
+extractor_config:
+ marker: "ANSWER:"
+ include_marker: false
+```
+
+## Next Steps
+
+- [Custom Extractors](/evals/extractors/custom-extractors) - Write your own extractors
+- [Extractors Concept](/evals/core-concepts/extractors) - Understanding extractors
diff --git a/fern/pages/evals/extractors/custom.mdx b/fern/pages/evals/extractors/custom.mdx
new file mode 100644
index 000000000..1a16b4755
--- /dev/null
+++ b/fern/pages/evals/extractors/custom.mdx
@@ -0,0 +1,55 @@
+# Custom Extractors
+
+Create your own extractors to pull exactly what you need from agent trajectories.
+
+<Note>
+While built-in extractors cover common cases, custom extractors let you implement specialized extraction logic for your specific use case.
+</Note>
+
+## Why Custom Extractors?
+
+Use custom extractors when you need to:
+- **Extract structured data**: Parse JSON fields from agent responses
+- **Filter specific patterns**: Extract code blocks, URLs, or formatted content
+- **Combine data sources**: Merge information from multiple messages or memory blocks
+- **Count occurrences**: Track how many times something happened
+- **Complex logic**: Implement domain-specific extraction
+
+## Basic Structure
+
+```python
+from letta_evals.decorators import extractor
+from letta_client import LettaMessageUnion
+from typing import List
+
+@extractor
+def my_extractor(trajectory: List[List[LettaMessageUnion]], config: dict) -> str:
+ """Extract custom content from trajectory."""
+ # Your extraction logic here
+ return extracted_text
+```
+
+## Example: Extract Memory Insert
+
+```python
+from letta_evals.decorators import extractor
+
+@extractor
+def memory_insert_args(trajectory, config):
+ """Extract arguments from memory_insert tool calls."""
+ for turn in trajectory:
+ for message in turn:
+ if hasattr(message, 'tool_call') and message.tool_call:
+ if message.tool_call.name == "memory_insert":
+ return str(message.tool_call.arguments)
+ return ""
+```
+
+## Registration
+
+Custom extractors are automatically registered when you import them in your suite's setup script or custom evaluators file.
+
+## Next Steps
+
+- [Built-in Extractors](/evals/extractors/built-in-extractors) - Available extractors
+- [Extractors Concept](/evals/core-concepts/extractors) - Understanding extractors
diff --git a/fern/pages/evals/getting-started.mdx b/fern/pages/evals/getting-started.mdx
new file mode 100644
index 000000000..728bac79a
--- /dev/null
+++ b/fern/pages/evals/getting-started.mdx
@@ -0,0 +1,264 @@
+# Getting Started
+
+Run your first Letta agent evaluation in 5 minutes.
+
+## Prerequisites
+
+- Python 3.11 or higher
+- A running Letta server (local or Letta Cloud)
+- A Letta agent to test, either in agent file format or by ID (see [Targets](/evals/core-concepts/targets) for more details)
+
+## Installation
+
+```bash
+pip install letta-evals
+```
+
+Or with uv:
+
+```bash
+uv pip install letta-evals
+```
+
+## Getting an Agent to Test
+
+Export an existing agent to a file using the Letta SDK:
+
+```python
+from letta_client import Letta
+import os
+
+client = Letta(
+ base_url="http://localhost:8283", # or https://api.letta.com for Letta Cloud
+ token=os.getenv("LETTA_API_KEY") # required for Letta Cloud
+)
+
+# Export an agent to a file
+agent_file = client.agents.export_file(agent_id="agent-123")
+
+# Save to disk
+with open("my_agent.af", "w") as f:
+ f.write(agent_file)
+```
+
+Or export via the Agent Development Environment (ADE) by selecting "Export Agent".
+
+Then reference it in your suite:
+
+```yaml
+target:
+ kind: agent
+ agent_file: my_agent.af
+```
+
+<Note>
+**Other options:** You can also use existing agents by ID or programmatically generate agents. See [Targets](/evals/core-concepts/targets) for all agent configuration options.
+</Note>
+
+## Quick Start
+
+Let's create your first evaluation in 3 steps:
+
+### 1. Create a Test Dataset
+
+Create a file named `dataset.jsonl`:
+
+```jsonl
+{"input": "What's the capital of France?", "ground_truth": "Paris"}
+{"input": "Calculate 2+2", "ground_truth": "4"}
+{"input": "What color is the sky?", "ground_truth": "blue"}
+```
+
+Each line is a JSON object with:
+- `input`: The prompt to send to your agent
+- `ground_truth`: The expected answer (used for grading)
+
+<Note>
+`ground_truth` is optional for some graders (like rubric graders), but required for tool graders like `contains` and `exact_match`.
+</Note>
+
+Read more about [Datasets](/evals/core-concepts/datasets) for details on how to create your dataset.
+
+### 2. Create a Suite Configuration
+
+Create a file named `suite.yaml`:
+
+```yaml
+name: my-first-eval
+dataset: dataset.jsonl
+
+target:
+ kind: agent
+ agent_file: my_agent.af # Path to your agent file
+ base_url: http://localhost:8283 # Your Letta server
+
+graders:
+ quality:
+ kind: tool
+ function: contains # Check if response contains the ground truth
+ extractor: last_assistant # Use the last assistant message
+
+gate:
+ metric_key: quality
+ op: gte
+ value: 0.75 # Require 75% pass rate
+```
+
+The suite configuration defines:
+- The [dataset](/evals/core-concepts/datasets) to use
+- The [agent](/evals/core-concepts/targets) to test
+- The [graders](/evals/core-concepts/graders) to use
+- The [gate](/evals/core-concepts/gates) criteria
+
+Read more about [Suites](/evals/core-concepts/suites) for details on how to configure your evaluation.
+
+### 3. Run the Evaluation
+
+Run your evaluation with the following command:
+
+```bash
+letta-evals run suite.yaml
+```
+
+You'll see real-time progress as your evaluation runs:
+
+```
+Running evaluation: my-first-eval
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3/3 100%
+✓ PASSED (2.25/3.00 avg, 75.0% pass rate)
+```
+
+Read more about [CLI Commands](/evals/cli-reference/commands) for details about the available commands and options.
+
+## Understanding the Results
+
+The core evaluation flow is:
+
+**Dataset → Target (Agent) → Extractor → Grader → Gate → Result**
+
+The evaluation runner:
+1. Loads your dataset
+2. Sends each input to your agent (Target)
+3. Extracts the relevant information (using the Extractor)
+4. Grades the response (using the Grader function)
+5. Computes aggregate metrics
+6. Checks if metrics pass the Gate criteria
+
+The output shows:
+- **Average score**: Mean score across all samples
+- **Pass rate**: Percentage of samples that passed
+- **Gate status**: Whether the evaluation passed or failed overall
+
+## Next Steps
+
+Now that you've run your first evaluation, explore more advanced features:
+
+- [Core Concepts](/evals/core-concepts/concepts-overview) - Understand suites, datasets, graders, and extractors
+- [Grader Types](/evals/core-concepts/graders) - Learn about tool graders vs rubric graders
+- [Multi-Metric Evaluation](/evals/graders/multi-metric-grading) - Test multiple aspects simultaneously
+- [Custom Graders](/evals/advanced/custom-graders) - Write custom grading functions
+- [Multi-Turn Conversations](/evals/advanced/multi-turn-conversations) - Test conversational memory
+
+## Common Use Cases
+
+### Strict Answer Checking
+
+Use exact matching for cases where the answer must be precisely correct:
+
+```yaml
+graders:
+ accuracy:
+ kind: tool
+ function: exact_match
+ extractor: last_assistant
+```
+
+### Subjective Quality Evaluation
+
+Use an LLM judge to evaluate subjective qualities like helpfulness or tone:
+
+```yaml
+graders:
+ quality:
+ kind: rubric
+ prompt_path: rubric.txt
+ model: gpt-4o-mini
+ extractor: last_assistant
+```
+
+Then create `rubric.txt`:
+```
+Rate the helpfulness and accuracy of the response.
+- Score 1.0 if helpful and accurate
+- Score 0.5 if partially helpful
+- Score 0.0 if unhelpful or wrong
+```
+
+### Testing Tool Calls
+
+Verify that your agent calls specific tools with expected arguments:
+
+```yaml
+graders:
+ tool_check:
+ kind: tool
+ function: contains
+ extractor: tool_arguments
+ extractor_config:
+ tool_name: search
+```
+
+### Testing Memory Persistence
+
+Check if the agent correctly updates its memory blocks:
+
+```yaml
+graders:
+ memory_check:
+ kind: tool
+ function: contains
+ extractor: memory_block
+ extractor_config:
+ block_label: human
+```
+
+## Troubleshooting
+
+<Warning>
+**"Agent file not found"**
+
+Make sure your `agent_file` path is correct. Paths are relative to the suite YAML file location. Use absolute paths if needed:
+
+```yaml
+target:
+ agent_file: /absolute/path/to/my_agent.af
+```
+</Warning>
+
+<Warning>
+**"Connection refused"**
+
+Your Letta server isn't running or isn't accessible. Start it with:
+
+```bash
+letta server
+```
+
+By default, it runs at `http://localhost:8283`.
+</Warning>
+
+<Warning>
+**"No ground_truth provided"**
+
+Tool graders like `exact_match` and `contains` require `ground_truth` in your dataset. Either:
+- Add `ground_truth` to your samples, or
+- Use a rubric grader which doesn't require ground truth
+</Warning>
+
+<Tip>
+**Agent didn't respond as expected**
+
+Try testing your agent manually first using the Letta SDK or Agent Development Environment (ADE) to see how it behaves before running evaluations. See the [Letta documentation](https://docs.letta.com) for more information.
+</Tip>
+
+For more help, see the [Troubleshooting Guide](/evals/troubleshooting/common-issues).
diff --git a/fern/pages/evals/graders/multi-metric.mdx b/fern/pages/evals/graders/multi-metric.mdx
new file mode 100644
index 000000000..d414f7ae9
--- /dev/null
+++ b/fern/pages/evals/graders/multi-metric.mdx
@@ -0,0 +1,58 @@
+# Multi-Metric Evaluation
+
+Evaluate multiple aspects of agent performance simultaneously in a single evaluation suite.
+
+<Note>
+Multi-metric evaluation allows you to define multiple graders, each measuring a different dimension of your agent's behavior.
+</Note>
+
+## Why Multiple Metrics?
+
+Agents are complex systems. You might want to evaluate:
+- **Correctness**: Does the answer match the expected output?
+- **Quality**: Is the explanation clear and complete?
+- **Tool usage**: Does the agent call the right tools with correct arguments?
+- **Memory**: Does the agent correctly update its memory blocks?
+- **Format**: Does the output follow required formatting rules?
+
+## Configuration
+
+```yaml
+graders:
+ accuracy: # Check if answer is correct
+ kind: tool
+ function: exact_match
+ extractor: last_assistant
+
+ completeness: # LLM judges response quality
+ kind: rubric
+ prompt_path: rubrics/completeness.txt
+ model: gpt-4o-mini
+ extractor: last_assistant
+
+ tool_usage: # Verify correct tool was called
+ kind: tool
+ function: contains
+ extractor: tool_arguments
+ extractor_config:
+ tool_name: search
+```
+
+## Gating on One Metric
+
+The gate can check any of these metrics:
+
+```yaml
+gate:
+ metric_key: accuracy # Gate on accuracy (others still computed)
+ op: gte
+ value: 0.9
+```
+
+Results will include scores for all graders, even if you only gate on one.
+
+## Next Steps
+
+- [Tool Graders](/evals/graders/tool-graders) - Deterministic evaluation
+- [Rubric Graders](/evals/graders/rubric-graders) - LLM-as-judge evaluation
+- [Gates](/evals/core-concepts/gates) - Setting pass/fail criteria
diff --git a/fern/pages/evals/graders/rubric-graders.mdx b/fern/pages/evals/graders/rubric-graders.mdx
new file mode 100644
index 000000000..1b73ba7a0
--- /dev/null
+++ b/fern/pages/evals/graders/rubric-graders.mdx
@@ -0,0 +1,82 @@
+# Rubric Graders
+
+Rubric graders use language models to evaluate submissions based on custom criteria. They're ideal for subjective, nuanced evaluation.
+
+<Note>
+Rubric graders work by providing the LLM with a prompt that describes the evaluation criteria, then the language model generates a structured JSON response with a score and rationale.
+</Note>
+
+## Basic Configuration
+
+```yaml
+graders:
+ quality:
+ kind: rubric
+ prompt_path: quality_rubric.txt # Evaluation criteria
+ model: gpt-4o-mini # Judge model
+ temperature: 0.0 # Deterministic
+ extractor: last_assistant # What to evaluate
+```
+
+## Rubric Prompt Format
+
+Your rubric file should describe the evaluation criteria. Use placeholders:
+
+- `{input}`: The original input from the dataset
+- `{submission}`: The extracted agent response
+- `{ground_truth}`: Ground truth from dataset (if available)
+
+Example `quality_rubric.txt`:
+```
+Evaluate the response for:
+1. Accuracy: Does it correctly answer the question?
+2. Completeness: Is the answer thorough?
+3. Clarity: Is it well-explained?
+
+Input: {input}
+Expected: {ground_truth}
+Response: {submission}
+
+Score from 0.0 to 1.0 where:
+
+- 1.0: Perfect response
+- 0.75: Good with minor issues
+- 0.5: Acceptable but incomplete
+- 0.25: Poor quality
+- 0.0: Completely wrong
+```
+
+## Model Configuration
+
+```yaml
+graders:
+ quality:
+ kind: rubric
+ prompt_path: rubric.txt
+ model: gpt-4o-mini # Judge model
+ temperature: 0.0 # Deterministic
+ provider: openai # LLM provider
+ max_retries: 5 # API retry attempts
+ timeout: 120.0 # Request timeout
+```
+
+## Agent-as-Judge
+
+Use a Letta agent as the judge instead of a direct LLM API call:
+
+```yaml
+graders:
+ agent_judge:
+ kind: rubric
+ agent_file: judge.af # Judge agent with submit_grade tool
+ prompt_path: rubric.txt # Evaluation criteria
+ extractor: last_assistant
+```
+
+**Requirements**: The judge agent must have a tool with signature `submit_grade(score: float, rationale: str)`.
+
+## Next Steps
+
+- [Tool Graders](/evals/graders/tool-graders) - Deterministic grading functions
+- [Multi-Metric](/evals/graders/multi-metric-grading) - Combine multiple graders
+- [Custom Graders](/evals/advanced/custom-graders) - Write your own grading logic
diff --git a/fern/pages/evals/graders/tool-graders.mdx b/fern/pages/evals/graders/tool-graders.mdx
new file mode 100644
index 000000000..0edb327f5
--- /dev/null
+++ b/fern/pages/evals/graders/tool-graders.mdx
@@ -0,0 +1,85 @@
+# Tool Graders
+
+Tool graders use Python functions to programmatically evaluate submissions. They're ideal for deterministic, rule-based evaluation.
+
+## Overview
+
+Tool graders:
+- Execute Python functions that take `(sample, submission)` and return a `GradeResult`
+- Are fast and deterministic
+- Don't require external API calls
+- Can implement any custom logic
+
+## Configuration
+
+```yaml
+graders:
+ my_metric:
+ kind: tool
+ function: exact_match # Function name
+ extractor: last_assistant # What to extract from trajectory
+```
+
+## Built-in Functions
+
+### exact_match
+
+Checks if submission exactly matches ground truth (case-sensitive, whitespace-trimmed).
+
+```yaml
+graders:
+ accuracy:
+ kind: tool
+ function: exact_match
+ extractor: last_assistant
+```
+
+**Requires**: `ground_truth` in dataset | **Score**: 1.0 if exact match, 0.0 otherwise
+
+### contains
+
+Checks if submission contains ground truth (case-insensitive).
+
+```yaml
+graders:
+ contains_answer:
+ kind: tool
+ function: contains
+ extractor: last_assistant
+```
+
+**Requires**: `ground_truth` in dataset | **Score**: 1.0 if found, 0.0 otherwise
+
+### regex_match
+
+Checks if submission matches a regex pattern in ground truth.
+
+```yaml
+graders:
+ pattern:
+ kind: tool
+ function: regex_match
+ extractor: last_assistant
+```
+
+**Score**: 1.0 if pattern matches, 0.0 otherwise
+
+### ascii_printable_only
+
+Validates that all characters are printable ASCII.
+
+```yaml
+graders:
+ ascii_check:
+ kind: tool
+ function: ascii_printable_only
+ extractor: last_assistant
+```
+
+**Score**: 1.0 if all characters are printable ASCII, 0.0 otherwise
+
+## Next Steps
+
+- [Rubric Graders](/evals/graders/rubric-graders) - LLM-as-judge evaluation
+- [Custom Graders](/evals/advanced/custom-graders) - Write your own grading functions
+- [Multi-Metric](/evals/graders/multi-metric-grading) - Combine multiple graders
diff --git a/fern/pages/evals/overview.mdx b/fern/pages/evals/overview.mdx
new file mode 100644
index 000000000..7856bc458
--- /dev/null
+++ b/fern/pages/evals/overview.mdx
@@ -0,0 +1,47 @@
+# Letta Evals
+
+**Systematic testing for stateful AI agents.** Validate changes, prevent regressions, and ship with confidence.
+
+Test agent memory, tool usage, multi-turn conversations, and state evolution with automated grading and pass/fail gates.
+
+<Note>
+**Ready to start?** Jump to [Getting Started](/evals/get-started/getting-started) or learn the [Core Concepts](/evals/core-concepts/concepts-overview) first.
+</Note>
+
+## Core Concepts
+
+Understand the building blocks of evaluations:
+
+- [Suites](/evals/core-concepts/suites) - Configure your evaluation
+- [Datasets](/evals/core-concepts/datasets) - Define test cases
+- [Targets](/evals/core-concepts/targets) - Specify the agent to test
+- [Graders](/evals/core-concepts/graders) - Score agent outputs
+- [Extractors](/evals/core-concepts/extractors) - Extract content from responses
+- [Gates](/evals/core-concepts/gates) - Set pass/fail criteria
+
+### Grading & Extraction
+
+Choose how to score your agents:
+
+- [Tool Graders](/evals/graders/tool-graders) - Fast, deterministic grading with Python functions
+- [Rubric Graders](/evals/graders/rubric-graders) - Flexible LLM-as-judge evaluation
+- [Built-in Extractors](/evals/extractors/built-in-extractors) - Pre-built content extractors
+- [Multi-Metric Grading](/evals/graders/multi-metric-grading) - Evaluate multiple dimensions
+
+### Advanced
+
+- [Custom Graders](/evals/advanced/custom-graders) - Write your own grading logic
+- [Custom Extractors](/evals/extractors/custom-extractors) - Build custom extractors
+- [Multi-Turn Conversations](/evals/advanced/multi-turn-conversations) - Test memory and state
+- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete configuration schema
+
+### Reference
+
+- [CLI Commands](/evals/cli-reference/commands) - Command-line interface
+- [Understanding Results](/evals/results-metrics/understanding-results) - Interpret metrics
+- [Troubleshooting](/evals/troubleshooting/common-issues) - Common issues and solutions
+
+## Resources
+
+- **[GitHub Repository](https://github.com/letta-ai/letta-evals)** - Source code, issues, and contributions
+- **[PyPI Package](https://pypi.org/project/letta-evals/)** - Install with `pip install letta-evals`
diff --git a/fern/pages/evals/results/overview.mdx b/fern/pages/evals/results/overview.mdx
new file mode 100644
index 000000000..2487c034d
--- /dev/null
+++ b/fern/pages/evals/results/overview.mdx
@@ -0,0 +1,484 @@
+# Understanding Results
+
+This guide explains how to interpret evaluation results.
+
+## Result Structure
+
+An evaluation produces three types of output:
+
+1. **Console output**: Real-time progress and summary
+2. **Summary JSON**: Aggregate metrics and configuration
+3. **Results JSONL**: Per-sample detailed results
+
+## Console Output
+
+### Progress Display
+
+```
+Running evaluation: my-eval-suite
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3/3 100%
+
+Results:
+ Total samples: 3
+ Attempted: 3
+ Avg score: 0.83 (attempted: 0.83)
+ Passed: 2 (66.7%)
+
+Gate (quality >= 0.75): PASSED
+```
+
+### Quiet Mode
+
+```bash
+letta-evals run suite.yaml --quiet
+```
+
+Output:
+```
+✓ PASSED
+```
+
+or
+
+```
+✗ FAILED
+```
+
+## JSON Output
+
+### Saving Results
+
+```bash
+letta-evals run suite.yaml --output results/
+```
+
+Creates three files:
+
+#### header.json
+
+Evaluation metadata:
+
+```json
+{
+ "suite_name": "my-eval-suite",
+ "timestamp": "2025-01-15T10:30:00Z",
+ "version": "0.3.0"
+}
+```
+
+#### summary.json
+
+Complete evaluation summary:
+
+```json
+{
+ "suite": "my-eval-suite",
+ "config": {
+ "target": {...},
+ "graders": {...},
+ "gate": {...}
+ },
+ "metrics": {
+ "total": 10,
+ "total_attempted": 10,
+ "avg_score_attempted": 0.85,
+ "avg_score_total": 0.85,
+ "passed_attempts": 8,
+ "failed_attempts": 2,
+ "by_metric": {
+ "accuracy": {
+ "avg_score_attempted": 0.90,
+ "pass_rate": 90.0,
+ "passed_attempts": 9,
+ "failed_attempts": 1
+ },
+ "quality": {
+ "avg_score_attempted": 0.80,
+ "pass_rate": 70.0,
+ "passed_attempts": 7,
+ "failed_attempts": 3
+ }
+ }
+ },
+ "gates_passed": true
+}
+```
+
+#### results.jsonl
+
+One JSON object per line, each representing one sample:
+
+```jsonl
+{"sample": {"id": 0, "input": "What is 2+2?", "ground_truth": "4"}, "submission": "4", "grade": {"score": 1.0, "rationale": "Exact match: true"}, "trajectory": [...], "agent_id": "agent-123", "model_name": "default"}
+{"sample": {"id": 1, "input": "What is 3+3?", "ground_truth": "6"}, "submission": "6", "grade": {"score": 1.0, "rationale": "Exact match: true"}, "trajectory": [...], "agent_id": "agent-124", "model_name": "default"}
+```
+
+## Metrics Explained
+
+### total
+
+Total number of samples in the evaluation (including errors).
+
+### total_attempted
+
+Number of samples that completed without errors.
+
+If a sample fails during agent execution or grading, it's counted in `total` but not `total_attempted`.
+
+### avg_score_attempted
+
+Average score across samples that completed successfully.
+
+Formula: `sum(scores) / total_attempted`
+
+Range: 0.0 to 1.0
+
+### avg_score_total
+
+Average score across all samples, treating errors as 0.0.
+
+Formula: `sum(scores) / total`
+
+Range: 0.0 to 1.0
+
+### passed_attempts / failed_attempts
+
+Number of samples that passed/failed the gate's per-sample criteria.
+
+By default:
+- If gate metric is `accuracy`: sample passes if score `>= 1.0`
+- If gate metric is `avg_score`: sample passes if score `>=` gate value
+
+Can be customized with `pass_op` and `pass_value` in gate config.
+
+### by_metric
+
+For multi-metric evaluation, shows aggregate stats for each metric:
+
+```json
+"by_metric": {
+ "accuracy": {
+ "avg_score_attempted": 0.90,
+ "avg_score_total": 0.85,
+ "pass_rate": 90.0,
+ "passed_attempts": 9,
+ "failed_attempts": 1
+ }
+}
+```
+
+## Sample Results
+
+Each sample result includes:
+
+### sample
+The original dataset sample:
+```json
+"sample": {
+ "id": 0,
+ "input": "What is 2+2?",
+ "ground_truth": "4",
+ "metadata": {...}
+}
+```
+
+### submission
+The extracted text that was graded:
+```json
+"submission": "The answer is 4"
+```
+
+### grade
+The grading result:
+```json
+"grade": {
+ "score": 1.0,
+ "rationale": "Contains ground_truth: true",
+ "metadata": {"model": "gpt-4o-mini", "usage": {...}}
+}
+```
+
+### grades (multi-metric)
+For multi-metric evaluation:
+```json
+"grades": {
+ "accuracy": {"score": 1.0, "rationale": "Exact match"},
+ "quality": {"score": 0.85, "rationale": "Good but verbose"}
+}
+```
+
+### trajectory
+The complete conversation history:
+```json
+"trajectory": [
+ [
+ {"role": "user", "content": "What is 2+2?"},
+ {"role": "assistant", "content": "The answer is 4"}
+ ]
+]
+```
+
+### agent_id
+The ID of the agent that generated this response:
+```json
+"agent_id": "agent-abc-123"
+```
+
+### model_name
+The model configuration used:
+```json
+"model_name": "gpt-4o-mini"
+```
+
+### agent_usage
+Token usage statistics (if available):
+```json
+"agent_usage": [
+ {"completion_tokens": 10, "prompt_tokens": 50, "total_tokens": 60}
+]
+```
+
+## Interpreting Scores
+
+### Score Ranges
+
+- **1.0**: Perfect - fully meets criteria
+- **0.8-0.99**: Very good - minor issues
+- **0.6-0.79**: Good - notable improvements possible
+- **0.4-0.59**: Acceptable - significant issues
+- **0.2-0.39**: Poor - major problems
+- **0.0-0.19**: Failed - did not meet criteria
+
+### Binary vs Continuous
+
+**Tool graders** typically return binary scores:
+- 1.0: Passed
+- 0.0: Failed
+
+**Rubric graders** return continuous scores:
+- Any value from 0.0 to 1.0
+- Allows for partial credit
+
+## Multi-Model Results
+
+When testing multiple models:
+
+```json
+"metrics": {
+ "per_model": [
+ {
+ "model_name": "gpt-4o-mini",
+ "avg_score_attempted": 0.85,
+ "passed_samples": 8,
+ "failed_samples": 2
+ },
+ {
+ "model_name": "claude-3-5-sonnet",
+ "avg_score_attempted": 0.90,
+ "passed_samples": 9,
+ "failed_samples": 1
+ }
+ ]
+}
+```
+
+Console output:
+```
+Results by model:
+ gpt-4o-mini - Avg: 0.85, Pass: 80.0%
+ claude-3-5-sonnet - Avg: 0.90, Pass: 90.0%
+```
+
+## Multiple Runs Statistics
+
+Run evaluations multiple times to measure consistency and get aggregate statistics.
+
+### Configuration
+
+Specify in YAML:
+```yaml
+name: my-eval-suite
+dataset: dataset.jsonl
+num_runs: 5 # Run 5 times
+target:
+ kind: agent
+ agent_file: my_agent.af
+graders:
+ accuracy:
+ kind: tool
+ function: exact_match
+gate:
+ metric_key: accuracy
+ op: gte
+ value: 0.8
+```
+
+Or via CLI:
+```bash
+letta-evals run suite.yaml --num-runs 10 --output results/
+```
+
+### Output Structure
+
+```
+results/
+├── run_1/
+│ ├── header.json
+│ ├── results.jsonl
+│ └── summary.json
+├── run_2/
+│ ├── header.json
+│ ├── results.jsonl
+│ └── summary.json
+├── ...
+└── aggregate_stats.json # Statistics across all runs
+```
+
+### Aggregate Statistics File
+
+The `aggregate_stats.json` includes statistics across all runs:
+
+```json
+{
+ "num_runs": 10,
+ "runs_passed": 8,
+ "mean_avg_score_attempted": 0.847,
+ "std_avg_score_attempted": 0.042,
+ "mean_avg_score_total": 0.847,
+ "std_avg_score_total": 0.042,
+ "mean_scores": {
+ "accuracy": 0.89,
+ "quality": 0.82
+ },
+ "std_scores": {
+ "accuracy": 0.035,
+ "quality": 0.051
+ },
+ "individual_run_metrics": [
+ {
+ "avg_score_attempted": 0.85,
+ "avg_score_total": 0.85,
+ "pass_rate": 0.85,
+ "by_metric": {
+ "accuracy": {
+ "avg_score_attempted": 0.90,
+ "avg_score_total": 0.90,
+ "pass_rate": 0.90
+ }
+ }
+ }
+ // ... metrics from runs 2-10
+ ]
+}
+```
+
+**Key fields**:
+- `num_runs`: Total number of runs executed
+- `runs_passed`: Number of runs that passed the gate
+- `mean_avg_score_attempted`: Mean score across runs (only attempted samples)
+- `std_avg_score_attempted`: Standard deviation (measures consistency)
+- `mean_scores`: Mean for each metric (e.g., `{"accuracy": 0.89}`)
+- `std_scores`: Standard deviation for each metric (e.g., `{"accuracy": 0.035}`)
+- `individual_run_metrics`: Full metrics object from each individual run
+
+### Use Cases
+
+**Measure consistency of non-deterministic agents:**
+```bash
+letta-evals run suite.yaml --num-runs 20 --output results/
+# Check std_avg_score_attempted in aggregate_stats.json
+# Low std = consistent, high std = variable
+```
+
+**Get confidence intervals:**
+```python
+import json
+import math
+
+with open("results/aggregate_stats.json") as f:
+ stats = json.load(f)
+
+mean = stats["mean_avg_score_attempted"]
+std = stats["std_avg_score_attempted"]
+n = stats["num_runs"]
+
+# 95% confidence interval (assuming normal distribution)
+margin = 1.96 * (std / math.sqrt(n))
+print(f"Score: {mean:.3f} ± {margin:.3f}")
+```
+
+**Compare metric consistency:**
+```python
+with open("results/aggregate_stats.json") as f:
+ stats = json.load(f)
+
+for metric_name, mean in stats["mean_scores"].items():
+ std = stats["std_scores"][metric_name]
+ consistency = "consistent" if std < 0.05 else "variable"
+ print(f"{metric_name}: {mean:.3f} ± {std:.3f} ({consistency})")
+```
+
+## Error Handling
+
+If a sample encounters an error:
+
+```json
+{
+ "sample": {...},
+ "submission": "",
+ "grade": {
+ "score": 0.0,
+ "rationale": "Error during grading: Connection timeout",
+ "metadata": {"error": "timeout", "error_type": "ConnectionError"}
+ }
+}
+```
+
+Errors:
+- Count toward `total` but not `total_attempted`
+- Get score of 0.0
+- Include error details in rationale and metadata
+
+## Analyzing Results
+
+### Find Low Scores
+
+```python
+import json
+
+with open("results/results.jsonl") as f:
+ results = [json.loads(line) for line in f]
+
+low_scores = [r for r in results if r["grade"]["score"] < 0.5]
+print(f"Found {len(low_scores)} samples with score < 0.5")
+
+for result in low_scores:
+ print(f"Sample {result['sample']['id']}: {result['grade']['rationale']}")
+```
+
+### Compare Metrics
+
+```python
+# Load summary
+with open("results/summary.json") as f:
+ summary = json.load(f)
+
+metrics = summary["metrics"]["by_metric"]
+for name, stats in metrics.items():
+ print(f"{name}: {stats['avg_score_attempted']:.2f} avg, {stats['pass_rate']:.1f}% pass")
+```
+
+### Extract Failures
+
+```python
+# Find samples that failed gate criteria
+failures = [
+ r for r in results
+ if not gate_passed(r["grade"]["score"]) # Your gate logic
+]
+```
+
+## Next Steps
+
+- [Gates](/evals/core-concepts/gates) - Setting pass/fail criteria
+- [CLI Commands](/evals/cli-reference/commands) - Running evaluations
diff --git a/fern/pages/evals/troubleshooting.mdx b/fern/pages/evals/troubleshooting.mdx
new file mode 100644
index 000000000..dc259c6ec
--- /dev/null
+++ b/fern/pages/evals/troubleshooting.mdx
@@ -0,0 +1,267 @@
+# Troubleshooting
+
+Common issues and solutions when using Letta Evals.
+
+## Installation Issues
+
+<Warning>
+**"Command not found: letta-evals"**
+
+**Problem**: CLI not available after installation
+
+**Solution**:
+```bash
+# Verify installation
+pip list | grep letta-evals
+
+# Reinstall if needed
+pip install --upgrade letta-evals
+```
+</Warning>
+
+<Warning>
+**Import errors**
+
+**Problem**: `ModuleNotFoundError: No module named 'letta_evals'`
+
+**Solution**:
+```bash
+# Ensure you're in the right environment
+which python
+
+# Install in correct environment
+source .venv/bin/activate
+pip install letta-evals
+```
+</Warning>
+
+## Configuration Issues
+
+<Warning>
+**"Agent file not found"**
+
+**Problem**: `FileNotFoundError: agent.af`
+
+**Solution**:
+- Check the path is correct relative to the suite YAML
+- Use absolute paths if needed
+- Verify file exists: `ls -la path/to/agent.af`
+
+```yaml
+# Correct relative path
+target:
+ agent_file: ./agents/my_agent.af
+```
+</Warning>
+
+<Warning>
+**"Dataset not found"**
+
+**Problem**: Cannot load dataset file
+
+**Solution**:
+- Verify dataset path in YAML
+- Check file exists: `ls -la dataset.jsonl`
+- Ensure proper JSONL format (one JSON object per line)
+
+```bash
+# Validate JSONL format
+cat dataset.jsonl | jq .
+```
+</Warning>
+
+<Warning>
+**"Validation failed: unknown function"**
+
+**Problem**: Grader function not found
+
+**Solution**:
+```bash
+# List available graders
+letta-evals list-graders
+
+# Check spelling in suite.yaml
+graders:
+ my_metric:
+ function: exact_match # Correct
+```
+</Warning>
+
+## Connection Issues
+
+<Warning>
+**"Connection refused"**
+
+**Problem**: Cannot connect to Letta server
+
+**Solution**:
+```bash
+# Verify server is running
+curl http://localhost:8283/v1/health
+
+# Check base_url in suite.yaml
+target:
+ base_url: http://localhost:8283
+```
+</Warning>
+
+<Warning>
+**"Unauthorized" or "Invalid API key"**
+
+**Problem**: Authentication failed
+
+**Solution**:
+```bash
+# Set API key
+export LETTA_API_KEY=your-key-here
+
+# Verify key is correct
+echo $LETTA_API_KEY
+```
+</Warning>
+
+## Runtime Issues
+
+<Warning>
+**"No ground_truth provided"**
+
+**Problem**: Grader requires ground truth but sample doesn't have it
+
+**Solution**:
+- Add ground_truth to dataset samples:
+```jsonl
+{"input": "What is 2+2?", "ground_truth": "4"}
+```
+
+- Or use a grader that doesn't require ground truth:
+```yaml
+graders:
+ quality:
+ kind: rubric # Doesn't require ground_truth
+ prompt_path: rubric.txt
+```
+</Warning>
+
+## Performance Issues
+
+<Tip>
+**Evaluation is very slow**
+
+**Solutions**:
+
+1. Increase concurrency:
+```bash
+letta-evals run suite.yaml --max-concurrent 20
+```
+
+2. Reduce samples for testing:
+```yaml
+max_samples: 10 # Test with small subset first
+```
+
+3. Use tool graders instead of rubric graders:
+```yaml
+graders:
+ accuracy:
+ kind: tool # Much faster than rubric
+ function: exact_match
+```
+</Tip>
+
+<Tip>
+**High API costs**
+
+**Solutions**:
+
+1. Use cheaper models:
+```yaml
+graders:
+ quality:
+ model: gpt-4o-mini # Cheaper than gpt-4o
+```
+
+2. Test with small sample first:
+```yaml
+max_samples: 5 # Verify before running full suite
+```
+</Tip>
+
+## Results Issues
+
+<Warning>
+**"All scores are 0.0"**
+
+**Solutions**:
+
+1. Verify extractor is getting content
+2. Check grader logic
+3. Test agent manually first
+</Warning>
+
+<Warning>
+**"Gates failed but scores look good"**
+
+**Solution**:
+- Check gate configuration:
+```yaml
+gate:
+ metric_key: accuracy # Correct metric?
+ metric: avg_score # Or accuracy?
+ op: gte # Correct operator?
+ value: 0.8 # Correct threshold?
+```
+</Warning>
+
+## Debug Tips
+
+### Enable verbose output
+
+Run without `--quiet` to see detailed progress:
+```bash
+letta-evals run suite.yaml
+```
+
+### Examine output files
+
+```bash
+letta-evals run suite.yaml --output debug/
+
+# Check summary
+cat debug/summary.json | jq .
+
+# Check individual results
+cat debug/results.jsonl | jq .
+```
+
+### Validate configuration
+
+```bash
+letta-evals validate suite.yaml
+```
+
+### Check component availability
+
+```bash
+letta-evals list-graders
+letta-evals list-extractors
+```
+
+## Getting Help
+
+If you're still stuck:
+
+1. Check the [Getting Started guide](/evals/get-started/getting-started)
+2. Review the [Core Concepts](/evals/core-concepts/concepts-overview)
+3. Report issues at the [Letta Evals GitHub repository](https://github.com/letta-ai/letta-evals)
+
+When reporting issues, include:
+- Suite YAML configuration
+- Dataset sample (if not sensitive)
+- Error message and full stack trace
+- Environment info (OS, Python version)
+
+```bash
+# Get environment info
+python --version
+pip show letta-evals
+```