Platform for stateful agents: AI with advanced memory that can learn and self-improve over time.
feat: integrate Letta Evals documentation (#5665)
Cameron Pfiffer237a50255235
diff --git a/fern/docs.yml b/fern/docs.ymlindex 37e97adda..bcaf277bd 100644--- a/fern/docs.yml+++ b/fern/docs.yml@@ -50,6 +50,9 @@ tabs:leaderboard:display-name: Leaderboardskip-slug: true+ evals:+ display-name: Evals+ slug: evalslanding-page:page: homepath: pages/index.mdx@@ -723,6 +726,76 @@ navigation:- page: Contributing Resultspath: pages/leaderboard/contributing.mdx+ - tab: evals+ layout:+ - link: Chat on Discord+ icon: fa-brands fa-discord+ href: https://discord.gg/letta+ - link: Developer Forum+ icon: fa-sharp fa-light fa-comments+ href: https://forum.letta.com+ - link: DeepLearning.AI Course+ icon: fa-sharp fa-light fa-building-columns+ href: https://www.deeplearning.ai/short-courses/llms-as-operating-systems-agent-memory/?utm_campaign=memgpt-launch&utm_content=331638345&utm_medium=social&utm_source=docs&hss_channel=tw-992153930095251456+ - section: Get Started+ contents:+ - page: Overview+ path: pages/evals/overview.mdx+ - page: Getting Started+ path: pages/evals/getting-started.mdx+ - section: Core Concepts+ contents:+ - page: Concepts Overview+ path: pages/evals/concepts/overview.mdx+ - page: Suites+ path: pages/evals/concepts/suites.mdx+ - page: Datasets+ path: pages/evals/concepts/datasets.mdx+ - page: Targets+ path: pages/evals/concepts/targets.mdx+ - page: Graders+ path: pages/evals/concepts/graders.mdx+ - page: Extractors+ path: pages/evals/concepts/extractors.mdx+ - page: Gates+ path: pages/evals/concepts/gates.mdx+ - section: Configuration+ contents:+ - page: Suite YAML Reference+ path: pages/evals/configuration/suite-yaml.mdx+ - section: Graders+ contents:+ - page: Tool Graders+ path: pages/evals/graders/tool-graders.mdx+ - page: Rubric Graders+ path: pages/evals/graders/rubric-graders.mdx+ - page: Multi-Metric Grading+ path: pages/evals/graders/multi-metric.mdx+ - section: Extractors+ contents:+ - page: Built-in Extractors+ path: pages/evals/extractors/builtin.mdx+ - page: Custom Extractors+ path: pages/evals/extractors/custom.mdx+ - section: Advanced+ contents:+ - page: Custom Graders+ path: pages/evals/advanced/custom-graders.mdx+ - page: Multi-Turn Conversations+ path: pages/evals/advanced/multi-turn-conversations.mdx+ - section: CLI Reference+ contents:+ - page: Commands+ path: pages/evals/cli/commands.mdx+ - section: Results & Metrics+ contents:+ - page: Understanding Results+ path: pages/evals/results/overview.mdx+ - section: Troubleshooting+ contents:+ - page: Common Issues+ path: pages/evals/troubleshooting.mdx+# - tab: cookbooks# layout:# - section: Cookbooksdiff --git a/fern/pages/evals/advanced/custom-graders.mdx b/fern/pages/evals/advanced/custom-graders.mdxnew file mode 100644index 000000000..bce7b8938--- /dev/null+++ b/fern/pages/evals/advanced/custom-graders.mdx@@ -0,0 +1,66 @@+# Custom Graders++Write your own grading functions to implement custom evaluation logic.++<Note>+Custom graders let you implement domain-specific evaluation, parse complex formats, and apply custom scoring algorithms.+</Note>++## Basic Structure++```python+from letta_evals.decorators import grader+from letta_evals.models import GradeResult, Sample++@grader+def my_custom_grader(sample: Sample, submission: str) -> GradeResult:+ """Custom grading logic."""++ # Your evaluation logic+ score = calculate_score(submission, sample.ground_truth)++ # Ensure score is between 0.0 and 1.0+ score = max(0.0, min(1.0, score))++ return GradeResult(+ score=score,+ rationale=f"Score based on custom logic: {score}"+ )+```++## Example: JSON Validation++```python+import json+from letta_evals.decorators import grader+from letta_evals.models import GradeResult, Sample++@grader+def valid_json(sample: Sample, submission: str) -> GradeResult:+ """Check if submission is valid JSON."""+ try:+ json.loads(submission)+ return GradeResult(score=1.0, rationale="Valid JSON")+ except json.JSONDecodeError as e:+ return GradeResult(score=0.0, rationale=f"Invalid JSON: {e}")+```++## Registration++Custom graders are automatically registered when you import them in your suite's setup script or custom evaluators file.++## Configuration++```yaml+graders:+ my_metric:+ kind: tool+ function: my_custom_grader # Your function name+ extractor: last_assistant+```++## Next Steps++- [Tool Graders](/evals/graders/tool-graders) - Built-in grading functions+- [Graders Concept](/evals/core-concepts/graders) - Understanding graders+- [Example Custom Graders](https://github.com/letta-ai/letta-evals/tree/main/examples) - See examples in the letta-evals repodiff --git a/fern/pages/evals/advanced/multi-turn-conversations.mdx b/fern/pages/evals/advanced/multi-turn-conversations.mdxnew file mode 100644index 000000000..269761ef1--- /dev/null+++ b/fern/pages/evals/advanced/multi-turn-conversations.mdx@@ -0,0 +1,94 @@+# Multi-Turn Conversations++Multi-turn conversations allow you to test how agents handle context across multiple exchanges.++<Note>+This is essential for stateful agents where behavior depends on conversation history.+</Note>++## Why Use Multi-Turn?++Multi-turn conversations enable testing that single-turn prompts cannot:++- **Memory storage**: Verify agents persist information to memory blocks+- **Tool call sequences**: Test multi-step workflows+- **Context retention**: Ensure agents remember details from earlier+- **State evolution**: Track how agent state changes across interactions+- **Conversational coherence**: Test if agents maintain context appropriately++## Format++### Single-Turn (Default)++```jsonl+{"input": "What is the capital of France?", "ground_truth": "Paris"}+```++### Multi-Turn++```jsonl+{"input": ["My name is Alice", "What's my name?"], "ground_truth": "Alice"}+```++The agent processes each input in sequence, with state carrying over between turns.++## Example 1: Memory Recall Testing++Test if the agent remembers information across turns:++```jsonl+{"input": ["Remember that my favorite color is blue", "What's my favorite color?"], "ground_truth": "blue"}+```++Suite configuration:+```yaml+graders:+ response_check:+ kind: tool+ function: contains+ extractor: last_assistant # Check the agent's response+```++## Example 2: Memory Correction Testing++Test if the agent correctly updates memory when users correct themselves:++```jsonl+{"input": ["Please remember that I like bananas.", "Actually, sorry, I meant I like apples."], "ground_truth": "apples"}+```++Suite configuration:+```yaml+graders:+ memory_check:+ kind: tool+ function: contains+ extractor: memory_block+ extractor_config:+ block_label: human # Check the actual memory block, not just the response+```++<Note>+**Key difference:** The `memory_block` extractor verifies the agent actually stored the corrected information in memory, not just that it responded correctly. This tests real memory persistence.+</Note>++## When to Test Memory Blocks vs. Responses++**Use `last_assistant` or `all_assistant` extractors when:**+- Testing what the agent says in conversation+- Verifying response content and phrasing+- Checking conversational coherence++**Use `memory_block` extractor when:**+- Verifying information was actually stored in memory+- Testing memory updates and corrections+- Validating persistent state changes+- Ensuring the agent's internal state is correct++See the [multiturn-memory-block-extractor example](https://github.com/letta-ai/letta-evals/tree/main/examples/multiturn-memory-block-extractor) for a complete working implementation.++## Next Steps++- [Datasets](/evals/core-concepts/datasets) - Creating test datasets+- [Extractors](/evals/core-concepts/extractors) - Extracting from trajectories+- [Targets](/evals/core-concepts/targets) - Agent lifecycle and testing behaviordiff --git a/fern/pages/evals/cli/commands.mdx b/fern/pages/evals/cli/commands.mdxnew file mode 100644index 000000000..0b8eee31c--- /dev/null+++ b/fern/pages/evals/cli/commands.mdx@@ -0,0 +1,342 @@+# CLI Commands++The **letta-evals** command-line interface lets you run evaluations, validate configurations, and inspect available components.++<Note>+**Quick overview:**+- **`run`** - Execute an evaluation suite (most common)+- **`validate`** - Check suite configuration without running+- **`list-extractors`** - Show available extractors+- **`list-graders`** - Show available grader functions+- **Exit codes** - 0 for pass, 1 for fail (perfect for CI/CD)+</Note>++**Typical workflow:**+1. Validate your suite: `letta-evals validate suite.yaml`+2. Run evaluation: `letta-evals run suite.yaml --output results/`+3. Check exit code: `echo $?` (0 = passed, 1 = failed)++## run++Run an evaluation suite.++```bash+letta-evals run <suite.yaml> [options]+```++### Arguments++- `suite.yaml`: Path to the suite configuration file (required)++### Options++#### --output, -o+Save results to a directory.++```bash+letta-evals run suite.yaml --output results/+```++Creates:+- `results/header.json`: Evaluation metadata+- `results/summary.json`: Aggregate metrics and configuration+- `results/results.jsonl`: Per-sample results (one JSON per line)++#### --quiet, -q+Quiet mode - only show pass/fail result.++```bash+letta-evals run suite.yaml --quiet+```++Output:+```+✓ PASSED+```++#### --max-concurrent+Maximum concurrent sample evaluations. **Default**: 15++```bash+letta-evals run suite.yaml --max-concurrent 10+```++Higher values = faster evaluation but more resource usage.++#### --api-key+Letta API key (overrides LETTA_API_KEY environment variable).++```bash+letta-evals run suite.yaml --api-key your-key+```++#### --base-url+Letta server base URL (overrides suite config and environment variable).++```bash+letta-evals run suite.yaml --base-url http://localhost:8283+```++#### --project-id+Letta project ID for cloud deployments.++```bash+letta-evals run suite.yaml --project-id proj_abc123+```++#### --cached, -c+Path to cached results (JSONL) for re-grading trajectories without re-running the agent.++```bash+letta-evals run suite.yaml --cached previous_results.jsonl+```++Use this to test different graders on the same agent trajectories.++#### --num-runs+Run the evaluation multiple times to measure consistency. **Default**: 1++```bash+letta-evals run suite.yaml --num-runs 10+```++**Output with multiple runs:**+- Each run creates a separate `run_N/` directory with individual results+- An `aggregate_stats.json` file contains statistics across all runs (mean, standard deviation, pass rate)++### Examples++Basic run:+```bash+letta-evals run suite.yaml # Run evaluation, show results in terminal+```++Save results:+```bash+letta-evals run suite.yaml --output evaluation-results/ # Save to directory+```++Letta Cloud:+```bash+letta-evals run suite.yaml \+ --base-url https://api.letta.com \+ --api-key $LETTA_API_KEY \+ --project-id proj_abc123+```++Quiet CI mode:+```bash+letta-evals run suite.yaml --quiet+if [ $? -eq 0 ]; then+ echo "Evaluation passed"+else+ echo "Evaluation failed"+ exit 1+fi+```++### Exit Codes++- `0`: Evaluation passed (gate criteria met)+- `1`: Evaluation failed (gate criteria not met or error)++## validate++Validate a suite configuration without running it.++```bash+letta-evals validate <suite.yaml>+```++Checks:+- YAML syntax is valid+- Required fields are present+- Paths exist+- Configuration is consistent+- Grader/extractor combinations are valid++Output on success:+```+✓ Suite configuration is valid+```++Output on error:+```+✗ Validation failed:+ - Agent file not found: agent.af+ - Grader 'my_metric' references unknown function+```++## list-extractors++List all available extractors.++```bash+letta-evals list-extractors+```++Output:+```+Available extractors:+ last_assistant - Extract the last assistant message+ first_assistant - Extract the first assistant message+ all_assistant - Concatenate all assistant messages+ pattern - Extract content matching regex+ tool_arguments - Extract tool call arguments+ tool_output - Extract tool return value+ after_marker - Extract content after a marker+ memory_block - Extract from memory block (requires agent_state)+```++## list-graders++List all available grader functions.++```bash+letta-evals list-graders+```++Output:+```+Available graders:+ exact_match - Exact string match with ground_truth+ contains - Check if contains ground_truth+ regex_match - Match regex pattern+ ascii_printable_only - Validate ASCII-only content+```++## help++Show help information.++```bash+letta-evals --help+```++Show help for a specific command:++```bash+letta-evals run --help+letta-evals validate --help+```++## Environment Variables++### LETTA_API_KEY+API key for Letta authentication.++```bash+export LETTA_API_KEY=your-key-here+```++### LETTA_BASE_URL+Letta server base URL.++```bash+export LETTA_BASE_URL=http://localhost:8283+```++### LETTA_PROJECT_ID+Letta project ID (for cloud).++```bash+export LETTA_PROJECT_ID=proj_abc123+```++### OPENAI_API_KEY+OpenAI API key (for rubric graders).++```bash+export OPENAI_API_KEY=your-openai-key+```++## Configuration Priority++Configuration values are resolved in this order (highest to lowest priority):++1. CLI arguments (`--api-key`, `--base-url`, `--project-id`)+2. Suite YAML configuration+3. Environment variables++## Using in CI/CD++### GitHub Actions++```yaml+name: Run Evals+on: [push]++jobs:+ evaluate:+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v2++ - name: Install dependencies+ run: pip install letta-evals++ - name: Run evaluation+ env:+ LETTA_API_KEY: ${{ secrets.LETTA_API_KEY }}+ OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}+ run: |+ letta-evals run suite.yaml --quiet --output results/++ - name: Upload results+ uses: actions/upload-artifact@v2+ with:+ name: eval-results+ path: results/+```++### GitLab CI++```yaml+evaluate:+ script:+ - pip install letta-evals+ - letta-evals run suite.yaml --quiet --output results/+ artifacts:+ paths:+ - results/+ variables:+ LETTA_API_KEY: $LETTA_API_KEY+ OPENAI_API_KEY: $OPENAI_API_KEY+```++## Debugging++### Common Issues++<Warning>+**"Agent file not found"**++```bash+# Check file exists relative to suite YAML location+ls -la path/to/agent.af+```+</Warning>++<Warning>+**"Connection refused"**++```bash+# Verify Letta server is running+curl http://localhost:8283/v1/health+```+</Warning>++<Warning>+**"Invalid API key"**++```bash+# Check environment variable is set+echo $LETTA_API_KEY+```+</Warning>++## Next Steps++- [Understanding Results](/evals/results-metrics/understanding-results) - Interpreting evaluation output+- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete configuration options+- [Getting Started](/evals/get-started/getting-started) - Complete tutorial with examplesdiff --git a/fern/pages/evals/concepts/datasets.mdx b/fern/pages/evals/concepts/datasets.mdxnew file mode 100644index 000000000..70c031839--- /dev/null+++ b/fern/pages/evals/concepts/datasets.mdx@@ -0,0 +1,425 @@+# Datasets++**Datasets** are the test cases that define what your agent will be evaluated on. Each sample in your dataset represents one evaluation scenario.++<Note>+**Quick overview:**+- **Two formats**: JSONL (flexible, powerful) or CSV (simple, spreadsheet-friendly)+- **Required field**: `input` - the prompt(s) to send to the agent+- **Common fields**: `ground_truth` (expected answer), `tags` (for filtering), `metadata` (extra info)+- **Advanced fields**: `agent_args` (customize agent per sample), `rubric_vars` (per-sample rubric context)+- **Multi-turn support**: Send multiple messages in sequence using arrays+</Note>++**Typical workflow:**+1. Create a JSONL or CSV file with test cases+2. Reference it in your suite YAML: `dataset: test_cases.jsonl`+3. Run evaluation - each sample is tested independently+4. Results show per-sample and aggregate scores++Datasets can be created in two formats: **JSONL** or **CSV**. Choose based on your team's workflow and complexity needs.++## Dataset Formats++### JSONL Format++Each line is a JSON object representing one test case:++```jsonl+{"input": "What's the capital of France?", "ground_truth": "Paris"}+{"input": "Calculate 2+2", "ground_truth": "4"}+{"input": "What color is the sky?", "ground_truth": "blue"}+```++**Best for:**+- Complex data structures (nested objects, arrays)+- Multi-turn conversations+- Advanced features (agent_args, rubric_vars)+- Teams comfortable with JSON/code+- Version control (clean line-by-line diffs)++### CSV Format++Standard CSV with headers:++```csv+input,ground_truth+"What's the capital of France?","Paris"+"Calculate 2+2","4"+"What color is the sky?","blue"+```++**Best for:**+- Simple question-answer pairs+- Teams that prefer spreadsheets (Excel, Google Sheets)+- Non-technical collaborators creating test cases+- Quick dataset creation and editing+- Easy sharing with non-developers++## Quick Reference++| Field | Required | Type | Purpose |+|-------|----------|------|---------|+| `input` | ✅ | string or array | Prompt(s) to send to agent |+| `ground_truth` | ❌ | string | Expected answer (for tool graders) |+| `tags` | ❌ | array of strings | For filtering samples |+| `agent_args` | ❌ | object | Per-sample agent customization |+| `rubric_vars` | ❌ | object | Per-sample rubric variables |+| `metadata` | ❌ | object | Arbitrary extra data |+| `id` | ❌ | integer | Sample ID (auto-assigned if omitted) |++## Field Reference++### Required Fields++#### input+The prompt(s) to send to the agent. Can be a string or array of strings:++Single message:+```json+{"input": "Hello, who are you?"}+```++Multi-turn conversation:+```json+{"input": ["Hello", "What's your name?", "Tell me about yourself"]}+```++### Optional Fields++#### ground_truth+The expected answer or content to check against. Required for most tool graders (exact_match, contains, etc.):++```json+{"input": "What is 2+2?", "ground_truth": "4"}+```++#### metadata+Arbitrary additional data about the sample:++```json+{+ "input": "What is photosynthesis?",+ "ground_truth": "process where plants convert light into energy",+ "metadata": {+ "category": "biology",+ "difficulty": "medium"+ }+}+```++#### tags+List of tags for filtering samples:++```json+{"input": "Solve x^2 = 16", "ground_truth": "4", "tags": ["math", "algebra"]}+```++Filter by tags in your suite:+```yaml+sample_tags: [math] # Only samples tagged "math" will be evaluated+```++#### agent_args++Custom arguments passed to programmatic agent creation when using `agent_script`. Allows per-sample agent customization.++JSONL:+```json+{+ "input": "What items do we have?",+ "agent_args": {+ "item": {"sku": "SKU-123", "name": "Widget A", "price": 19.99}+ }+}+```++CSV:+```csv+input,agent_args+"What items do we have?","{""item"": {""sku"": ""SKU-123"", ""name"": ""Widget A"", ""price"": 19.99}}"+```++Your agent factory function can access these values via `sample.agent_args` to customize agent configuration.++See [Targets - agent_script](/evals/core-concepts/targets#agent_script) for details on programmatic agent creation.++#### rubric_vars++Variables to inject into rubric templates when using rubric graders. This allows you to provide per-sample context or examples to the LLM judge.++**Example:** Evaluating code quality against a reference implementation.++JSONL:+```jsonl+{"input": "Write a function to calculate fibonacci numbers", "rubric_vars": {"reference_code": "def fib(n):\n if n <= 1: return n\n return fib(n-1) + fib(n-2)", "required_features": "recursion, base case"}}+```++CSV:+```csv+input,rubric_vars+"Write a function to calculate fibonacci numbers","{""reference_code"": ""def fib(n):\n if n <= 1: return n\n return fib(n-1) + fib(n-2)"", ""required_features"": ""recursion, base case""}"+```++In your rubric template file, reference variables with `{variable_name}`:++**rubric.txt:**+```+Evaluate the submitted code against this reference implementation:++{reference_code}++Required features: {required_features}++Score on correctness (0.6) and code quality (0.4).+```++When the rubric grader runs, variables are replaced with values from `rubric_vars`:++**Final formatted prompt sent to LLM:**+```+Evaluate the submitted code against this reference implementation:++def fib(n):+ if n <= 1: return n+ return fib(n-1) + fib(n-2)++Required features: recursion, base case++Score on correctness (0.6) and code quality (0.4).+```++This lets you customize evaluation criteria per sample using the same rubric template.++See [Rubric Graders](/evals/graders/rubric-graders) for details on rubric templates.++#### id+Sample ID is automatically assigned (0-based index) if not provided. You can override:++```json+{"id": 42, "input": "Test case 42"}+```++## Complete Example++```jsonl+{"id": 1, "input": "What is the capital of France?", "ground_truth": "Paris", "tags": ["geography", "easy"], "metadata": {"region": "Europe"}}+{"id": 2, "input": "Calculate the square root of 144", "ground_truth": "12", "tags": ["math", "medium"]}+{"id": 3, "input": ["Hello", "What can you help me with?"], "tags": ["conversation"]}+```++## Dataset Best Practices++### 1. Clear Ground Truth++Make ground truth specific enough to grade but flexible enough to match valid responses:++<Tip>+Good:+```json+{"input": "What's the largest planet?", "ground_truth": "Jupiter"}+```+</Tip>++<Warning>+Too strict (might miss valid answers):+```json+{"input": "What's the largest planet?", "ground_truth": "Jupiter is the largest planet in our solar system."}+```+</Warning>++### 2. Diverse Test Cases++Include edge cases and variations:++```jsonl+{"input": "What is 2+2?", "ground_truth": "4", "tags": ["math", "easy"]}+{"input": "What is 0.1 + 0.2?", "ground_truth": "0.3", "tags": ["math", "floating_point"]}+{"input": "What is 999999999 + 1?", "ground_truth": "1000000000", "tags": ["math", "large_numbers"]}+```++### 3. Use Tags for Organization++Organize samples by type, difficulty, or feature:++```json+{"tags": ["tool_usage", "search"]}+{"tags": ["memory", "recall"]}+{"tags": ["reasoning", "multi_step"]}+```++### 4. Multi-Turn Conversations++Test conversational context and memory updates:++```jsonl+{"input": ["My name is Alice", "What's my name?"], "ground_truth": "Alice", "tags": ["memory", "recall"]}+{"input": ["Please remember that I like bananas.", "Actually, sorry, I meant I like apples."], "ground_truth": "apples", "tags": ["memory", "correction"]}+{"input": ["I work at Google", "Update my workplace to Microsoft", "Where do I work?"], "ground_truth": "Microsoft", "tags": ["memory", "multi_step"]}+```++<Tip>+**Testing memory corrections:** Use multi-turn inputs to test if agents properly update memory when users correct themselves. Combine with the `memory_block` extractor to verify the final memory state, not just the response.+</Tip>++### 5. No Ground Truth for LLM Judges++If using rubric graders, ground truth is optional:++```jsonl+{"input": "Write a creative story about a robot", "tags": ["creative"]}+{"input": "Explain quantum computing simply", "tags": ["explanation"]}+```++The LLM judge evaluates based on the rubric, not ground truth.++## Loading Datasets++Datasets are automatically loaded by the runner:++```yaml+dataset: path/to/dataset.jsonl # Path to your test cases (JSONL or CSV)+```++Paths are relative to the suite YAML file location.++## Dataset Filtering++### Limit Sample Count++```yaml+max_samples: 10 # Only evaluate first 10 samples (useful for testing)+```++### Filter by Tags++```yaml+sample_tags: [math, medium] # Only samples with ALL these tags+```++## Creating Datasets Programmatically++You can generate datasets with Python:++```python+import json++samples = []+for i in range(100):+ samples.append({+ "input": f"What is {i} + {i}?",+ "ground_truth": str(i + i),+ "tags": ["math", "addition"]+ })++with open("dataset.jsonl", "w") as f:+ for sample in samples:+ f.write(json.dumps(sample) + "\n")+```++## Dataset Format Validation++The runner validates:+- Each line is valid JSON+- Required fields are present+- Field types are correct++Validation errors will be reported with line numbers.++## Examples by Use Case++### Question Answering++JSONL:+```jsonl+{"input": "What is the capital of France?", "ground_truth": "Paris"}+{"input": "Who wrote Romeo and Juliet?", "ground_truth": "Shakespeare"}+```++CSV:+```csv+input,ground_truth+"What is the capital of France?","Paris"+"Who wrote Romeo and Juliet?","Shakespeare"+```++### Tool Usage Testing++JSONL:+```jsonl+{"input": "Search for information about pandas", "ground_truth": "search"}+{"input": "Calculate 15 * 23", "ground_truth": "calculator"}+```++CSV:+```csv+input,ground_truth+"Search for information about pandas","search"+"Calculate 15 * 23","calculator"+```++Ground truth = expected tool name.++### Memory Testing (Multi-turn)++JSONL:+```jsonl+{"input": ["Remember that my favorite color is blue", "What's my favorite color?"], "ground_truth": "blue"}+{"input": ["I live in Tokyo", "Where do I live?"], "ground_truth": "Tokyo"}+```++CSV (using JSON array strings):+```csv+input,ground_truth+"[""Remember that my favorite color is blue"", ""What's my favorite color?""]","blue"+"[""I live in Tokyo"", ""Where do I live?""]","Tokyo"+```++### Code Generation++JSONL:+```jsonl+{"input": "Write a function to reverse a string in Python"}+{"input": "Create a SQL query to find users older than 21"}+```++CSV:+```csv+input+"Write a function to reverse a string in Python"+"Create a SQL query to find users older than 21"+```++Use rubric graders to evaluate code quality.++## CSV Advanced Features++CSV supports all the same features as JSONL by encoding complex data as JSON strings in cells:++**Multi-turn conversations** (requires escaped JSON array string):+```csv+input,ground_truth+"[""Hello"", ""What's your name?""]","Alice"+```++**Agent arguments** (requires escaped JSON object string):+```csv+input,agent_args+"What items do we have?","{""initial_inventory"": [""apple"", ""banana""]}"+```++**Rubric variables** (requires escaped JSON object string):+```csv+input,rubric_vars+"Write a story","{""max_length"": 500, ""genre"": ""sci-fi""}"+```++<Note>+**Note:** Complex data structures require JSON encoding in CSV. If you're frequently using these advanced features, JSONL may be easier to read and maintain.+</Note>++## Next Steps++- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete configuration options including filtering+- [Graders](/evals/core-concepts/graders) - How to evaluate agent responses+- [Multi-Turn Conversations](/evals/advanced/multi-turn-conversations) - Testing conversational flowsdiff --git a/fern/pages/evals/concepts/extractors.mdx b/fern/pages/evals/concepts/extractors.mdxnew file mode 100644index 000000000..247e7904c--- /dev/null+++ b/fern/pages/evals/concepts/extractors.mdx@@ -0,0 +1,374 @@+# Extractors++**Extractors** select what content to evaluate from an agent's response. They navigate the conversation trajectory and extract the specific piece you want to grade.++<Note>+**Quick overview:**+- **Purpose**: Agent responses are complex (messages, tool calls, memory) - extractors isolate what to grade+- **Built-in options**: last_assistant, tool_arguments, memory_block, pattern, and more+- **Flexible**: Different graders can use different extractors in the same suite+- **Automatic**: No setup needed - just specify in your grader config+</Note>++**Common patterns:**+- `last_assistant` - Most common, gets the agent's final message (90% of use cases)+- `tool_arguments` - Verify agent called the right tool with correct args+- `memory_block` - Check if agent updated memory correctly+- `pattern` - Extract structured data with regex++Extractors determine what part of the agent's response gets graded. They pull out specific content from the conversation trajectory.++## Why Extractors?++An agent's response is complex - it includes assistant messages, tool calls, tool returns, memory updates, etc. Extractors let you focus on exactly what you want to evaluate.++**The evaluation flow:**+```+Agent Response → Extractor → Submission Text → Grader → Score+```++For example:+```+Full trajectory:+ UserMessage: "What's the capital of France?"+ ToolCallMessage: search(query="capital of france")+ ToolReturnMessage: "Paris is the capital..."+ AssistantMessage: "The capital of France is Paris."++↓ extractor: last_assistant ↓++Extracted: "The capital of France is Paris."++↓ grader: contains (ground_truth="Paris") ↓++Score: 1.0+```++## Trajectory Structure++A trajectory is a list of turns, where each turn is a list of Letta messages:++```python+[+ [UserMessage(...), AssistantMessage(...), ToolCallMessage(...), ToolReturnMessage(...)], # Turn 1+ [AssistantMessage(...)] # Turn 2+]+```++Extractors navigate this structure to pull out the submission text.++## Built-in Extractors++### last_assistant++Extracts the last assistant message content.++```yaml+graders:+ quality:+ kind: tool+ function: contains+ extractor: last_assistant # Extract final agent message+```++Most common extractor - gets the agent's final response.++### first_assistant++Extracts the first assistant message content.++```yaml+graders:+ initial_response:+ kind: tool+ function: contains+ extractor: first_assistant # Extract first agent message+```++Useful for testing immediate responses before tool usage.++### all_assistant++Concatenates all assistant messages with a separator.++```yaml+graders:+ complete_response:+ kind: rubric+ prompt_path: rubric.txt+ extractor: all_assistant # Concatenate all agent messages+ extractor_config:+ separator: "\n\n" # Join messages with double newline+```++Use when you need the full conversation context.++### last_turn++Extracts all assistant messages from the last turn only.++```yaml+graders:+ final_turn:+ kind: tool+ function: contains+ extractor: last_turn # Messages from final turn only+ extractor_config:+ separator: " " # Join with spaces+```++Useful when the agent makes multiple statements in the final turn.++### pattern++Extracts content matching a regex pattern from assistant messages.++```yaml+graders:+ extract_number:+ kind: tool+ function: exact_match+ extractor: pattern # Extract using regex+ extractor_config:+ pattern: 'Result: (\d+)' # Regex pattern to match+ group: 1 # Extract capture group 1+ search_all: false # Only find first match+```++Example: Extract "42" from "The answer is Result: 42"++### tool_arguments++Extracts arguments from a specific tool call.++```yaml+graders:+ search_query:+ kind: tool+ function: contains+ extractor: tool_arguments # Extract tool call arguments+ extractor_config:+ tool_name: search # Which tool to extract from+```++Returns the JSON arguments as a string.++Example: If agent calls `search(query="pandas", limit=10)`, extracts:+```json+{"query": "pandas", "limit": 10}+```++### tool_output++Extracts the return value from a specific tool call.++```yaml+graders:+ search_results:+ kind: tool+ function: contains+ extractor: tool_output # Extract tool return value+ extractor_config:+ tool_name: search # Which tool's output to extract+```++Finds the tool call and its corresponding return message.++### after_marker++Extracts content after a specific marker string.++```yaml+graders:+ answer_section:+ kind: tool+ function: contains+ extractor: after_marker # Extract content after marker+ extractor_config:+ marker: "ANSWER:" # Marker string to find+ include_marker: false # Don't include "ANSWER:" in output+```++Example: From "Here's my analysis... ANSWER: Paris", extracts "Paris"++### memory_block++Extracts content from a specific memory block (requires agent_state).++```yaml+graders:+ human_memory:+ kind: tool+ function: exact_match+ extractor: memory_block # Extract from agent memory+ extractor_config:+ block_label: human # Which memory block to extract+```++<Warning>+**Important**: This extractor requires the agent's final state, which adds overhead. The runner automatically fetches agent_state when this extractor is used.+</Warning>++Example use case: Verify the agent correctly updated its memory about the user.++## Extractor Configuration++Some extractors accept additional configuration via `extractor_config`:++```yaml+graders:+ my_metric:+ kind: tool+ function: contains+ extractor: pattern # Use pattern extractor+ extractor_config: # Configuration for this extractor+ pattern: 'Answer: (.*)' # Regex pattern+ group: 1 # Extract capture group 1+```++## Choosing an Extractor++| Use Case | Recommended Extractor |+|----------|---------------------|+| Final agent response | `last_assistant` |+| First response before tools | `first_assistant` |+| Complete conversation | `all_assistant` |+| Specific format extraction | `pattern` |+| Tool usage validation | `tool_arguments` |+| Tool result checking | `tool_output` |+| Memory validation | `memory_block` |+| Structured output | `after_marker` |++## Content Flattening++Assistant messages can contain multiple content parts. Extractors automatically flatten complex content to plain text.++## Empty Extraction++If an extractor finds no matching content, it returns an empty string `""`. This typically results in a score of 0.0 from the grader.++## Custom Extractors++You can write custom extractors. See [Custom Extractors](/evals/extractors/custom-extractors) for details.++Example:++```python+from letta_evals.decorators import extractor+from letta_client import LettaMessageUnion++@extractor+def my_extractor(trajectory: List[List[LettaMessageUnion]], config: dict) -> str:+ # Custom extraction logic+ return extracted_text+```++Register by importing in your suite's setup script or custom evaluators file.++## Multi-Metric Extraction++Different graders can use different extractors:++```yaml+graders:+ response_quality: # Evaluate final message quality+ kind: rubric+ prompt_path: quality.txt+ extractor: last_assistant # Extract final response++ tool_usage: # Check tool was called correctly+ kind: tool+ function: exact_match+ extractor: tool_arguments # Extract tool args+ extractor_config:+ tool_name: search # From search tool++ memory_update: # Verify memory updated+ kind: tool+ function: contains+ extractor: memory_block # Extract from memory+ extractor_config:+ block_label: human # Human memory block+```++Each grader independently extracts and evaluates different aspects.++## Listing Extractors++See all available extractors:++```bash+letta-evals list-extractors+```++## Examples++### Extract Final Answer++```yaml+extractor: last_assistant # Get final agent message+```++Agent: "Let me search... *uses tool* ... The answer is Paris."+Extracted: "The answer is Paris."++### Extract Tool Arguments++```yaml+extractor: tool_arguments # Get tool call args+extractor_config:+ tool_name: search # From search tool+```++Agent calls: `search(query="pandas", limit=5)`+Extracted: `{"query": "pandas", "limit": 5}`++### Extract Pattern++```yaml+extractor: pattern # Extract with regex+extractor_config:+ pattern: 'RESULT: (\w+)' # Match pattern+ group: 1 # Extract capture group 1+```++Agent: "After calculation... RESULT: SUCCESS"+Extracted: "SUCCESS"++### Extract Memory++```yaml+extractor: memory_block # Extract from agent memory+extractor_config:+ block_label: human # Human memory block+```++Agent updates memory block "human" to: "User's name is Alice"+Extracted: "User's name is Alice"++## Troubleshooting++<Warning>+**Extractor returns empty string**++**Problem**: Grader always gives score 0.0 because extractor finds nothing.++**Common causes**:+- **Wrong extractor**: Using `first_assistant` but agent doesn't respond until after tool use → use `last_assistant`+- **Wrong tool name**: `tool_arguments` with `tool_name: "search"` but agent calls `"web_search"` → check actual tool name+- **Wrong memory block**: `memory_block` with `block_label: "user"` but block is actually labeled `"human"` → check block labels+- **Pattern doesn't match**: `pattern: "Answer: (.*)"` but agent says "The answer is..." → adjust regex+</Warning>++<Tip>+**Debug tips**:+1. Check the trajectory in results JSON to see actual agent output+2. Use `last_assistant` first to see what's there+3. Verify tool names with `letta-evals list-extractors`+</Tip>++## Next Steps++- [Built-in Extractors Reference](/evals/extractors/built-in-extractors) - Complete extractor documentation+- [Custom Extractors Guide](/evals/extractors/custom-extractors) - Write your own extractors+- [Graders](/evals/core-concepts/graders) - How to use extractors with gradersdiff --git a/fern/pages/evals/concepts/gates.mdx b/fern/pages/evals/concepts/gates.mdxnew file mode 100644index 000000000..29d5f31d2--- /dev/null+++ b/fern/pages/evals/concepts/gates.mdx@@ -0,0 +1,384 @@+# Gates++**Gates** are the pass/fail criteria for your evaluation. They determine whether your agent meets the required performance threshold by checking aggregate metrics.++<Note>+**Quick overview:**+- **Single decision**: One gate per suite determines pass/fail+- **Two metrics**: `avg_score` (average of all scores) or `accuracy` (percentage passing threshold)+- **Flexible operators**: `>=`, `>`, `<=`, `<`, `==` for threshold comparison+- **Customizable pass criteria**: Define what counts as "passing" for accuracy calculations+- **Exit codes**: Suite exits 0 for pass, 1 for fail+</Note>++**Common patterns:**+- Average score must be 80%+: `avg_score >= 0.8`+- 90%+ of samples must pass: `accuracy >= 0.9`+- Custom threshold: Define per-sample pass criteria with `pass_value`++Gates define the pass/fail criteria for your evaluation. They check if aggregate metrics meet a threshold.++## Basic Structure++```yaml+gate:+ metric_key: accuracy # Which grader to evaluate+ metric: avg_score # Use average score (default)+ op: gte # Greater than or equal+ value: 0.8 # 80% threshold+```++## Why Use Gates?++Gates provide **automated pass/fail decisions** for your evaluations, which is essential for:++**CI/CD Integration**: Gates let you block deployments if agent performance drops:+```bash+letta-evals run suite.yaml+# Exit code 0 = pass (continue deployment)+# Exit code 1 = fail (block deployment)+```++**Regression Testing**: Set a baseline threshold and ensure new changes don't degrade performance:+```yaml+gate:+ metric: avg_score+ op: gte+ value: 0.85 # Must maintain 85%+ to pass+```++**Quality Enforcement**: Require agents meet minimum standards before production:+```yaml+gate:+ metric: accuracy+ op: gte+ value: 0.95 # 95% of test cases must pass+```++### What Happens When Gates Fail?++When a gate condition is not met:++1. **Console output** shows failure message:+ ```text+ ✗ FAILED (0.72/1.00 avg, 72.0% pass rate)+ Gate check failed: avg_score (0.72) not >= 0.80+ ```++2. **Exit code** is 1 (non-zero indicates failure):+ ```bash+ letta-evals run suite.yaml+ echo $? # Prints 1 if gate failed+ ```++3. **Results JSON** includes `gate_passed: false`:+ ```json+ {+ "gate_passed": false,+ "gate_check": {+ "metric": "avg_score",+ "value": 0.72,+ "threshold": 0.80,+ "operator": "gte",+ "passed": false+ },+ "metrics": { ... }+ }+ ```++4. **All other data is preserved** - you still get full results, scores, and trajectories even when gating fails++<Tip>+**Common use case in CI**:++```bash+#!/bin/bash+letta-evals run suite.yaml --output results.json++if [ $? -ne 0 ]; then+ echo "❌ Agent evaluation failed - blocking merge"+ exit 1+else+ echo "✅ Agent evaluation passed - safe to merge"+fi+```+</Tip>++## Required Fields++### metric_key++Which grader to evaluate. Must match a key in your `graders` section:++```yaml+graders:+ accuracy: # Grader name+ kind: tool+ function: exact_match+ extractor: last_assistant++gate:+ metric_key: accuracy # Must match grader name above+ op: gte # >=+ value: 0.8 # 80% threshold+```++If you only have one grader, `metric_key` can be omitted - it will default to your single grader.++### metric++Which aggregate statistic to compare. Two options:++#### avg_score++Average score across all samples (0.0 to 1.0):++```yaml+gate:+ metric_key: quality # Check quality grader+ metric: avg_score # Use average of all scores+ op: gte # >=+ value: 0.7 # Must average 70%++```++Example: If scores are [0.8, 0.9, 0.6], avg_score = 0.77++#### accuracy++Pass rate as a percentage (0.0 to 1.0):++```yaml+gate:+ metric_key: accuracy # Check accuracy grader+ metric: accuracy # Use pass rate, not average+ op: gte # >=+ value: 0.8 # 80% of samples must pass+```++By default, samples with score `>= 1.0` are considered "passing".++You can customize the per-sample threshold with `pass_op` and `pass_value` (see below).++<Note>+**Note**: The default `metric` is `avg_score`, so you can omit it if that's what you want:++```yaml+gate:+ metric_key: quality # Check quality grader+ op: gte # >=+ value: 0.7 # 70% threshold (defaults to avg_score)+```+</Note>++### op++Comparison operator:++- `gte`: Greater than or equal (`>=`)+- `gt`: Greater than (`>`)+- `lte`: Less than or equal (`<=`)+- `lt`: Less than (`<`)+- `eq`: Equal (`==`)++Most common: `gte` (at least X)++### value++Threshold value for comparison:++- For `avg_score`: 0.0 to 1.0+- For `accuracy`: 0.0 to 1.0 (representing percentage)++```yaml+gate:+ metric: avg_score # Average score+ op: gte # >=+ value: 0.75 # 75% threshold+```++```yaml+gate:+ metric: accuracy # Pass rate+ op: gte # >=+ value: 0.9 # 90% must pass+```++## Optional Fields++### pass_op and pass_value++Customize when individual samples are considered "passing" (used for accuracy calculation):++```yaml+gate:+ metric_key: quality # Check quality grader+ metric: accuracy # Use pass rate+ op: gte # >=+ value: 0.8 # 80% must pass+ pass_op: gte # Sample passes if >=+ pass_value: 0.7 # This threshold (70%)+```++Default behavior:+- If `metric` is `avg_score`: samples pass if score `>=` the gate value+- If `metric` is `accuracy`: samples pass if score `>= 1.0` (perfect)++## Examples++### Require 80% Average Score++```yaml+gate:+ metric_key: quality # Check quality grader+ metric: avg_score # Use average+ op: gte # >=+ value: 0.8 # 80% average+```++Passes if the average score across all samples is `>= 0.8`++### Require 90% Pass Rate (Perfect Scores)++```yaml+gate:+ metric_key: accuracy # Check accuracy grader+ metric: accuracy # Use pass rate+ op: gte # >=+ value: 0.9 # 90% must pass (default: score >= 1.0 to pass)+```++Passes if 90% of samples have score = 1.0++### Require 75% Pass Rate (Score `>= 0.7`)++```yaml+gate:+ metric_key: quality # Check quality grader+ metric: accuracy # Use pass rate+ op: gte # >=+ value: 0.75 # 75% must pass+ pass_op: gte # Sample passes if >=+ pass_value: 0.7 # 70% threshold per sample+```++Passes if 75% of samples have score `>= 0.7`++### Maximum Error Rate++```yaml+gate:+ metric_key: quality # Check quality grader+ metric: accuracy # Use pass rate+ op: gte # >=+ value: 0.95 # 95% must pass (allows 5% failures)+ pass_op: gt # Sample passes if >+ pass_value: 0.0 # 0.0 (any non-zero score)+```++Allows up to 5% failures.++### Exact Pass Rate++```yaml+gate:+ metric_key: quality # Check quality grader+ metric: accuracy # Use pass rate+ op: eq # Exactly equal+ value: 1.0 # 100% (all samples must pass)+```++All samples must pass.++## Multi-Metric Gating++When you have multiple graders, you can only gate on one metric:++```yaml+graders:+ accuracy: # First metric+ kind: tool+ function: exact_match+ extractor: last_assistant++ completeness: # Second metric+ kind: rubric+ prompt_path: completeness.txt+ model: gpt-4o-mini+ extractor: last_assistant++gate:+ metric_key: accuracy # Only gate on accuracy (completeness still computed)+ metric: avg_score # Use average+ op: gte # >=+ value: 0.8 # 80% threshold+```++The evaluation passes/fails based on the gated metric, but results include scores for all metrics.++## Understanding avg_score vs accuracy++### avg_score+- Arithmetic mean of all scores+- Sensitive to partial credit+- Good for continuous evaluation++Example:+- Scores: [1.0, 0.8, 0.6]+- avg_score = (1.0 + 0.8 + 0.6) / 3 = 0.8++### accuracy+- Percentage of samples meeting a threshold+- Binary pass/fail per sample+- Good for strict requirements++Example:+- Scores: [1.0, 0.8, 0.6]+- pass_value: 0.7+- Passing: [1.0, 0.8] = 2 out of 3+- accuracy = 2/3 = 0.667 (66.7%)++## Errors and Attempted Samples++If a sample fails (error during evaluation), it:+- Gets a score of 0.0+- Counts toward `total` but not `total_attempted`+- Included in `avg_score_total` but not `avg_score_attempted`++You can gate on either:+- `avg_score_total`: Includes errors as 0.0+- `avg_score_attempted`: Excludes errors (only successfully attempted samples)++<Note>+**Note**: The `metric` field currently only supports `avg_score` and `accuracy`. By default, gates use `avg_score_attempted`.+</Note>++## Gate Results++After evaluation, you'll see:++```text+✓ PASSED (2.25/3.00 avg, 75.0% pass rate)+```++or++```text+✗ FAILED (1.80/3.00 avg, 60.0% pass rate)+```++The evaluation exit code reflects the gate result:+- 0: Passed+- 1: Failed++## Advanced Gating++For complex gating logic (e.g., "pass if accuracy `>= 80%` OR avg_score `>= 0.9`"), you'll need to:+1. Run evaluation with one gate+2. Examine the results JSON+3. Apply custom logic in a post-processing script++## Next Steps++- [Understanding Results](/evals/results-metrics/understanding-results) - Interpreting evaluation output+- [Multi-Metric Evaluation](/evals/graders/multi-metric-grading) - Using multiple graders+- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete gate configurationdiff --git a/fern/pages/evals/concepts/graders.mdx b/fern/pages/evals/concepts/graders.mdxnew file mode 100644index 000000000..bc0e498e0--- /dev/null+++ b/fern/pages/evals/concepts/graders.mdx@@ -0,0 +1,330 @@+# Graders++**Graders** are the scoring functions that evaluate agent responses. They take the extracted submission (from an extractor) and assign a score between 0.0 (complete failure) and 1.0 (perfect success).++<Note>+**Quick overview:**+- **Two types**: Tool graders (deterministic Python functions) and Rubric graders (LLM-as-judge)+- **Built-in functions**: exact_match, contains, regex_match, ascii_printable_only+- **Custom graders**: Write your own grading logic+- **Multi-metric**: Combine multiple graders in one suite+- **Flexible extraction**: Each grader can use a different extractor+</Note>++**When to use each:**+- **Tool graders**: Fast, deterministic, free - perfect for exact matching, patterns, tool validation+- **Rubric graders**: Flexible, subjective, costs API calls - ideal for quality, creativity, nuanced evaluation++Graders evaluate agent responses and assign scores between 0.0 (complete failure) and 1.0 (perfect success).++## Grader Types++There are two types of graders:++### Tool Graders++Python functions that programmatically compare the submission to ground truth or apply deterministic checks.++```yaml+graders:+ accuracy:+ kind: tool # Deterministic grading+ function: exact_match # Built-in grading function+ extractor: last_assistant # Use final agent response+```++Best for:+- Exact matching+- Pattern checking+- Tool call validation+- Deterministic criteria++### Rubric Graders++LLM-as-judge evaluation using custom prompts and criteria. Can use either direct LLM API calls or a Letta agent as the judge.++**Standard rubric grading (LLM API):**+```yaml+graders:+ quality:+ kind: rubric # LLM-as-judge+ prompt_path: rubric.txt # Custom evaluation criteria+ model: gpt-4o-mini # Judge model+ extractor: last_assistant # What to evaluate+```++**Agent-as-judge (Letta agent):**+```yaml+graders:+ agent_judge:+ kind: rubric # Still "rubric" kind+ agent_file: judge.af # Judge agent with submit_grade tool+ prompt_path: rubric.txt # Evaluation criteria+ extractor: last_assistant # What to evaluate+```++Best for:+- Subjective quality assessment+- Open-ended responses+- Nuanced evaluation+- Complex criteria+- Judges that need tools (when using agent-as-judge)++## Built-in Tool Graders++### exact_match++Checks if submission exactly matches ground truth (case-sensitive, whitespace-trimmed).++```yaml+graders:+ accuracy:+ kind: tool+ function: exact_match # Case-sensitive, whitespace-trimmed+ extractor: last_assistant # Extract final response+```++Requires: `ground_truth` in dataset++Score: 1.0 if exact match, 0.0 otherwise++### contains++Checks if submission contains ground truth (case-insensitive).++```yaml+graders:+ contains_answer:+ kind: tool+ function: contains # Case-insensitive substring match+ extractor: last_assistant # Search in final response+```++Requires: `ground_truth` in dataset++Score: 1.0 if found, 0.0 otherwise++### regex_match++Checks if submission matches a regex pattern in ground truth.++```yaml+graders:+ pattern:+ kind: tool+ function: regex_match # Pattern matching+ extractor: last_assistant # Check final response+```++Dataset sample:+```json+{"input": "Generate a UUID", "ground_truth": "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"}+```++Score: 1.0 if pattern matches, 0.0 otherwise++### ascii_printable_only++Validates that all characters are printable ASCII (useful for ASCII art, formatted output).++```yaml+graders:+ ascii_check:+ kind: tool+ function: ascii_printable_only # Validate ASCII characters+ extractor: last_assistant # Check final response+```++Does not require ground truth.++Score: 1.0 if all characters are printable ASCII, 0.0 if any non-printable characters found++## Rubric Graders++Rubric graders use an LLM to evaluate responses based on custom criteria.++### Basic Configuration++```yaml+graders:+ quality:+ kind: rubric # LLM-as-judge+ prompt_path: quality_rubric.txt # Evaluation criteria+ model: gpt-4o-mini # Judge model+ temperature: 0.0 # Deterministic+ extractor: last_assistant # What to evaluate+```++### Rubric Prompt Format++Your rubric file should describe the evaluation criteria. Use placeholders:++- `{input}`: The original input from the dataset+- `{submission}`: The extracted agent response+- `{ground_truth}`: Ground truth from dataset (if available)++Example `quality_rubric.txt`:+```+Evaluate the response for:+1. Accuracy: Does it correctly answer the question?+2. Completeness: Is the answer thorough?+3. Clarity: Is it well-explained?++Input: {input}+Expected: {ground_truth}+Response: {submission}++Score from 0.0 to 1.0 where:+- 1.0: Perfect response+- 0.75: Good with minor issues+- 0.5: Acceptable but incomplete+- 0.25: Poor quality+- 0.0: Completely wrong+```++### Inline Prompt++Instead of a file, you can include the prompt inline:++```yaml+graders:+ quality:+ kind: rubric # LLM-as-judge+ prompt: | # Inline prompt instead of file+ Evaluate the creativity and originality of the response.+ Score 1.0 for highly creative, 0.0 for generic or unoriginal.+ model: gpt-4o-mini # Judge model+ extractor: last_assistant # What to evaluate+```++### Model Configuration++```yaml+graders:+ quality:+ kind: rubric+ prompt_path: rubric.txt # Evaluation criteria+ model: gpt-4o-mini # Judge model+ temperature: 0.0 # Deterministic (0.0-2.0)+ provider: openai # LLM provider (default: openai)+ max_retries: 5 # API retry attempts+ timeout: 120.0 # Request timeout in seconds+```++Supported providers:+- `openai` (default)++Models:+- Any OpenAI-compatible model+- Special handling for reasoning models (o1, o3) - temperature automatically adjusted to 1.0++### Structured Output++Rubric graders use JSON mode to get structured responses:++```json+{+ "score": 0.85,+ "rationale": "The response is accurate and complete but could be more concise."+}+```++The score is validated to be between 0.0 and 1.0.++## Multi-Metric Configuration++Evaluate multiple aspects in one suite:++```yaml+graders:+ accuracy: # Tool grader for factual correctness+ kind: tool+ function: contains+ extractor: last_assistant++ completeness: # Rubric grader for thoroughness+ kind: rubric+ prompt_path: completeness_rubric.txt+ model: gpt-4o-mini+ extractor: last_assistant++ tool_usage: # Tool grader for tool call validation+ kind: tool+ function: exact_match+ extractor: tool_arguments # Extract tool call args+ extractor_config:+ tool_name: search # Which tool to check+```++Each grader can use a different extractor.++## Extractor Configuration++Every grader must specify an `extractor` to select what to grade:++```yaml+graders:+ my_metric:+ kind: tool+ function: contains # Grading function+ extractor: last_assistant # What to extract and grade+```++Some extractors need additional configuration:++```yaml+graders:+ tool_check:+ kind: tool+ function: contains # Check if ground truth in tool args+ extractor: tool_arguments # Extract tool call arguments+ extractor_config: # Configuration for this extractor+ tool_name: search # Which tool to extract from+```++See [Extractors](/evals/core-concepts/extractors) for all available extractors.++## Custom Graders++You can write custom grading functions. See [Custom Graders](/evals/advanced/custom-graders) for details.++## Grader Selection Guide++| Use Case | Recommended Grader |+|----------|-------------------|+| Exact answer matching | `exact_match` |+| Keyword checking | `contains` |+| Pattern validation | `regex_match` |+| Tool call validation | `exact_match` with `tool_arguments` extractor |+| Quality assessment | Rubric grader |+| Creativity evaluation | Rubric grader |+| Format checking | Custom tool grader |+| Multi-criteria evaluation | Multiple graders |++## Score Interpretation++All scores are between 0.0 and 1.0:++- **1.0**: Perfect - meets all criteria+- **0.75-0.99**: Good - minor issues+- **0.5-0.74**: Acceptable - notable gaps+- **0.25-0.49**: Poor - major problems+- **0.0-0.24**: Failed - did not meet criteria++Tool graders typically return binary scores (0.0 or 1.0), while rubric graders can return any value in the range.++## Error Handling++If grading fails (e.g., network error, invalid format):+- Score is set to 0.0+- Rationale includes error message+- Metadata includes error details++This ensures evaluations can continue even with individual failures.++## Next Steps++- [Tool Graders](/evals/graders/tool-graders) - Built-in and custom functions+- [Rubric Graders](/evals/graders/rubric-graders) - LLM-as-judge details+- [Multi-Metric Evaluation](/evals/graders/multi-metric-grading) - Using multiple graders+- [Extractors](/evals/core-concepts/extractors) - Selecting what to gradediff --git a/fern/pages/evals/concepts/overview.mdx b/fern/pages/evals/concepts/overview.mdxnew file mode 100644index 000000000..9a83c0fac--- /dev/null+++ b/fern/pages/evals/concepts/overview.mdx@@ -0,0 +1,207 @@+# Core Concepts++Understanding how Letta Evals works and what makes it different.++<Note>+**Just want to run an eval?** Skip to [Getting Started](/evals/get-started/getting-started) for a hands-on quickstart.+</Note>++## Built for Stateful Agents++Letta Evals is a testing framework specifically designed for agents that maintain state. Unlike traditional eval frameworks built for simple input-output models, Letta Evals understands that agents:++- Maintain memory across conversations+- Use tools and external functions+- Evolve their behavior based on interactions+- Have persistent context and state++This means you can test aspects of your agent that other frameworks can't: memory updates, multi-turn conversations, tool usage patterns, and state evolution over time.++## The Evaluation Flow++Every evaluation follows this flow:++**Dataset → Target (Agent) → Extractor → Grader → Gate → Result**++1. **Dataset**: Your test cases (questions, scenarios, expected outputs)+2. **Target**: The agent being evaluated+3. **Extractor**: Pulls out the relevant information from the agent's response+4. **Grader**: Scores the extracted information+5. **Gate**: Pass/fail criteria for the overall evaluation+6. **Result**: Metrics, scores, and detailed results++### What You Can Test++With Letta Evals, you can test aspects of agents that traditional frameworks can't:++- **Memory updates**: Did the agent correctly remember the user's name?+- **Multi-turn conversations**: Can the agent maintain context across multiple exchanges?+- **Tool usage**: Does the agent call the right tools with the right arguments?+- **State evolution**: How does the agent's internal state change over time?++<Note>+**Example: Testing Memory Updates**++```yaml+graders:+ memory_check:+ kind: tool # Deterministic grading+ function: contains # Check if ground_truth in extracted content+ extractor: memory_block # Extract from agent memory (not just response!)+ extractor_config:+ block_label: human # Which memory block to check+```++Dataset:+```jsonl+{"input": "Please remember that I like bananas.", "ground_truth": "bananas"}+```++This doesn't just check if the agent responded correctly - it verifies the agent actually stored "bananas" in its memory block. Traditional eval frameworks can't inspect agent state like this.+</Note>++## Why Evals Matter++AI agents are complex systems that can behave unpredictably. Without systematic evaluation, you can't:+- **Know if changes improve or break your agent** - Did that prompt tweak help or hurt?+- **Prevent regressions** - Catch when "fixes" break existing functionality+- **Compare approaches objectively** - Which model works better for your use case?+- **Build confidence before deployment** - Ensure quality before shipping to users+- **Track improvement over time** - Measure progress as you iterate++Manual testing doesn't scale. Evals let you test hundreds of scenarios in minutes.++## What Evals Are Useful For++### 1. Development & Iteration+- Test prompt changes instantly across your entire test suite+- Experiment with different models and compare results+- Validate that new features work as expected++### 2. Quality Assurance+- Prevent regressions when modifying agent behavior+- Ensure agents handle edge cases correctly+- Verify tool usage and memory updates++### 3. Model Selection+- Compare GPT-4 vs Claude vs other models on your specific use case+- Test different model configurations (temperature, system prompts, etc.)+- Find the right cost/performance tradeoff++### 4. Benchmarking+- Measure agent performance on standard tasks+- Track improvements over time+- Share reproducible results with your team++### 5. Production Readiness+- Validate agents meet quality thresholds before deployment+- Run continuous evaluation in CI/CD pipelines+- Monitor production agent quality++## How Letta Evals Works++Letta Evals is built around a few key concepts that work together to create a flexible evaluation framework.++## Key Components++### Suite++An **evaluation suite** is a complete test configuration defined in a YAML file. It ties together:+- Which dataset to use+- Which agent to test+- How to grade responses+- What criteria determine pass/fail++Think of a suite as a reusable test specification.++### Dataset++A **dataset** is a JSONL file where each line represents one test case. Each sample has:+- An input (what to ask the agent)+- Optional ground truth (the expected answer)+- Optional metadata (tags, custom fields)++### Target++The **target** is what you're evaluating. Currently, this is a Letta agent, specified by:+- An agent file (.af)+- An existing agent ID+- A Python script that creates agents programmatically++### Trajectory++A **trajectory** is the complete conversation history from one test case. It's a list of turns, where each turn contains a list of Letta messages (assistant messages, tool calls, tool returns, etc.).++### Extractor++An **extractor** determines what part of the trajectory to evaluate. For example:+- The last thing the agent said+- All tool calls made+- Content from agent memory+- Text matching a pattern++### Grader++A **grader** scores how well the agent performed. There are two types:+- **Tool graders**: Python functions that compare submission to ground truth+- **Rubric graders**: LLM judges that evaluate based on custom criteria++### Gate++A **gate** is the pass/fail threshold for your evaluation. It compares aggregate metrics (like average score or pass rate) against a target value.++## Multi-Metric Evaluation++You can define multiple graders in one suite to evaluate different aspects:++```yaml+graders:+ accuracy: # Check if answer is correct+ kind: tool+ function: exact_match+ extractor: last_assistant # Use final response++ tool_usage: # Check if agent called the right tool+ kind: tool+ function: contains+ extractor: tool_arguments # Extract tool call args+ extractor_config:+ tool_name: search # From search tool+```++The gate can check any of these metrics:++```yaml+gate:+ metric_key: accuracy # Gate on accuracy (tool_usage still computed)+ op: gte # >=+ value: 0.8 # 80% threshold+```++## Score Normalization++All scores are normalized to the range [0.0, 1.0]:+- 0.0 = complete failure+- 1.0 = perfect success+- Values in between = partial credit++This allows different grader types to be compared and combined.++## Aggregate Metrics++Individual sample scores are aggregated in two ways:++1. **Average Score**: Mean of all scores (0.0 to 1.0)+2. **Accuracy/Pass Rate**: Percentage of samples passing a threshold++You can gate on either metric type.++## Next Steps++Dive deeper into each concept:+- [Suites](/evals/core-concepts/suites) - Suite configuration in detail+- [Datasets](/evals/core-concepts/datasets) - Creating effective test datasets+- [Targets](/evals/core-concepts/targets) - Agent configuration options+- [Graders](/evals/core-concepts/graders) - Understanding grader types+- [Extractors](/evals/core-concepts/extractors) - Extraction strategies+- [Gates](/evals/core-concepts/gates) - Setting pass/fail criteriadiff --git a/fern/pages/evals/concepts/suites.mdx b/fern/pages/evals/concepts/suites.mdxnew file mode 100644index 000000000..132d46c35--- /dev/null+++ b/fern/pages/evals/concepts/suites.mdx@@ -0,0 +1,275 @@+# Suites++A **suite** is a YAML configuration file that defines a complete evaluation specification. It's the central piece that ties together your dataset, target agent, grading criteria, and pass/fail thresholds.++<Note>+**Quick overview:**+- **Single file defines everything**: Dataset, agent, graders, and success criteria all in one YAML+- **Reusable and shareable**: Version control your evaluation specs alongside your code+- **Multi-metric support**: Evaluate multiple aspects (accuracy, quality, tool usage) in one run+- **Multi-model testing**: Run the same suite across different LLM models+- **Flexible filtering**: Test subsets using tags or sample limits+</Note>++**Typical workflow:**+1. Create a suite YAML defining what and how to test+2. Run `letta-evals run suite.yaml`+3. Review results showing scores for each metric+4. Suite passes or fails based on gate criteria++An evaluation suite is a YAML configuration file that defines a complete test specification.++## Basic Structure++```yaml+name: my-evaluation # Suite identifier+description: Optional description of what this tests # Human-readable explanation+dataset: path/to/dataset.jsonl # Test cases++target: # What agent to evaluate+ kind: agent+ agent_file: agent.af # Agent to test+ base_url: http://localhost:8283 # Letta server++graders: # How to evaluate responses+ my_metric:+ kind: tool # Deterministic grading+ function: exact_match # Grading function+ extractor: last_assistant # What to extract from agent response++gate: # Pass/fail criteria+ metric_key: my_metric # Which metric to check+ op: gte # Greater than or equal+ value: 0.8 # 80% threshold+```++## Required Fields++### name+The name of your evaluation suite. Used in output and results.++```yaml+name: question-answering-eval+```++### dataset+Path to the JSONL or CSV dataset file. Can be relative (to the suite YAML) or absolute.++```yaml+dataset: ./datasets/qa.jsonl # Relative to suite YAML location+```++### target+Specifies what agent to evaluate. See [Targets](/evals/core-concepts/targets) for details.++### graders+One or more graders to evaluate agent performance. See [Graders](/evals/core-concepts/graders) for details.++### gate+Pass/fail criteria. See [Gates](/evals/core-concepts/gates) for details.++## Optional Fields++### description+A human-readable description of what this suite tests:++```yaml+description: Tests the agent's ability to answer factual questions accurately+```++### max_samples+Limit the number of samples to evaluate (useful for quick tests):++```yaml+max_samples: 10 # Only evaluate first 10 samples+```++### sample_tags+Filter samples by tags (only evaluate samples with these tags):++```yaml+sample_tags: [math, easy] # Only samples tagged with "math" AND "easy"+```++Dataset samples can include tags:+```jsonl+{"input": "What is 2+2?", "ground_truth": "4", "tags": ["math", "easy"]}+```++### num_runs+Number of times to run the entire evaluation suite (useful for testing non-deterministic behavior):++```yaml+num_runs: 5 # Run the evaluation 5 times+```++Default: 1++### setup_script+Path to a Python script with a setup function to run before evaluation:++```yaml+setup_script: setup.py:prepare_environment # script.py:function_name+```++The setup function should have this signature:+```python+def prepare_environment(suite: SuiteSpec) -> None:+ # Setup code here+ pass+```++## Path Resolution++Paths in the suite YAML are resolved relative to the YAML file location:++```+project/+├── suite.yaml+├── dataset.jsonl+└── agents/+ └── my_agent.af+```++```yaml+# In suite.yaml+dataset: dataset.jsonl # Resolves to project/dataset.jsonl+target:+ agent_file: agents/my_agent.af # Resolves to project/agents/my_agent.af+```++Absolute paths are used as-is.++## Multi-Grader Suites++You can evaluate multiple metrics in one suite:++```yaml+graders:+ accuracy: # Check if answer is correct+ kind: tool+ function: exact_match+ extractor: last_assistant++ completeness: # LLM judges response quality+ kind: rubric+ prompt_path: rubrics/completeness.txt+ model: gpt-4o-mini+ extractor: last_assistant++ tool_usage: # Verify correct tool was called+ kind: tool+ function: contains+ extractor: tool_arguments # Extract tool call arguments+```++The gate can check any of these metrics:++```yaml+gate:+ metric_key: accuracy # Gate on accuracy metric (others still computed)+ op: gte # Greater than or equal+ value: 0.9 # 90% threshold+```++Results will include scores for all graders, even if you only gate on one.++## Examples++### Simple Tool Grader Suite++```yaml+name: basic-qa # Suite name+dataset: questions.jsonl # Test questions++target:+ kind: agent+ agent_file: qa_agent.af # Pre-configured agent+ base_url: http://localhost:8283 # Local server++graders:+ accuracy: # Single metric+ kind: tool # Deterministic grading+ function: contains # Check if ground truth is in response+ extractor: last_assistant # Use final agent message++gate:+ metric_key: accuracy # Gate on this metric+ op: gte # Must be >=+ value: 0.75 # 75% to pass+```++### Rubric Grader Suite++```yaml+name: quality-eval # Quality evaluation+dataset: prompts.jsonl # Test prompts++target:+ kind: agent+ agent_id: existing-agent-123 # Use existing agent+ base_url: https://api.letta.com # Letta Cloud++graders:+ quality: # LLM-as-judge metric+ kind: rubric # Subjective evaluation+ prompt_path: quality_rubric.txt # Rubric template+ model: gpt-4o-mini # Judge model+ temperature: 0.0 # Deterministic+ extractor: last_assistant # Evaluate final response++gate:+ metric_key: quality # Gate on this metric+ metric: avg_score # Use average score+ op: gte # Must be >=+ value: 0.7 # 70% to pass+```++### Multi-Model Suite++Test the same agent configuration across different models:++```yaml+name: model-comparison # Compare model performance+dataset: test.jsonl # Same test for all models++target:+ kind: agent+ agent_file: agent.af # Same agent configuration+ base_url: http://localhost:8283 # Local server+ model_configs: [gpt-4o-mini, claude-3-5-sonnet] # Test both models++graders:+ accuracy: # Single metric for comparison+ kind: tool+ function: exact_match+ extractor: last_assistant++gate:+ metric_key: accuracy # Both models must pass this+ op: gte # Must be >=+ value: 0.8 # 80% threshold+```++Results will show per-model metrics.++## Validation++Validate your suite configuration before running:++```bash+letta-evals validate suite.yaml+```++This checks:+- Required fields are present+- Paths exist+- Configuration is valid+- Grader/extractor combinations are compatible++## Next Steps++- [Dataset Configuration](/evals/core-concepts/datasets)+- [Target Configuration](/evals/core-concepts/targets)+- [Grader Configuration](/evals/core-concepts/graders)+- [Gate Configuration](/evals/core-concepts/gates)diff --git a/fern/pages/evals/concepts/targets.mdx b/fern/pages/evals/concepts/targets.mdxnew file mode 100644index 000000000..f5dfb1d23--- /dev/null+++ b/fern/pages/evals/concepts/targets.mdx@@ -0,0 +1,329 @@+# Targets++A **target** is the agent you're evaluating. In Letta Evals, the target configuration determines how agents are created, accessed, and tested.++<Note>+**Quick overview:**+- **Three ways to specify agents**: agent file (`.af`), existing agent ID, or programmatic creation script+- **Critical distinction**: `agent_file`/`agent_script` create fresh agents per sample (isolated tests), while `agent_id` uses one agent for all samples (stateful conversation)+- **Multi-model support**: Test the same agent configuration across different LLM models+- **Flexible connection**: Connect to local Letta servers or Letta Cloud+</Note>++**When to use each approach:**+- `agent_file` - Pre-configured agents saved as `.af` files (most common)+- `agent_id` - Testing existing agents or multi-turn conversations with state+- `agent_script` - Dynamic agent creation with per-sample customization++The target configuration specifies how to create or access the agent for evaluation.++## Target Configuration++All targets have a `kind` field (currently only `agent` is supported):++```yaml+target:+ kind: agent # Currently only "agent" is supported+ # ... agent-specific configuration+```++## Agent Sources++You must specify exactly ONE of these:++### agent_file++Path to a `.af` (Agent File) to upload:++```yaml+target:+ kind: agent+ agent_file: path/to/agent.af # Path to .af file+ base_url: http://localhost:8283 # Letta server URL+```++The agent file will be uploaded to the Letta server and a new agent created for the evaluation.++### agent_id++ID of an existing agent on the server:++```yaml+target:+ kind: agent+ agent_id: agent-123-abc # ID of existing agent+ base_url: http://localhost:8283 # Letta server URL+```++<Warning>+**Modifies agent in-place:** Using `agent_id` will modify your agent's state, memory, and message history during evaluation. The same agent instance is used for all samples, processing them sequentially. **Do not use production agents or agents you don't want to modify.** Use `agent_file` or `agent_script` for reproducible, isolated testing.+</Warning>++### agent_script++Path to a Python script with an agent factory function for programmatic agent creation:++```yaml+target:+ kind: agent+ agent_script: create_agent.py:create_inventory_agent # script.py:function_name+ base_url: http://localhost:8283 # Letta server URL+```++Format: `path/to/script.py:function_name`++The function must be decorated with `@agent_factory` and have the signature `async (client: AsyncLetta, sample: Sample) -> str`:++```python+from letta_client import AsyncLetta, CreateBlock+from letta_evals.decorators import agent_factory+from letta_evals.models import Sample++@agent_factory+async def create_inventory_agent(client: AsyncLetta, sample: Sample) -> str:+ """Create and return agent ID for this sample."""+ # Access custom arguments from the dataset+ item = sample.agent_args.get("item", {})++ # Create agent with sample-specific configuration+ agent = await client.agents.create(+ name="inventory-assistant",+ memory_blocks=[+ CreateBlock(+ label="item_context",+ value=f"Item: {item.get('name', 'Unknown')}"+ )+ ],+ agent_type="letta_v1_agent",+ model="openai/gpt-4.1-mini",+ embedding="openai/text-embedding-3-small",+ )++ return agent.id+```++**Key features:**+- Creates a fresh agent for each sample+- Can customize agents using `sample.agent_args` from the dataset+- Allows testing agent creation logic itself+- Useful when you don't have pre-saved agent files++**When to use:**+- Testing agent creation workflows+- Dynamic per-sample agent configuration+- Agents that need sample-specific memory or tools+- Programmatic agent testing++## Connection Configuration++### base_url++Letta server URL:++```yaml+target:+ base_url: http://localhost:8283 # Local Letta server+ # or+ base_url: https://api.letta.com # Letta Cloud+```++Default: `http://localhost:8283`++### api_key++API key for authentication (required for Letta Cloud):++```yaml+target:+ api_key: your-api-key-here # Required for Letta Cloud+```++Or set via environment variable:+```bash+export LETTA_API_KEY=your-api-key-here+```++### project_id++Letta project ID (for Letta Cloud):++```yaml+target:+ project_id: proj_abc123 # Letta Cloud project+```++Or set via environment variable:+```bash+export LETTA_PROJECT_ID=proj_abc123+```++### timeout++Request timeout in seconds:++```yaml+target:+ timeout: 300.0 # Request timeout (5 minutes)+```++Default: 300 seconds++## Multi-Model Evaluation++Test the same agent across different models:++### model_configs++List of model configuration names from JSON files:++```yaml+target:+ kind: agent+ agent_file: agent.af+ model_configs: [gpt-4o-mini, claude-3-5-sonnet] # Test with both models+```++The evaluation will run once for each model config. Model configs are JSON files in `letta_evals/llm_model_configs/`.++### model_handles++List of model handles (cloud-compatible identifiers):++```yaml+target:+ kind: agent+ agent_file: agent.af+ model_handles: ["openai/gpt-4o-mini", "anthropic/claude-3-5-sonnet"] # Cloud model identifiers+```++Use this for Letta Cloud deployments.++<Warning>+**Note**: You cannot specify both `model_configs` and `model_handles`.+</Warning>++## Complete Examples++### Local Development++```yaml+target:+ kind: agent+ agent_file: ./agents/my_agent.af # Pre-configured agent+ base_url: http://localhost:8283 # Local server+```++### Letta Cloud++```yaml+target:+ kind: agent+ agent_id: agent-cloud-123 # Existing cloud agent+ base_url: https://api.letta.com # Letta Cloud+ api_key: ${LETTA_API_KEY} # From environment variable+ project_id: proj_abc # Your project ID+```++### Multi-Model Testing++```yaml+target:+ kind: agent+ agent_file: agent.af # Same agent configuration+ base_url: http://localhost:8283 # Local server+ model_configs: [gpt-4o-mini, gpt-4o, claude-3-5-sonnet] # Test 3 models+```++Results will include per-model metrics:+```+Model: gpt-4o-mini - Avg: 0.85, Pass: 85.0%+Model: gpt-4o - Avg: 0.92, Pass: 92.0%+Model: claude-3-5-sonnet - Avg: 0.88, Pass: 88.0%+```++### Programmatic Agent Creation++```yaml+target:+ kind: agent+ agent_script: setup.py:CustomAgentFactory # Programmatic creation+ base_url: http://localhost:8283 # Local server+```++## Environment Variable Precedence++Configuration values are resolved in this order (highest priority first):++1. CLI arguments (`--api-key`, `--base-url`, `--project-id`)+2. Suite YAML configuration+3. Environment variables (`LETTA_API_KEY`, `LETTA_BASE_URL`, `LETTA_PROJECT_ID`)++## Agent Lifecycle and Testing Behavior++The way your agent is specified fundamentally changes how the evaluation runs:++### With agent_file or agent_script: Independent Testing++**Agent lifecycle:**+1. A fresh agent instance is created for each sample+2. Agent processes the sample input(s)+3. Agent remains on the server after the sample completes++**Testing behavior:** Each sample is an independent, isolated test. Agent state (memory, message history) does not carry over between samples. This enables parallel execution and ensures reproducible results.++**Use cases:**+- Testing how the agent responds to various independent inputs+- Ensuring consistent behavior across different scenarios+- Regression testing where each case should be isolated+- Evaluating agent responses without prior context++<Note>+**Example:** If you have 10 test cases, 10 separate agent instances will be created (one per test case), and they can run in parallel.+</Note>++### With agent_id: Sequential Script Testing++**Agent lifecycle:**+1. The same agent instance is used for all samples+2. Agent processes each sample in sequence+3. Agent state persists throughout the entire evaluation++**Testing behavior:** The dataset becomes a conversation script where each sample builds on previous ones. Agent memory and message history accumulate, and earlier interactions affect later responses. Samples must execute sequentially.++**Use cases:**+- Testing multi-turn conversations with context+- Evaluating how agent memory evolves over time+- Simulating a single user session with multiple interactions+- Testing scenarios where context should accumulate++<Note>+**Example:** If you have 10 test cases, they all run against the same agent instance in order, with state carrying over between each test.+</Note>++### Critical Differences++| Aspect | agent_file / agent_script | agent_id |+|--------|---------------------------|----------|+| **Agent instances** | New agent per sample | Same agent for all samples |+| **State isolation** | Fully isolated | State carries over |+| **Execution** | Can run in parallel | Must run sequentially |+| **Memory** | Fresh for each sample | Accumulates across samples |+| **Use case** | Independent test cases | Conversation scripts |+| **Reproducibility** | Highly reproducible | Depends on execution order |++<Tip>+**Best practice:** Use `agent_file` or `agent_script` for most evaluations to ensure reproducible, isolated tests. Use `agent_id` only when you specifically need to test how agent state evolves across multiple interactions.+</Tip>++## Validation++The runner validates:+- Exactly one of `agent_file`, `agent_id`, or `agent_script` is specified+- Agent files have `.af` extension+- Agent script paths are valid++## Next Steps++- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete target configuration options+- [Datasets](/evals/core-concepts/datasets) - Using agent_args for sample-specific configuration+- [Getting Started](/evals/get-started/getting-started) - Complete tutorial with target examplesdiff --git a/fern/pages/evals/configuration/suite-yaml.mdx b/fern/pages/evals/configuration/suite-yaml.mdxnew file mode 100644index 000000000..440e44cfb--- /dev/null+++ b/fern/pages/evals/configuration/suite-yaml.mdx@@ -0,0 +1,427 @@+# Suite YAML Reference++Complete reference for suite configuration files.++A **suite** is a YAML file that defines an evaluation: what agent to test, what dataset to use, how to grade responses, and what criteria determine pass/fail. This is your evaluation specification.++<Note>+**Quick overview:**+- **name**: Identifier for your evaluation+- **dataset**: JSONL file with test cases+- **target**: Which agent to evaluate (via file, ID, or script)+- **graders**: How to score responses (tool or rubric graders)+- **gate**: Pass/fail criteria+</Note>++See [Getting Started](/evals/get-started/getting-started) for a tutorial, or [Core Concepts](/evals/core-concepts/suites) for conceptual overview.++## File Structure++```yaml+name: string (required)+description: string (optional)+dataset: path (required)+max_samples: integer (optional)+sample_tags: array (optional)+num_runs: integer (optional)+setup_script: string (optional)++target: object (required)+ kind: "agent"+ base_url: string+ api_key: string+ timeout: float+ project_id: string+ agent_id: string (one of: agent_id, agent_file, agent_script)+ agent_file: path+ agent_script: string+ model_configs: array+ model_handles: array++graders: object (required)+ <metric_key>: object+ kind: "tool" | "rubric"+ display_name: string+ extractor: string+ extractor_config: object+ # Tool grader fields+ function: string+ # Rubric grader fields (LLM API)+ prompt: string+ prompt_path: path+ model: string+ temperature: float+ provider: string+ max_retries: integer+ timeout: float+ rubric_vars: array+ # Rubric grader fields (agent-as-judge)+ agent_file: path+ judge_tool_name: string++gate: object (required)+ metric_key: string+ metric: "avg_score" | "accuracy"+ op: "gte" | "gt" | "lte" | "lt" | "eq"+ value: float+ pass_op: "gte" | "gt" | "lte" | "lt" | "eq"+ pass_value: float+```++## Top-Level Fields++### name (required)++Suite name, used in output and results.++**Type**: string++```yaml+name: question-answering-eval+```++### description (optional)++Human-readable description of what the suite tests.++**Type**: string++```yaml+description: Tests agent's ability to answer factual questions accurately+```++### dataset (required)++Path to JSONL dataset file. Relative paths are resolved from the suite YAML location.++**Type**: path (string)++```yaml+dataset: ./datasets/qa.jsonl+dataset: /absolute/path/to/dataset.jsonl+```++### max_samples (optional)++Limit the number of samples to evaluate. Useful for quick tests.++**Type**: integer | **Default**: All samples++```yaml+max_samples: 10 # Only evaluate first 10 samples+```++### sample_tags (optional)++Filter samples by tags. Only samples with ALL specified tags are evaluated.++**Type**: array of strings++```yaml+sample_tags: [math, easy] # Only samples tagged with both+```++### num_runs (optional)++Number of times to run the evaluation suite.++**Type**: integer | **Default**: 1++```yaml+num_runs: 5 # Run the evaluation 5 times+```++### setup_script (optional)++Path to Python script with setup function.++**Type**: string (format: `path/to/script.py:function_name`)++```yaml+setup_script: setup.py:prepare_environment+```++## target (required)++Configuration for the agent being evaluated.++### kind (required)++Type of target. Currently only `"agent"` is supported.++```yaml+target:+ kind: agent+```++### base_url (optional)++Letta server URL. **Default**: `http://localhost:8283`++```yaml+target:+ base_url: http://localhost:8283+ # or+ base_url: https://api.letta.com+```++### api_key (optional)++API key for Letta authentication. Can also be set via `LETTA_API_KEY` environment variable.++```yaml+target:+ api_key: your-api-key-here+```++### timeout (optional)++Request timeout in seconds. **Default**: 300.0++```yaml+target:+ timeout: 600.0 # 10 minutes+```++### Agent Source (required, pick one)++Exactly one of these must be specified:++#### agent_id++ID of existing agent on the server.++```yaml+target:+ agent_id: agent-123-abc+```++#### agent_file++Path to `.af` agent file.++```yaml+target:+ agent_file: ./agents/my_agent.af+```++#### agent_script++Path to Python script with agent factory.++```yaml+target:+ agent_script: factory.py:MyAgentFactory+```++See [Targets](/evals/core-concepts/targets) for details on agent sources.++### model_configs (optional)++List of model configuration names to test. Cannot be used with `model_handles`.++```yaml+target:+ model_configs: [gpt-4o-mini, claude-3-5-sonnet]+```++### model_handles (optional)++List of model handles for cloud deployments. Cannot be used with `model_configs`.++```yaml+target:+ model_handles: ["openai/gpt-4o-mini", "anthropic/claude-3-5-sonnet"]+```++## graders (required)++One or more graders, each with a unique key.++### kind (required)++Grader type: `"tool"` or `"rubric"`.++```yaml+graders:+ my_metric:+ kind: tool+```++### extractor (required)++Name of the extractor to use.++```yaml+graders:+ my_metric:+ extractor: last_assistant+```++### Tool Grader Fields++#### function (required for tool graders)++Name of the grading function.++```yaml+graders:+ accuracy:+ kind: tool+ function: exact_match+```++### Rubric Grader Fields++#### prompt or prompt_path (required)++Inline rubric prompt or path to rubric file.++```yaml+graders:+ quality:+ kind: rubric+ prompt: |+ Evaluate response quality from 0.0 to 1.0.+```++#### model (optional)++LLM model for judging. **Default**: `gpt-4o-mini`++```yaml+graders:+ quality:+ kind: rubric+ model: gpt-4o+```++#### temperature (optional)++Temperature for LLM generation. **Default**: 0.0++```yaml+graders:+ quality:+ kind: rubric+ temperature: 0.0+```++#### agent_file (agent-as-judge)++Path to `.af` agent file to use as judge.++```yaml+graders:+ agent_judge:+ kind: rubric+ agent_file: judge.af+ prompt_path: rubric.txt+```++## gate (required)++Pass/fail criteria for the evaluation.++### metric_key (optional)++Which grader to evaluate. If only one grader, this can be omitted.++```yaml+gate:+ metric_key: accuracy+```++### metric (optional)++Which aggregate to compare: `avg_score` or `accuracy`. **Default**: `avg_score`++```yaml+gate:+ metric: avg_score+```++### op (required)++Comparison operator: `gte`, `gt`, `lte`, `lt`, `eq`++```yaml+gate:+ op: gte # Greater than or equal+```++### value (required)++Threshold value for comparison (0.0 to 1.0).++```yaml+gate:+ value: 0.8 # Require >= 0.8+```++## Complete Examples++### Minimal Suite++```yaml+name: basic-eval+dataset: dataset.jsonl++target:+ kind: agent+ agent_file: agent.af++graders:+ accuracy:+ kind: tool+ function: exact_match+ extractor: last_assistant++gate:+ op: gte+ value: 0.8+```++### Multi-Metric Suite++```yaml+name: comprehensive-eval+description: Tests accuracy and quality+dataset: test_data.jsonl++target:+ kind: agent+ agent_file: agent.af++graders:+ accuracy:+ kind: tool+ function: contains+ extractor: last_assistant++ quality:+ kind: rubric+ prompt_path: rubrics/quality.txt+ model: gpt-4o-mini+ extractor: last_assistant++gate:+ metric_key: accuracy+ op: gte+ value: 0.85+```++## Validation++Validate your suite before running:++```bash+letta-evals validate suite.yaml+```++## Next Steps++- [Targets](/evals/core-concepts/targets) - Understanding agent sources and configuration+- [Graders](/evals/core-concepts/graders) - Tool graders vs rubric graders+- [Extractors](/evals/core-concepts/extractors) - What to extract from agent responses+- [Gates](/evals/core-concepts/gates) - Setting pass/fail criteriadiff --git a/fern/pages/evals/extractors/builtin.mdx b/fern/pages/evals/extractors/builtin.mdxnew file mode 100644index 000000000..b8be5d486--- /dev/null+++ b/fern/pages/evals/extractors/builtin.mdx@@ -0,0 +1,96 @@+# Built-in Extractors++Letta Evals provides a set of built-in extractors that cover the most common extraction needs.++<Note>+**What are extractors?** Extractors determine what part of an agent's response gets evaluated. They take the full conversation trajectory and extract just the piece you want to grade.+</Note>++## Common Extractors++### last_assistant++Extracts the last assistant message content.++```yaml+extractor: last_assistant # Most common - gets final response+```++### first_assistant++Extracts the first assistant message content.++```yaml+extractor: first_assistant+```++### all_assistant++Concatenates all assistant messages with a separator.++```yaml+extractor: all_assistant+extractor_config:+ separator: "\n\n" # Join messages with double newline+```++### pattern++Extracts content matching a regex pattern.++```yaml+extractor: pattern+extractor_config:+ pattern: 'Result: (\d+)' # Regex pattern to match+ group: 1 # Extract capture group 1+```++### tool_arguments++Extracts arguments from a specific tool call.++```yaml+extractor: tool_arguments+extractor_config:+ tool_name: search # Which tool to extract from+```++### tool_output++Extracts the return value from a specific tool call.++```yaml+extractor: tool_output+extractor_config:+ tool_name: search+```++### memory_block++Extracts content from a specific memory block.++```yaml+extractor: memory_block+extractor_config:+ block_label: human # Which memory block to extract+```++<Warning>+**Important**: This extractor requires the agent's final state, which adds overhead.+</Warning>++### after_marker++Extracts content after a specific marker string.++```yaml+extractor: after_marker+extractor_config:+ marker: "ANSWER:"+ include_marker: false+```++## Next Steps++- [Custom Extractors](/evals/extractors/custom-extractors) - Write your own extractors+- [Extractors Concept](/evals/core-concepts/extractors) - Understanding extractorsdiff --git a/fern/pages/evals/extractors/custom.mdx b/fern/pages/evals/extractors/custom.mdxnew file mode 100644index 000000000..1a16b4755--- /dev/null+++ b/fern/pages/evals/extractors/custom.mdx@@ -0,0 +1,55 @@+# Custom Extractors++Create your own extractors to pull exactly what you need from agent trajectories.++<Note>+While built-in extractors cover common cases, custom extractors let you implement specialized extraction logic for your specific use case.+</Note>++## Why Custom Extractors?++Use custom extractors when you need to:+- **Extract structured data**: Parse JSON fields from agent responses+- **Filter specific patterns**: Extract code blocks, URLs, or formatted content+- **Combine data sources**: Merge information from multiple messages or memory blocks+- **Count occurrences**: Track how many times something happened+- **Complex logic**: Implement domain-specific extraction++## Basic Structure++```python+from letta_evals.decorators import extractor+from letta_client import LettaMessageUnion+from typing import List++@extractor+def my_extractor(trajectory: List[List[LettaMessageUnion]], config: dict) -> str:+ """Extract custom content from trajectory."""+ # Your extraction logic here+ return extracted_text+```++## Example: Extract Memory Insert++```python+from letta_evals.decorators import extractor++@extractor+def memory_insert_args(trajectory, config):+ """Extract arguments from memory_insert tool calls."""+ for turn in trajectory:+ for message in turn:+ if hasattr(message, 'tool_call') and message.tool_call:+ if message.tool_call.name == "memory_insert":+ return str(message.tool_call.arguments)+ return ""+```++## Registration++Custom extractors are automatically registered when you import them in your suite's setup script or custom evaluators file.++## Next Steps++- [Built-in Extractors](/evals/extractors/built-in-extractors) - Available extractors+- [Extractors Concept](/evals/core-concepts/extractors) - Understanding extractorsdiff --git a/fern/pages/evals/getting-started.mdx b/fern/pages/evals/getting-started.mdxnew file mode 100644index 000000000..728bac79a--- /dev/null+++ b/fern/pages/evals/getting-started.mdx@@ -0,0 +1,264 @@+# Getting Started++Run your first Letta agent evaluation in 5 minutes.++## Prerequisites++- Python 3.11 or higher+- A running Letta server (local or Letta Cloud)+- A Letta agent to test, either in agent file format or by ID (see [Targets](/evals/core-concepts/targets) for more details)++## Installation++```bash+pip install letta-evals+```++Or with uv:++```bash+uv pip install letta-evals+```++## Getting an Agent to Test++Export an existing agent to a file using the Letta SDK:++```python+from letta_client import Letta+import os++client = Letta(+ base_url="http://localhost:8283", # or https://api.letta.com for Letta Cloud+ token=os.getenv("LETTA_API_KEY") # required for Letta Cloud+)++# Export an agent to a file+agent_file = client.agents.export_file(agent_id="agent-123")++# Save to disk+with open("my_agent.af", "w") as f:+ f.write(agent_file)+```++Or export via the Agent Development Environment (ADE) by selecting "Export Agent".++Then reference it in your suite:++```yaml+target:+ kind: agent+ agent_file: my_agent.af+```++<Note>+**Other options:** You can also use existing agents by ID or programmatically generate agents. See [Targets](/evals/core-concepts/targets) for all agent configuration options.+</Note>++## Quick Start++Let's create your first evaluation in 3 steps:++### 1. Create a Test Dataset++Create a file named `dataset.jsonl`:++```jsonl+{"input": "What's the capital of France?", "ground_truth": "Paris"}+{"input": "Calculate 2+2", "ground_truth": "4"}+{"input": "What color is the sky?", "ground_truth": "blue"}+```++Each line is a JSON object with:+- `input`: The prompt to send to your agent+- `ground_truth`: The expected answer (used for grading)++<Note>+`ground_truth` is optional for some graders (like rubric graders), but required for tool graders like `contains` and `exact_match`.+</Note>++Read more about [Datasets](/evals/core-concepts/datasets) for details on how to create your dataset.++### 2. Create a Suite Configuration++Create a file named `suite.yaml`:++```yaml+name: my-first-eval+dataset: dataset.jsonl++target:+ kind: agent+ agent_file: my_agent.af # Path to your agent file+ base_url: http://localhost:8283 # Your Letta server++graders:+ quality:+ kind: tool+ function: contains # Check if response contains the ground truth+ extractor: last_assistant # Use the last assistant message++gate:+ metric_key: quality+ op: gte+ value: 0.75 # Require 75% pass rate+```++The suite configuration defines:+- The [dataset](/evals/core-concepts/datasets) to use+- The [agent](/evals/core-concepts/targets) to test+- The [graders](/evals/core-concepts/graders) to use+- The [gate](/evals/core-concepts/gates) criteria++Read more about [Suites](/evals/core-concepts/suites) for details on how to configure your evaluation.++### 3. Run the Evaluation++Run your evaluation with the following command:++```bash+letta-evals run suite.yaml+```++You'll see real-time progress as your evaluation runs:++```+Running evaluation: my-first-eval+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3/3 100%+✓ PASSED (2.25/3.00 avg, 75.0% pass rate)+```++Read more about [CLI Commands](/evals/cli-reference/commands) for details about the available commands and options.++## Understanding the Results++The core evaluation flow is:++**Dataset → Target (Agent) → Extractor → Grader → Gate → Result**++The evaluation runner:+1. Loads your dataset+2. Sends each input to your agent (Target)+3. Extracts the relevant information (using the Extractor)+4. Grades the response (using the Grader function)+5. Computes aggregate metrics+6. Checks if metrics pass the Gate criteria++The output shows:+- **Average score**: Mean score across all samples+- **Pass rate**: Percentage of samples that passed+- **Gate status**: Whether the evaluation passed or failed overall++## Next Steps++Now that you've run your first evaluation, explore more advanced features:++- [Core Concepts](/evals/core-concepts/concepts-overview) - Understand suites, datasets, graders, and extractors+- [Grader Types](/evals/core-concepts/graders) - Learn about tool graders vs rubric graders+- [Multi-Metric Evaluation](/evals/graders/multi-metric-grading) - Test multiple aspects simultaneously+- [Custom Graders](/evals/advanced/custom-graders) - Write custom grading functions+- [Multi-Turn Conversations](/evals/advanced/multi-turn-conversations) - Test conversational memory++## Common Use Cases++### Strict Answer Checking++Use exact matching for cases where the answer must be precisely correct:++```yaml+graders:+ accuracy:+ kind: tool+ function: exact_match+ extractor: last_assistant+```++### Subjective Quality Evaluation++Use an LLM judge to evaluate subjective qualities like helpfulness or tone:++```yaml+graders:+ quality:+ kind: rubric+ prompt_path: rubric.txt+ model: gpt-4o-mini+ extractor: last_assistant+```++Then create `rubric.txt`:+```+Rate the helpfulness and accuracy of the response.+- Score 1.0 if helpful and accurate+- Score 0.5 if partially helpful+- Score 0.0 if unhelpful or wrong+```++### Testing Tool Calls++Verify that your agent calls specific tools with expected arguments:++```yaml+graders:+ tool_check:+ kind: tool+ function: contains+ extractor: tool_arguments+ extractor_config:+ tool_name: search+```++### Testing Memory Persistence++Check if the agent correctly updates its memory blocks:++```yaml+graders:+ memory_check:+ kind: tool+ function: contains+ extractor: memory_block+ extractor_config:+ block_label: human+```++## Troubleshooting++<Warning>+**"Agent file not found"**++Make sure your `agent_file` path is correct. Paths are relative to the suite YAML file location. Use absolute paths if needed:++```yaml+target:+ agent_file: /absolute/path/to/my_agent.af+```+</Warning>++<Warning>+**"Connection refused"**++Your Letta server isn't running or isn't accessible. Start it with:++```bash+letta server+```++By default, it runs at `http://localhost:8283`.+</Warning>++<Warning>+**"No ground_truth provided"**++Tool graders like `exact_match` and `contains` require `ground_truth` in your dataset. Either:+- Add `ground_truth` to your samples, or+- Use a rubric grader which doesn't require ground truth+</Warning>++<Tip>+**Agent didn't respond as expected**++Try testing your agent manually first using the Letta SDK or Agent Development Environment (ADE) to see how it behaves before running evaluations. See the [Letta documentation](https://docs.letta.com) for more information.+</Tip>++For more help, see the [Troubleshooting Guide](/evals/troubleshooting/common-issues).diff --git a/fern/pages/evals/graders/multi-metric.mdx b/fern/pages/evals/graders/multi-metric.mdxnew file mode 100644index 000000000..d414f7ae9--- /dev/null+++ b/fern/pages/evals/graders/multi-metric.mdx@@ -0,0 +1,58 @@+# Multi-Metric Evaluation++Evaluate multiple aspects of agent performance simultaneously in a single evaluation suite.++<Note>+Multi-metric evaluation allows you to define multiple graders, each measuring a different dimension of your agent's behavior.+</Note>++## Why Multiple Metrics?++Agents are complex systems. You might want to evaluate:+- **Correctness**: Does the answer match the expected output?+- **Quality**: Is the explanation clear and complete?+- **Tool usage**: Does the agent call the right tools with correct arguments?+- **Memory**: Does the agent correctly update its memory blocks?+- **Format**: Does the output follow required formatting rules?++## Configuration++```yaml+graders:+ accuracy: # Check if answer is correct+ kind: tool+ function: exact_match+ extractor: last_assistant++ completeness: # LLM judges response quality+ kind: rubric+ prompt_path: rubrics/completeness.txt+ model: gpt-4o-mini+ extractor: last_assistant++ tool_usage: # Verify correct tool was called+ kind: tool+ function: contains+ extractor: tool_arguments+ extractor_config:+ tool_name: search+```++## Gating on One Metric++The gate can check any of these metrics:++```yaml+gate:+ metric_key: accuracy # Gate on accuracy (others still computed)+ op: gte+ value: 0.9+```++Results will include scores for all graders, even if you only gate on one.++## Next Steps++- [Tool Graders](/evals/graders/tool-graders) - Deterministic evaluation+- [Rubric Graders](/evals/graders/rubric-graders) - LLM-as-judge evaluation+- [Gates](/evals/core-concepts/gates) - Setting pass/fail criteriadiff --git a/fern/pages/evals/graders/rubric-graders.mdx b/fern/pages/evals/graders/rubric-graders.mdxnew file mode 100644index 000000000..1b73ba7a0--- /dev/null+++ b/fern/pages/evals/graders/rubric-graders.mdx@@ -0,0 +1,82 @@+# Rubric Graders++Rubric graders use language models to evaluate submissions based on custom criteria. They're ideal for subjective, nuanced evaluation.++<Note>+Rubric graders work by providing the LLM with a prompt that describes the evaluation criteria, then the language model generates a structured JSON response with a score and rationale.+</Note>++## Basic Configuration++```yaml+graders:+ quality:+ kind: rubric+ prompt_path: quality_rubric.txt # Evaluation criteria+ model: gpt-4o-mini # Judge model+ temperature: 0.0 # Deterministic+ extractor: last_assistant # What to evaluate+```++## Rubric Prompt Format++Your rubric file should describe the evaluation criteria. Use placeholders:++- `{input}`: The original input from the dataset+- `{submission}`: The extracted agent response+- `{ground_truth}`: Ground truth from dataset (if available)++Example `quality_rubric.txt`:+```+Evaluate the response for:+1. Accuracy: Does it correctly answer the question?+2. Completeness: Is the answer thorough?+3. Clarity: Is it well-explained?++Input: {input}+Expected: {ground_truth}+Response: {submission}++Score from 0.0 to 1.0 where:++- 1.0: Perfect response+- 0.75: Good with minor issues+- 0.5: Acceptable but incomplete+- 0.25: Poor quality+- 0.0: Completely wrong+```++## Model Configuration++```yaml+graders:+ quality:+ kind: rubric+ prompt_path: rubric.txt+ model: gpt-4o-mini # Judge model+ temperature: 0.0 # Deterministic+ provider: openai # LLM provider+ max_retries: 5 # API retry attempts+ timeout: 120.0 # Request timeout+```++## Agent-as-Judge++Use a Letta agent as the judge instead of a direct LLM API call:++```yaml+graders:+ agent_judge:+ kind: rubric+ agent_file: judge.af # Judge agent with submit_grade tool+ prompt_path: rubric.txt # Evaluation criteria+ extractor: last_assistant+```++**Requirements**: The judge agent must have a tool with signature `submit_grade(score: float, rationale: str)`.++## Next Steps++- [Tool Graders](/evals/graders/tool-graders) - Deterministic grading functions+- [Multi-Metric](/evals/graders/multi-metric-grading) - Combine multiple graders+- [Custom Graders](/evals/advanced/custom-graders) - Write your own grading logicdiff --git a/fern/pages/evals/graders/tool-graders.mdx b/fern/pages/evals/graders/tool-graders.mdxnew file mode 100644index 000000000..0edb327f5--- /dev/null+++ b/fern/pages/evals/graders/tool-graders.mdx@@ -0,0 +1,85 @@+# Tool Graders++Tool graders use Python functions to programmatically evaluate submissions. They're ideal for deterministic, rule-based evaluation.++## Overview++Tool graders:+- Execute Python functions that take `(sample, submission)` and return a `GradeResult`+- Are fast and deterministic+- Don't require external API calls+- Can implement any custom logic++## Configuration++```yaml+graders:+ my_metric:+ kind: tool+ function: exact_match # Function name+ extractor: last_assistant # What to extract from trajectory+```++## Built-in Functions++### exact_match++Checks if submission exactly matches ground truth (case-sensitive, whitespace-trimmed).++```yaml+graders:+ accuracy:+ kind: tool+ function: exact_match+ extractor: last_assistant+```++**Requires**: `ground_truth` in dataset | **Score**: 1.0 if exact match, 0.0 otherwise++### contains++Checks if submission contains ground truth (case-insensitive).++```yaml+graders:+ contains_answer:+ kind: tool+ function: contains+ extractor: last_assistant+```++**Requires**: `ground_truth` in dataset | **Score**: 1.0 if found, 0.0 otherwise++### regex_match++Checks if submission matches a regex pattern in ground truth.++```yaml+graders:+ pattern:+ kind: tool+ function: regex_match+ extractor: last_assistant+```++**Score**: 1.0 if pattern matches, 0.0 otherwise++### ascii_printable_only++Validates that all characters are printable ASCII.++```yaml+graders:+ ascii_check:+ kind: tool+ function: ascii_printable_only+ extractor: last_assistant+```++**Score**: 1.0 if all characters are printable ASCII, 0.0 otherwise++## Next Steps++- [Rubric Graders](/evals/graders/rubric-graders) - LLM-as-judge evaluation+- [Custom Graders](/evals/advanced/custom-graders) - Write your own grading functions+- [Multi-Metric](/evals/graders/multi-metric-grading) - Combine multiple gradersdiff --git a/fern/pages/evals/overview.mdx b/fern/pages/evals/overview.mdxnew file mode 100644index 000000000..7856bc458--- /dev/null+++ b/fern/pages/evals/overview.mdx@@ -0,0 +1,47 @@+# Letta Evals++**Systematic testing for stateful AI agents.** Validate changes, prevent regressions, and ship with confidence.++Test agent memory, tool usage, multi-turn conversations, and state evolution with automated grading and pass/fail gates.++<Note>+**Ready to start?** Jump to [Getting Started](/evals/get-started/getting-started) or learn the [Core Concepts](/evals/core-concepts/concepts-overview) first.+</Note>++## Core Concepts++Understand the building blocks of evaluations:++- [Suites](/evals/core-concepts/suites) - Configure your evaluation+- [Datasets](/evals/core-concepts/datasets) - Define test cases+- [Targets](/evals/core-concepts/targets) - Specify the agent to test+- [Graders](/evals/core-concepts/graders) - Score agent outputs+- [Extractors](/evals/core-concepts/extractors) - Extract content from responses+- [Gates](/evals/core-concepts/gates) - Set pass/fail criteria++### Grading & Extraction++Choose how to score your agents:++- [Tool Graders](/evals/graders/tool-graders) - Fast, deterministic grading with Python functions+- [Rubric Graders](/evals/graders/rubric-graders) - Flexible LLM-as-judge evaluation+- [Built-in Extractors](/evals/extractors/built-in-extractors) - Pre-built content extractors+- [Multi-Metric Grading](/evals/graders/multi-metric-grading) - Evaluate multiple dimensions++### Advanced++- [Custom Graders](/evals/advanced/custom-graders) - Write your own grading logic+- [Custom Extractors](/evals/extractors/custom-extractors) - Build custom extractors+- [Multi-Turn Conversations](/evals/advanced/multi-turn-conversations) - Test memory and state+- [Suite YAML Reference](/evals/configuration/suite-yaml-reference) - Complete configuration schema++### Reference++- [CLI Commands](/evals/cli-reference/commands) - Command-line interface+- [Understanding Results](/evals/results-metrics/understanding-results) - Interpret metrics+- [Troubleshooting](/evals/troubleshooting/common-issues) - Common issues and solutions++## Resources++- **[GitHub Repository](https://github.com/letta-ai/letta-evals)** - Source code, issues, and contributions+- **[PyPI Package](https://pypi.org/project/letta-evals/)** - Install with `pip install letta-evals`diff --git a/fern/pages/evals/results/overview.mdx b/fern/pages/evals/results/overview.mdxnew file mode 100644index 000000000..2487c034d--- /dev/null+++ b/fern/pages/evals/results/overview.mdx@@ -0,0 +1,484 @@+# Understanding Results++This guide explains how to interpret evaluation results.++## Result Structure++An evaluation produces three types of output:++1. **Console output**: Real-time progress and summary+2. **Summary JSON**: Aggregate metrics and configuration+3. **Results JSONL**: Per-sample detailed results++## Console Output++### Progress Display++```+Running evaluation: my-eval-suite+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3/3 100%++Results:+ Total samples: 3+ Attempted: 3+ Avg score: 0.83 (attempted: 0.83)+ Passed: 2 (66.7%)++Gate (quality >= 0.75): PASSED+```++### Quiet Mode++```bash+letta-evals run suite.yaml --quiet+```++Output:+```+✓ PASSED+```++or++```+✗ FAILED+```++## JSON Output++### Saving Results++```bash+letta-evals run suite.yaml --output results/+```++Creates three files:++#### header.json++Evaluation metadata:++```json+{+ "suite_name": "my-eval-suite",+ "timestamp": "2025-01-15T10:30:00Z",+ "version": "0.3.0"+}+```++#### summary.json++Complete evaluation summary:++```json+{+ "suite": "my-eval-suite",+ "config": {+ "target": {...},+ "graders": {...},+ "gate": {...}+ },+ "metrics": {+ "total": 10,+ "total_attempted": 10,+ "avg_score_attempted": 0.85,+ "avg_score_total": 0.85,+ "passed_attempts": 8,+ "failed_attempts": 2,+ "by_metric": {+ "accuracy": {+ "avg_score_attempted": 0.90,+ "pass_rate": 90.0,+ "passed_attempts": 9,+ "failed_attempts": 1+ },+ "quality": {+ "avg_score_attempted": 0.80,+ "pass_rate": 70.0,+ "passed_attempts": 7,+ "failed_attempts": 3+ }+ }+ },+ "gates_passed": true+}+```++#### results.jsonl++One JSON object per line, each representing one sample:++```jsonl+{"sample": {"id": 0, "input": "What is 2+2?", "ground_truth": "4"}, "submission": "4", "grade": {"score": 1.0, "rationale": "Exact match: true"}, "trajectory": [...], "agent_id": "agent-123", "model_name": "default"}+{"sample": {"id": 1, "input": "What is 3+3?", "ground_truth": "6"}, "submission": "6", "grade": {"score": 1.0, "rationale": "Exact match: true"}, "trajectory": [...], "agent_id": "agent-124", "model_name": "default"}+```++## Metrics Explained++### total++Total number of samples in the evaluation (including errors).++### total_attempted++Number of samples that completed without errors.++If a sample fails during agent execution or grading, it's counted in `total` but not `total_attempted`.++### avg_score_attempted++Average score across samples that completed successfully.++Formula: `sum(scores) / total_attempted`++Range: 0.0 to 1.0++### avg_score_total++Average score across all samples, treating errors as 0.0.++Formula: `sum(scores) / total`++Range: 0.0 to 1.0++### passed_attempts / failed_attempts++Number of samples that passed/failed the gate's per-sample criteria.++By default:+- If gate metric is `accuracy`: sample passes if score `>= 1.0`+- If gate metric is `avg_score`: sample passes if score `>=` gate value++Can be customized with `pass_op` and `pass_value` in gate config.++### by_metric++For multi-metric evaluation, shows aggregate stats for each metric:++```json+"by_metric": {+ "accuracy": {+ "avg_score_attempted": 0.90,+ "avg_score_total": 0.85,+ "pass_rate": 90.0,+ "passed_attempts": 9,+ "failed_attempts": 1+ }+}+```++## Sample Results++Each sample result includes:++### sample+The original dataset sample:+```json+"sample": {+ "id": 0,+ "input": "What is 2+2?",+ "ground_truth": "4",+ "metadata": {...}+}+```++### submission+The extracted text that was graded:+```json+"submission": "The answer is 4"+```++### grade+The grading result:+```json+"grade": {+ "score": 1.0,+ "rationale": "Contains ground_truth: true",+ "metadata": {"model": "gpt-4o-mini", "usage": {...}}+}+```++### grades (multi-metric)+For multi-metric evaluation:+```json+"grades": {+ "accuracy": {"score": 1.0, "rationale": "Exact match"},+ "quality": {"score": 0.85, "rationale": "Good but verbose"}+}+```++### trajectory+The complete conversation history:+```json+"trajectory": [+ [+ {"role": "user", "content": "What is 2+2?"},+ {"role": "assistant", "content": "The answer is 4"}+ ]+]+```++### agent_id+The ID of the agent that generated this response:+```json+"agent_id": "agent-abc-123"+```++### model_name+The model configuration used:+```json+"model_name": "gpt-4o-mini"+```++### agent_usage+Token usage statistics (if available):+```json+"agent_usage": [+ {"completion_tokens": 10, "prompt_tokens": 50, "total_tokens": 60}+]+```++## Interpreting Scores++### Score Ranges++- **1.0**: Perfect - fully meets criteria+- **0.8-0.99**: Very good - minor issues+- **0.6-0.79**: Good - notable improvements possible+- **0.4-0.59**: Acceptable - significant issues+- **0.2-0.39**: Poor - major problems+- **0.0-0.19**: Failed - did not meet criteria++### Binary vs Continuous++**Tool graders** typically return binary scores:+- 1.0: Passed+- 0.0: Failed++**Rubric graders** return continuous scores:+- Any value from 0.0 to 1.0+- Allows for partial credit++## Multi-Model Results++When testing multiple models:++```json+"metrics": {+ "per_model": [+ {+ "model_name": "gpt-4o-mini",+ "avg_score_attempted": 0.85,+ "passed_samples": 8,+ "failed_samples": 2+ },+ {+ "model_name": "claude-3-5-sonnet",+ "avg_score_attempted": 0.90,+ "passed_samples": 9,+ "failed_samples": 1+ }+ ]+}+```++Console output:+```+Results by model:+ gpt-4o-mini - Avg: 0.85, Pass: 80.0%+ claude-3-5-sonnet - Avg: 0.90, Pass: 90.0%+```++## Multiple Runs Statistics++Run evaluations multiple times to measure consistency and get aggregate statistics.++### Configuration++Specify in YAML:+```yaml+name: my-eval-suite+dataset: dataset.jsonl+num_runs: 5 # Run 5 times+target:+ kind: agent+ agent_file: my_agent.af+graders:+ accuracy:+ kind: tool+ function: exact_match+gate:+ metric_key: accuracy+ op: gte+ value: 0.8+```++Or via CLI:+```bash+letta-evals run suite.yaml --num-runs 10 --output results/+```++### Output Structure++```+results/+├── run_1/+│ ├── header.json+│ ├── results.jsonl+│ └── summary.json+├── run_2/+│ ├── header.json+│ ├── results.jsonl+│ └── summary.json+├── ...+└── aggregate_stats.json # Statistics across all runs+```++### Aggregate Statistics File++The `aggregate_stats.json` includes statistics across all runs:++```json+{+ "num_runs": 10,+ "runs_passed": 8,+ "mean_avg_score_attempted": 0.847,+ "std_avg_score_attempted": 0.042,+ "mean_avg_score_total": 0.847,+ "std_avg_score_total": 0.042,+ "mean_scores": {+ "accuracy": 0.89,+ "quality": 0.82+ },+ "std_scores": {+ "accuracy": 0.035,+ "quality": 0.051+ },+ "individual_run_metrics": [+ {+ "avg_score_attempted": 0.85,+ "avg_score_total": 0.85,+ "pass_rate": 0.85,+ "by_metric": {+ "accuracy": {+ "avg_score_attempted": 0.90,+ "avg_score_total": 0.90,+ "pass_rate": 0.90+ }+ }+ }+ // ... metrics from runs 2-10+ ]+}+```++**Key fields**:+- `num_runs`: Total number of runs executed+- `runs_passed`: Number of runs that passed the gate+- `mean_avg_score_attempted`: Mean score across runs (only attempted samples)+- `std_avg_score_attempted`: Standard deviation (measures consistency)+- `mean_scores`: Mean for each metric (e.g., `{"accuracy": 0.89}`)+- `std_scores`: Standard deviation for each metric (e.g., `{"accuracy": 0.035}`)+- `individual_run_metrics`: Full metrics object from each individual run++### Use Cases++**Measure consistency of non-deterministic agents:**+```bash+letta-evals run suite.yaml --num-runs 20 --output results/+# Check std_avg_score_attempted in aggregate_stats.json+# Low std = consistent, high std = variable+```++**Get confidence intervals:**+```python+import json+import math++with open("results/aggregate_stats.json") as f:+ stats = json.load(f)++mean = stats["mean_avg_score_attempted"]+std = stats["std_avg_score_attempted"]+n = stats["num_runs"]++# 95% confidence interval (assuming normal distribution)+margin = 1.96 * (std / math.sqrt(n))+print(f"Score: {mean:.3f} ± {margin:.3f}")+```++**Compare metric consistency:**+```python+with open("results/aggregate_stats.json") as f:+ stats = json.load(f)++for metric_name, mean in stats["mean_scores"].items():+ std = stats["std_scores"][metric_name]+ consistency = "consistent" if std < 0.05 else "variable"+ print(f"{metric_name}: {mean:.3f} ± {std:.3f} ({consistency})")+```++## Error Handling++If a sample encounters an error:++```json+{+ "sample": {...},+ "submission": "",+ "grade": {+ "score": 0.0,+ "rationale": "Error during grading: Connection timeout",+ "metadata": {"error": "timeout", "error_type": "ConnectionError"}+ }+}+```++Errors:+- Count toward `total` but not `total_attempted`+- Get score of 0.0+- Include error details in rationale and metadata++## Analyzing Results++### Find Low Scores++```python+import json++with open("results/results.jsonl") as f:+ results = [json.loads(line) for line in f]++low_scores = [r for r in results if r["grade"]["score"] < 0.5]+print(f"Found {len(low_scores)} samples with score < 0.5")++for result in low_scores:+ print(f"Sample {result['sample']['id']}: {result['grade']['rationale']}")+```++### Compare Metrics++```python+# Load summary+with open("results/summary.json") as f:+ summary = json.load(f)++metrics = summary["metrics"]["by_metric"]+for name, stats in metrics.items():+ print(f"{name}: {stats['avg_score_attempted']:.2f} avg, {stats['pass_rate']:.1f}% pass")+```++### Extract Failures++```python+# Find samples that failed gate criteria+failures = [+ r for r in results+ if not gate_passed(r["grade"]["score"]) # Your gate logic+]+```++## Next Steps++- [Gates](/evals/core-concepts/gates) - Setting pass/fail criteria+- [CLI Commands](/evals/cli-reference/commands) - Running evaluationsdiff --git a/fern/pages/evals/troubleshooting.mdx b/fern/pages/evals/troubleshooting.mdxnew file mode 100644index 000000000..dc259c6ec--- /dev/null+++ b/fern/pages/evals/troubleshooting.mdx@@ -0,0 +1,267 @@+# Troubleshooting++Common issues and solutions when using Letta Evals.++## Installation Issues++<Warning>+**"Command not found: letta-evals"**++**Problem**: CLI not available after installation++**Solution**:+```bash+# Verify installation+pip list | grep letta-evals++# Reinstall if needed+pip install --upgrade letta-evals+```+</Warning>++<Warning>+**Import errors**++**Problem**: `ModuleNotFoundError: No module named 'letta_evals'`++**Solution**:+```bash+# Ensure you're in the right environment+which python++# Install in correct environment+source .venv/bin/activate+pip install letta-evals+```+</Warning>++## Configuration Issues++<Warning>+**"Agent file not found"**++**Problem**: `FileNotFoundError: agent.af`++**Solution**:+- Check the path is correct relative to the suite YAML+- Use absolute paths if needed+- Verify file exists: `ls -la path/to/agent.af`++```yaml+# Correct relative path+target:+ agent_file: ./agents/my_agent.af+```+</Warning>++<Warning>+**"Dataset not found"**++**Problem**: Cannot load dataset file++**Solution**:+- Verify dataset path in YAML+- Check file exists: `ls -la dataset.jsonl`+- Ensure proper JSONL format (one JSON object per line)++```bash+# Validate JSONL format+cat dataset.jsonl | jq .+```+</Warning>++<Warning>+**"Validation failed: unknown function"**++**Problem**: Grader function not found++**Solution**:+```bash+# List available graders+letta-evals list-graders++# Check spelling in suite.yaml+graders:+ my_metric:+ function: exact_match # Correct+```+</Warning>++## Connection Issues++<Warning>+**"Connection refused"**++**Problem**: Cannot connect to Letta server++**Solution**:+```bash+# Verify server is running+curl http://localhost:8283/v1/health++# Check base_url in suite.yaml+target:+ base_url: http://localhost:8283+```+</Warning>++<Warning>+**"Unauthorized" or "Invalid API key"**++**Problem**: Authentication failed++**Solution**:+```bash+# Set API key+export LETTA_API_KEY=your-key-here++# Verify key is correct+echo $LETTA_API_KEY+```+</Warning>++## Runtime Issues++<Warning>+**"No ground_truth provided"**++**Problem**: Grader requires ground truth but sample doesn't have it++**Solution**:+- Add ground_truth to dataset samples:+```jsonl+{"input": "What is 2+2?", "ground_truth": "4"}+```++- Or use a grader that doesn't require ground truth:+```yaml+graders:+ quality:+ kind: rubric # Doesn't require ground_truth+ prompt_path: rubric.txt+```+</Warning>++## Performance Issues++<Tip>+**Evaluation is very slow**++**Solutions**:++1. Increase concurrency:+```bash+letta-evals run suite.yaml --max-concurrent 20+```++2. Reduce samples for testing:+```yaml+max_samples: 10 # Test with small subset first+```++3. Use tool graders instead of rubric graders:+```yaml+graders:+ accuracy:+ kind: tool # Much faster than rubric+ function: exact_match+```+</Tip>++<Tip>+**High API costs**++**Solutions**:++1. Use cheaper models:+```yaml+graders:+ quality:+ model: gpt-4o-mini # Cheaper than gpt-4o+```++2. Test with small sample first:+```yaml+max_samples: 5 # Verify before running full suite+```+</Tip>++## Results Issues++<Warning>+**"All scores are 0.0"**++**Solutions**:++1. Verify extractor is getting content+2. Check grader logic+3. Test agent manually first+</Warning>++<Warning>+**"Gates failed but scores look good"**++**Solution**:+- Check gate configuration:+```yaml+gate:+ metric_key: accuracy # Correct metric?+ metric: avg_score # Or accuracy?+ op: gte # Correct operator?+ value: 0.8 # Correct threshold?+```+</Warning>++## Debug Tips++### Enable verbose output++Run without `--quiet` to see detailed progress:+```bash+letta-evals run suite.yaml+```++### Examine output files++```bash+letta-evals run suite.yaml --output debug/++# Check summary+cat debug/summary.json | jq .++# Check individual results+cat debug/results.jsonl | jq .+```++### Validate configuration++```bash+letta-evals validate suite.yaml+```++### Check component availability++```bash+letta-evals list-graders+letta-evals list-extractors+```++## Getting Help++If you're still stuck:++1. Check the [Getting Started guide](/evals/get-started/getting-started)+2. Review the [Core Concepts](/evals/core-concepts/concepts-overview)+3. Report issues at the [Letta Evals GitHub repository](https://github.com/letta-ai/letta-evals)++When reporting issues, include:+- Suite YAML configuration+- Dataset sample (if not sensitive)+- Error message and full stack trace+- Environment info (OS, Python version)++```bash+# Get environment info+python --version+pip show letta-evals+```