The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
docs: add agent evals framework plan
Dottadb81a0638671
diff --git a/doc/plans/2026-03-13-agent-evals-framework.md b/doc/plans/2026-03-13-agent-evals-framework.mdnew file mode 100644index 0000000000..6c4cc55ebb--- /dev/null+++ b/doc/plans/2026-03-13-agent-evals-framework.md@@ -0,0 +1,775 @@+# Agent Evals Framework Plan++Date: 2026-03-13++## Context++We need evals for the thing Paperclip actually ships:++- agent behavior produced by adapter config+- prompt templates and bootstrap prompts+- skill sets and skill instructions+- model choice+- runtime policy choices that affect outcomes and cost++We do **not** primarily need a fine-tuning pipeline.+We need a regression framework that can answer:++- if we change prompts or skills, do agents still do the right thing?+- if we switch models, what got better, worse, or more expensive?+- if we optimize tokens, did we preserve task outcomes?+- can we grow the suite over time from real Paperclip usage?++This plan is based on:++- `doc/GOAL.md`+- `doc/PRODUCT.md`+- `doc/SPEC-implementation.md`+- `docs/agents-runtime.md`+- `doc/plans/2026-03-13-TOKEN-OPTIMIZATION-PLAN.md`+- Discussion #449: <https://github.com/paperclipai/paperclip/discussions/449>+- OpenAI eval best practices: <https://developers.openai.com/api/docs/guides/evaluation-best-practices>+- Promptfoo docs: <https://www.promptfoo.dev/docs/configuration/test-cases/> and <https://www.promptfoo.dev/docs/providers/custom-api/>+- LangSmith complex agent eval docs: <https://docs.langchain.com/langsmith/evaluate-complex-agent>+- Braintrust dataset/scorer docs: <https://www.braintrust.dev/docs/annotate/datasets> and <https://www.braintrust.dev/docs/evaluate/write-scorers>++## Recommendation++Paperclip should take a **two-stage approach**:++1. **Start with Promptfoo now** for narrow, prompt-and-skill behavior evals across models.+2. **Grow toward a first-party, repo-local eval harness in TypeScript** for full Paperclip scenario evals.++So the recommendation is no longer “skip Promptfoo.” It is:++- use Promptfoo as the fastest bootstrap layer+- keep eval cases and fixtures in this repo+- avoid making Promptfoo config the deepest long-term abstraction++More specifically:++1. The canonical eval definitions should live in this repo under a top-level `evals/` directory.+2. `v0` should use Promptfoo to run focused test cases across models and providers.+3. The longer-term harness should run **real Paperclip scenarios** against seeded companies/issues/agents, not just raw prompt completions.+4. The scoring model should combine:+ - deterministic checks+ - structured rubric scoring+ - pairwise candidate-vs-baseline judging+ - efficiency metrics from normalized usage/cost telemetry+5. The framework should compare **bundles**, not just models.++A bundle is:++- adapter type+- model id+- prompt template(s)+- bootstrap prompt template+- skill allowlist / skill content version+- relevant runtime flags++That is the right unit because that is what actually changes behavior in Paperclip.++## Why This Is The Right Shape++### 1. We need to evaluate system behavior, not only prompt output++Prompt-only tools are useful, but Paperclip’s real failure modes are often:++- wrong issue chosen+- wrong API call sequence+- bad delegation+- failure to respect approval boundaries+- stale session behavior+- over-reading context+- claiming completion without producing artifacts or comments++Those are control-plane behaviors. They require scenario setup, execution, and trace inspection.++### 2. The repo is already TypeScript-first++The existing monorepo already uses:++- `pnpm`+- `tsx`+- `vitest`+- TypeScript across server, UI, shared contracts, and adapters++A TypeScript-first harness will fit the repo and CI better than introducing a Python-first test subsystem as the default path.++Python can stay optional later for specialty scorers or research experiments.++### 3. We need provider/model comparison without vendor lock-in++OpenAI’s guidance is directionally right:++- eval early and often+- use task-specific evals+- log everything+- prefer pairwise/comparison-style judging over open-ended scoring++But OpenAI’s Evals API is not the right control plane for Paperclip as the primary system because our target is explicitly multi-model and multi-provider.++### 4. Hosted eval products are useful, and Promptfoo is the right bootstrap tool++The current tradeoff:++- Promptfoo is very good for local, repo-based prompt/provider matrices and CI integration.+- LangSmith is strong on trajectory-style agent evals.+- Braintrust has a clean dataset + scorer + experiment model and strong TypeScript support.++The community suggestion is directionally right:++- Promptfoo lets us start small+- it supports simple assertions like contains / not-contains / regex / custom JS+- it can run the same cases across multiple models+- it supports OpenRouter+- it can move into CI later++That makes it the best `v0` tool for “did this prompt/skill/model change obviously regress?”++But Paperclip should still avoid making a hosted platform or a third-party config format the core abstraction before we have our own stable eval model.++The right move is:++- start with Promptfoo for quick wins+- keep the data portable and repo-owned+- build a thin first-party harness around Paperclip concepts as the system grows+- optionally export to or integrate with other tools later if useful++## What We Should Evaluate++We should split evals into four layers.++### Layer 1: Deterministic contract evals++These should require no judge model.++Examples:++- agent comments on the assigned issue+- no mutation outside the agent’s company+- approval-required actions do not bypass approval flow+- task transitions are legal+- output contains required structured fields+- artifact links exist when the task required an artifact+- no full-thread refetch on delta-only cases once the API supports it++These are cheap, reliable, and should be the first line of defense.++### Layer 2: Single-step behavior evals++These test narrow behaviors in isolation.++Examples:++- chooses the correct issue from inbox+- writes a reasonable first status comment+- decides to ask for approval instead of acting directly+- delegates to the correct report+- recognizes blocked state and reports it clearly++These are the closest thing to prompt evals, but still framed in Paperclip terms.++### Layer 3: End-to-end scenario evals++These run a full heartbeat or short sequence of heartbeats against a seeded scenario.++Examples:++- new assignment pickup+- long-thread continuation+- mention-triggered clarification+- approval-gated hire request+- manager escalation+- workspace coding task that must leave a meaningful issue update++These should evaluate both final state and trace quality.++### Layer 4: Efficiency and regression evals++These are not “did the answer look good?” evals. They are “did we preserve quality while improving cost/latency?” evals.++Examples:++- normalized input tokens per successful heartbeat+- normalized tokens per completed issue+- session reuse rate+- full-thread reload rate+- wall-clock duration+- cost per successful scenario++This layer is especially important for token optimization work.++## Core Design++## 1. Canonical object: `EvalCase`++Each eval case should define:++- scenario setup+- target bundle(s)+- execution mode+- expected invariants+- scoring rubric+- tags/metadata++Suggested shape:++```ts+type EvalCase = {+ id: string;+ description: string;+ tags: string[];+ setup: {+ fixture: string;+ agentId: string;+ trigger: "assignment" | "timer" | "on_demand" | "comment" | "approval";+ };+ inputs?: Record<string, unknown>;+ checks: {+ hard: HardCheck[];+ rubric?: RubricCheck[];+ pairwise?: PairwiseCheck[];+ };+ metrics: MetricSpec[];+};+```++The important part is that the case is about a Paperclip scenario, not a standalone prompt string.++## 2. Canonical object: `EvalBundle`++Suggested shape:++```ts+type EvalBundle = {+ id: string;+ adapter: string;+ model: string;+ promptTemplate: string;+ bootstrapPromptTemplate?: string;+ skills: string[];+ flags?: Record<string, string | number | boolean>;+};+```++Every comparison run should say which bundle was tested.++This avoids the common mistake of saying “model X is better” when the real change was model + prompt + skills + runtime behavior.++## 3. Canonical output: `EvalTrace`++We should capture a normalized trace for scoring:++- run ids+- prompts actually sent+- session reuse metadata+- issue mutations+- comments created+- approvals requested+- artifacts created+- token/cost telemetry+- timing+- raw outputs++The scorer layer should never need to scrape ad hoc logs.++## Scoring Framework++## 1. Hard checks first++Every eval should start with pass/fail checks that can invalidate the run immediately.++Examples:++- touched wrong company+- skipped required approval+- no issue update produced+- returned malformed structured output+- marked task done without required artifact++If a hard check fails, the scenario fails regardless of style or judge score.++## 2. Rubric scoring second++Rubric scoring should use narrow criteria, not vague “how good was this?” prompts.++Good rubric dimensions:++- task understanding+- governance compliance+- useful progress communication+- correct delegation+- evidence of completion+- concision / unnecessary verbosity++Each rubric should be a small 0-1 or 0-2 decision, not a mushy 1-10 scale.++## 3. Pairwise judging for candidate vs baseline++OpenAI’s eval guidance is right that LLMs are better at discrimination than open-ended generation.++So for non-deterministic quality checks, the default pattern should be:++- run baseline bundle on the case+- run candidate bundle on the same case+- ask a judge model which is better on explicit criteria+- allow `baseline`, `candidate`, or `tie`++This is better than asking a judge for an absolute quality score with no anchor.++## 4. Efficiency scoring is separate++Do not bury efficiency inside a single blended quality score.++Record it separately:++- quality score+- cost score+- latency score++Then compute a summary decision such as:++- candidate is acceptable only if quality is non-inferior and efficiency is improved++That is much easier to reason about than one magic number.++## Suggested Decision Rule++For PR gating:++1. No hard-check regressions.+2. No significant regression on required scenario pass rate.+3. No significant regression on key rubric dimensions.+4. If the change is token-optimization-oriented, require efficiency improvement on target scenarios.++For deeper comparison reports, show:++- pass rate+- pairwise wins/losses/ties+- median normalized tokens+- median wall-clock time+- cost deltas++## Dataset Strategy++We should explicitly build the dataset from three sources.++### 1. Hand-authored seed cases++Start here.++These should cover core product invariants:++- assignment pickup+- status update+- blocked reporting+- delegation+- approval request+- cross-company access denial+- issue comment follow-up++These are small, clear, and stable.++### 2. Production-derived cases++Per OpenAI’s guidance, we should log everything and mine real usage for eval cases.++Paperclip should grow eval coverage by promoting real runs into cases when we see:++- regressions+- interesting failures+- edge cases+- high-value success patterns worth preserving++The initial version can be manual:++- take a real run+- redact/normalize it+- convert it into an `EvalCase`++Later we can automate trace-to-case generation.++### 3. Adversarial and guardrail cases++These should intentionally probe failure modes:++- approval bypass attempts+- wrong-company references+- stale context traps+- irrelevant long threads+- misleading instructions in comments+- verbosity traps++This is where promptfoo-style red-team ideas can become useful later, but it is not the first slice.++## Repo Layout++Recommended initial layout:++```text+evals/+ README.md+ promptfoo/+ promptfooconfig.yaml+ prompts/+ cases/+ cases/+ core/+ approvals/+ delegation/+ efficiency/+ fixtures/+ companies/+ issues/+ bundles/+ baseline/+ experiments/+ runners/+ scenario-runner.ts+ compare-runner.ts+ scorers/+ hard/+ rubric/+ pairwise/+ judges/+ rubric-judge.ts+ pairwise-judge.ts+ lib/+ types.ts+ traces.ts+ metrics.ts+ reports/+ .gitignore+```++Why top-level `evals/`:++- it makes evals feel first-class+- it avoids hiding them inside `server/` even though they span adapters and runtime behavior+- it leaves room for both TS and optional Python helpers later+- it gives us a clean place for Promptfoo `v0` config plus the later first-party runner++## Execution Model++The harness should support three modes.++### Mode A: Cheap local smoke++Purpose:++- run on PRs+- keep cost low+- catch obvious regressions++Characteristics:++- 5 to 20 cases+- 1 or 2 bundles+- mostly hard checks and narrow rubrics++### Mode B: Candidate vs baseline compare++Purpose:++- evaluate a prompt/skill/model change before merge++Characteristics:++- paired runs+- pairwise judging enabled+- quality + efficiency diff report++### Mode C: Nightly broader matrix++Purpose:++- compare multiple models and bundles+- grow historical benchmark data++Characteristics:++- larger case set+- multiple models+- more expensive rubric/pairwise judging++## CI and Developer Workflow++Suggested commands:++```sh+pnpm evals:smoke+pnpm evals:compare --baseline baseline/codex-default --candidate experiments/codex-lean-skillset+pnpm evals:nightly+```++PR behavior:++- run `evals:smoke` on prompt/skill/adapter/runtime changes+- optionally trigger `evals:compare` for labeled PRs or manual runs++Nightly behavior:++- run larger matrix+- save report artifact+- surface trend lines on pass rate, pairwise wins, and efficiency++## Framework Comparison++## Promptfoo++Best use for Paperclip:++- prompt-level micro-evals+- provider/model comparison+- quick local CI integration+- custom JS assertions and custom providers+- bootstrap-layer evals for one skill or one agent workflow++What changed in this recommendation:++- Promptfoo is now the recommended **starting point**+- especially for “one skill, a handful of cases, compare across models”++Why it still should not be the only long-term system:++- its primary abstraction is still prompt/provider/test-case oriented+- Paperclip needs scenario setup, control-plane state inspection, and multi-step traces as first-class concepts++Recommendation:++- use Promptfoo first+- store Promptfoo config and cases in-repo under `evals/promptfoo/`+- use custom JS/TS assertions and, if needed later, a custom provider that calls Paperclip scenario runners+- do not make Promptfoo YAML the only canonical Paperclip eval format once we outgrow prompt-level evals++## LangSmith++What it gets right:++- final response evals+- trajectory evals+- single-step evals++Why not the primary system today:++- stronger fit for teams already centered on LangChain/LangGraph+- introduces hosted/external workflow gravity before our own eval model is stable++Recommendation:++- copy the trajectory/final/single-step taxonomy+- do not adopt the platform as the default requirement++## Braintrust++What it gets right:++- TypeScript support+- clean dataset/task/scorer model+- production logging to datasets+- experiment comparison over time++Why not the primary system today:++- still externalizes the canonical dataset and review workflow+- we are not yet at the maturity where hosted experiment management should define the shape of the system++Recommendation:++- borrow its dataset/scorer/experiment mental model+- revisit once we want hosted review and experiment history at scale++## OpenAI Evals / Evals API++What it gets right:++- strong eval principles+- emphasis on task-specific evals+- continuous evaluation mindset++Why not the primary system:++- Paperclip must compare across models/providers+- we do not want our primary eval runner coupled to one model vendor++Recommendation:++- use the guidance+- do not use it as the core Paperclip eval runtime++## First Implementation Slice++The first version should be intentionally small.++## Phase 0: Promptfoo bootstrap++Build:++- `evals/promptfoo/promptfooconfig.yaml`+- 5 to 10 focused cases for one skill or one agent workflow+- model matrix using the providers we care about most+- mostly deterministic assertions:+ - contains+ - not-contains+ - regex+ - custom JS assertions++Target scope:++- one skill, or one narrow workflow such as assignment pickup / first status update+- compare a small set of bundles across several models++Success criteria:++- we can run one command and compare outputs across models+- prompt/skill regressions become visible quickly+- the team gets signal before building heavier infrastructure++## Phase 1: Skeleton and core cases++Build:++- `evals/` scaffold+- `EvalCase`, `EvalBundle`, `EvalTrace` types+- scenario runner for seeded local cases+- 10 hand-authored core cases+- hard checks only++Target cases:++- assigned issue pickup+- write progress comment+- ask for approval when required+- respect company boundary+- report blocked state+- avoid marking done without artifact/comment evidence++Success criteria:++- a developer can run a local smoke suite+- prompt/skill changes can fail the suite deterministically+- Promptfoo `v0` cases either migrate into or coexist with this layer cleanly++## Phase 2: Pairwise and rubric layer++Build:++- rubric scorer interface+- pairwise judge runner+- candidate vs baseline compare command+- markdown/html report output++Success criteria:++- model/prompt bundle changes produce a readable diff report+- we can tell “better”, “worse”, or “same” on curated scenarios++## Phase 3: Efficiency integration++Build:++- normalized token/cost metrics into eval traces+- cost and latency comparisons+- efficiency gates for token optimization work++Dependency:++- this should align with the telemetry normalization work in `2026-03-13-TOKEN-OPTIMIZATION-PLAN.md`++Success criteria:++- quality and efficiency can be judged together+- token-reduction work no longer relies on anecdotal improvements++## Phase 4: Production-case ingestion++Build:++- tooling to promote real runs into new eval cases+- metadata tagging+- failure corpus growth process++Success criteria:++- the eval suite grows from real product behavior instead of staying synthetic++## Initial Case Categories++We should start with these categories:++1. `core.assignment_pickup`+2. `core.progress_update`+3. `core.blocked_reporting`+4. `governance.approval_required`+5. `governance.company_boundary`+6. `delegation.correct_report`+7. `threads.long_context_followup`+8. `efficiency.no_unnecessary_reloads`++That is enough to start catching the classes of regressions we actually care about.++## Important Guardrails++### 1. Do not rely on judge models alone++Every important scenario needs deterministic checks first.++### 2. Do not gate PRs on a single noisy score++Use pass/fail invariants plus a small number of stable rubric or pairwise checks.++### 3. Do not confuse benchmark score with product quality++The suite must keep growing from real runs, otherwise it will become a toy benchmark.++### 4. Do not evaluate only final output++Trajectory matters for agents:++- did they call the right Paperclip APIs?+- did they ask for approval?+- did they communicate progress?+- did they choose the right issue?++### 5. Do not make the framework vendor-shaped++Our eval model should survive changes in:++- judge provider+- candidate provider+- adapter implementation+- hosted tooling choices++## Open Questions++1. Should the first scenario runner invoke the real server over HTTP, or call services directly in-process?+ My recommendation: start in-process for speed, then add HTTP-mode coverage once the model stabilizes.++2. Should we support Python scorers in v1?+ My recommendation: no. Keep v1 all-TypeScript.++3. Should we commit baseline outputs?+ My recommendation: commit case definitions and bundle definitions, but keep run artifacts out of git.++4. Should we add hosted experiment tracking immediately?+ My recommendation: no. Revisit after the local harness proves useful.++## Final Recommendation++Start with Promptfoo for immediate, narrow model-and-prompt comparisons, then grow into a first-party `evals/` framework in TypeScript that evaluates **Paperclip scenarios and bundles**, not just prompts.++Use this structure:++- Promptfoo for `v0` bootstrap+- deterministic hard checks as the foundation+- rubric and pairwise judging for non-deterministic quality+- normalized efficiency metrics as a separate axis+- repo-local datasets that grow from real runs++Use external tools selectively:++- Promptfoo as the initial path for narrow prompt/provider tests+- Braintrust or LangSmith later if we want hosted experiment management++But keep the canonical eval model inside the Paperclip repo and aligned to Paperclip’s actual control-plane behaviors.