Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(mcp): remove benchmark evals (#118775)

Radu Raicea34d9c3fc8df4
diff --git a/.agents/skills/improving-mcp-tools/SKILL.md b/.agents/skills/improving-mcp-tools/SKILL.md
index 6e1729e58951..d99a2ff7a3d1 100644
--- a/.agents/skills/improving-mcp-tools/SKILL.md
+++ b/.agents/skills/improving-mcp-tools/SKILL.md
@@ -2,7 +2,7 @@
name: improving-mcp-tools
description: >
Run an improve-my-MCP campaign: an autoresearch-style loop that measures the
- MCP agent experience with the eval harness, picks the highest-impact tool
+ MCP agent experience with a supplied eval harness, picks the highest-impact tool
problem from production data, makes one bounded fix, and keeps it only if
before/after scores improve. Use when asked to "improve my MCP", run an MCP
improvement campaign, fix tool discoverability or descriptions based on
@@ -21,22 +21,17 @@ repeating work.
## The objective function
-`services/mcp/evals/` is the harness. `benchmark/tasks.yaml` is a fixed set of
-agent tasks with `expected_tools` and `success_criteria`; scores are only
-comparable across runs of the same benchmark `version`.
+This repository does not include an MCP benchmark or eval runner.
+Before starting a campaign, obtain a benchmark and runnable harness from the operator.
+Record the task set, expected tools, success criteria, benchmark version, and exact commands in the journal.
+If those inputs are unavailable, report the missing evidence and stop the campaign.
-- **Probe mode** (deterministic, no LLM):
- `LIVE_MCP_URL=... LIVE_MCP_TOKEN=... pnpm exec tsx evals/runner/probe.ts --out score.json`
- from `services/mcp/`. Reports tool-presence misses (discoverability), probe
- failures, and latency p50/p95. Non-zero exit = regression.
-- **Agent mode** (LLM replay + judge): scores task success and tool-selection
- accuracy. Use it for description/discoverability changes — probes cannot
- detect that an agent picks the wrong tool.
+- **Probe mode** (deterministic, no LLM): measures tool-presence misses, probe failures, and latency p50/p95.
+- **Agent mode** (LLM replay + judge): scores task success and tool-selection accuracy.
+ Use it for description/discoverability changes, since probes cannot detect that an agent picks the wrong tool.
-Run the harness against a **seeded local or devbox stack**, never against a
-customer project. Local recipe: `NODE_ENV=development PORT=9876
-POSTHOG_API_BASE_URL=http://localhost:8000 pnpm dev:hono`, personal API key as
-`LIVE_MCP_TOKEN`.
+Run the supplied harness against a seeded local or devbox stack.
+Use the same benchmark version for baseline and after scores.
## One iteration
@@ -69,7 +64,7 @@ POSTHOG_API_BASE_URL=http://localhost:8000 pnpm dev:hono`, personal API key as
These are not suggestions; violating any of them ends the campaign pass.
- **Allowlist** — a campaign PR may only touch: `products/*/mcp/tools.yaml`,
- `products/*/skills/**`, `services/mcp/evals/**`, the codegen outputs of
+ `products/*/skills/**`, the codegen outputs of
`pnpm generate-tools` / `scaffold-yaml` (`services/mcp/src/tools/generated/**`
and `services/mcp/schema/generated-tool-definitions.json`), and docs.
Anything else (handler code, package manifests, workflows, migrations, auth
@@ -80,7 +75,7 @@ These are not suggestions; violating any of them ends the campaign pass.
measuring.
- **Evidence or it didn't happen.** No PR without a baseline score, an after
score, and the exact harness commands used.
-- **Benchmark integrity.** Never edit `benchmark/tasks.yaml` in the same PR as
+- **Benchmark integrity.** Never edit the supplied benchmark in the same PR as
a fix it validates — changing the exam and the answer together proves
nothing. Benchmark changes are their own PR and bump `version`.
- **Budgets.** Respect the operator's iteration/token/PR caps (default: stop
diff --git a/services/mcp/evals/README.md b/services/mcp/evals/README.md
deleted file mode 100644
index 954ca5a622ef..000000000000
--- a/services/mcp/evals/README.md
+++ /dev/null
@@ -1,76 +0,0 @@
-# MCP evals
-
-A benchmark of agent tasks for the PostHog MCP server, a script that runs each task's probe call against a live server, and a script that seeds the feature flags some tasks need.
-
-## Layout
-
-- `benchmark/tasks.yaml` — the task set, plus a `fixtures` block of feature flags that some tasks act on.
-- `benchmark/schema.ts` — the zod schema and loader for `tasks.yaml`.
-- `runner/probe.ts` — runs each task's `probe` against a live MCP server. See [Probe runner](#probe-runner).
-- `runner/results.ts` — the result types and summary for a probe run.
-- `runner/seed.ts` — writes the fixture flags to a PostHog project. See [Seeding](#seeding).
-
-`tests/evals/benchmark.test.ts` runs in CI and checks `tasks.yaml`. Among other checks, it fails when a task names a tool that is not in the MCP tool catalog, when a probe uses a tool that is not read-only, or when a fixture flag is named in no task intent or in more than one.
-
-## Task format
-
-```yaml
-- id: flags-list-active # kebab-case, unique
- category: feature-flags # one of TASK_CATEGORIES in schema.ts
- intent: 'List all our active feature flags.' # the request, phrased like a real user's
- expected_tools: [feature-flag-get-all] # tools a competent agent should call
- acceptable_tools: [execute-sql] # other tools that are also a reasonable path
- success_criteria: "Returns the project's active flags by key." # what a correct answer does
- probe: # optional: one read-only call the probe runner makes
- tool: feature-flag-get-all
- args: {}
- max_ms: 15000
-```
-
-The probe runner checks `expected_tools` and runs `probe`. No script here scores `intent` or `success_criteria`.
-
-## Probe runner
-
-Run from `services/mcp/`:
-
-```bash
-LIVE_MCP_URL=http://localhost:9876 LIVE_MCP_TOKEN=phx_... \
- pnpm exec tsx evals/runner/probe.ts [--out score.json]
-```
-
-It connects in tools mode, then:
-
-1. Lists the tools the server advertises, and reports each task's `expected_tools` or `probe` tool that is missing.
-2. Runs each task's `probe` call and records its status and latency.
- It refuses a probe whose tool the server does not advertise as read-only, so a bad fixture cannot change project data.
-3. Prints a summary: missing tools, probes passed and failed, and latency p50/p95. `--out` also writes it as JSON.
-
-The exit code is non-zero when a tool is missing or a probe fails.
-
-## Seeding
-
-Some tasks ask an agent to create or change feature flags.
-The `fixtures` block lists those flags, and `runner/seed.ts` puts them in a known state.
-Run from `services/mcp/`:
-
-```bash
-LIVE_POSTHOG_URL=http://localhost:8000 LIVE_MCP_TOKEN=phx_... \
- pnpm exec tsx evals/runner/seed.ts [--project 12345]
-```
-
-- It creates or updates every flag in `fixtures.feature_flags`, and soft-deletes every key in `fixtures.absent_feature_flags`. Running it again resets those flags.
-- `flag-create-routes-to-experiment` is the exception. The agent picks the key of the flag its experiment manages, so the seeder cannot reset it. Delete that experiment and flag by hand when you need a clean project.
-- It writes through the REST API, not the MCP server, so the tools under test do not build their own starting state.
-- Without `--project` it seeds the token's current project. Seed the project the MCP session acts on.
-- The token needs `feature_flag:write`, plus `user:read` when `--project` is omitted.
-- The exit code is non-zero when any fixture could not be written.
-
-Probes are read-only, so the probe runner needs no seeding.
-
-## Authoring rules
-
-- Sample intents from real usage (`$mcp_intent` clusters, `query-mcp-tool-sample-intents`) but **paraphrase — never paste customer text verbatim**, and never include PII.
-- Results are only comparable across runs of the same benchmark `version`. Bump it on breaking changes to the set, together with the version literal in `schema.ts`.
-- Weight new tasks toward observed pain: high-error tools, discoverability misses (intents where agents picked the wrong tool), and multi-step chains (resolve id → act).
-- Give each task that changes a flag its own fixture flag, so the result does not depend on task order. The fixture test enforces this.
-- A task must be achievable in a seeded demo project — don't write tasks that depend on one specific production dataset.
diff --git a/services/mcp/evals/benchmark/schema.ts b/services/mcp/evals/benchmark/schema.ts
deleted file mode 100644
index b2d3a86bd3d1..000000000000
--- a/services/mcp/evals/benchmark/schema.ts
+++ /dev/null
@@ -1,143 +0,0 @@
-import { readFileSync } from 'node:fs'
-import { fileURLToPath } from 'node:url'
-import { parse as parseYaml } from 'yaml'
-import { z } from 'zod'
-
-/**
- * Benchmark task file schema — the objective function for the MCP agent
- * experience. Tasks are sampled from real agent intents (paraphrased, never
- * verbatim) and describe what a competent agent should achieve against a
- * live MCP server, which tools it is expected to reach for, and optionally a
- * deterministic probe call the runner can execute without an LLM.
- */
-
-export const TASK_CATEGORIES = [
- 'sql',
- 'data-schema',
- 'product-analytics',
- 'insights',
- 'error-tracking',
- 'feature-flags',
- 'session-replay',
- 'llm-analytics',
- 'logs',
- 'dashboards',
- 'docs',
- 'project',
- 'mcp-analytics',
- 'metrics',
- 'data-catalog',
-] as const
-
-export const BenchmarkProbeSchema = z.object({
- // Probes must reference read-only tools; the runner refuses to execute a
- // probe whose tool lacks readOnlyHint so a bad fixture can't mutate data.
- tool: z.string().min(1),
- args: z.record(z.string(), z.unknown()).default({}),
- max_ms: z.number().int().positive().default(15_000),
-})
-
-export const BenchmarkTaskSchema = z.object({
- id: z
- .string()
- .regex(/^[a-z0-9][a-z0-9-]+$/, 'task ids are kebab-case')
- .max(64),
- category: z.enum(TASK_CATEGORIES),
- // The user-visible goal the agent is given, written like a real request.
- intent: z.string().min(10),
- // Tools a competent agent is expected to call to complete the task.
- expected_tools: z.array(z.string().min(1)).min(1),
- // Tools that also count as a reasonable path (no tool-selection penalty).
- acceptable_tools: z.array(z.string().min(1)).default([]),
- // Plain-language pass condition consumed by the agent-mode judge.
- success_criteria: z.string().min(10),
- probe: BenchmarkProbeSchema.optional(),
-})
-
-/** Prefix every seeded entity carries, so a fixture is never mistaken for real project data. */
-export const FIXTURE_KEY_PREFIX = 'mcp-eval-'
-
-/** Tag the seeder puts on every fixture flag, so a project's own flags stay distinguishable. */
-export const FIXTURE_TAG = 'mcp-eval'
-
-/** Both fixture lists name keys the same seeder owns, so they validate through one schema. */
-const FixtureFlagKeySchema = z
- .string()
- .regex(new RegExp(`^${FIXTURE_KEY_PREFIX}[a-z0-9-]+$`), `flag fixture keys start with ${FIXTURE_KEY_PREFIX}`)
- .max(200)
-
-export const BenchmarkFlagFixtureSchema = z
- .object({
- key: FixtureFlagKeySchema,
- name: z.string().min(1),
- active: z.boolean(),
- archived: z.boolean(),
- // Release conditions, variants and payloads, in the API's own shape. Tasks that assert
- // an edit preserved unrelated configuration need something here to preserve.
- filters: z.record(z.string(), z.unknown()).default({}),
- })
- .strict()
- // The API disables a flag on the way to archiving it, so a fixture asking for both would
- // describe a state the seeder cannot leave the flag in.
- .refine((fixture) => !(fixture.archived && fixture.active), 'an archived fixture flag cannot also be active')
-
-// Strict all the way down, because every fixture container defaults to empty. A misspelled
-// key would otherwise be stripped, leaving the seeder nothing to write and no error to say so.
-export const BenchmarkFixturesSchema = z
- .object({
- feature_flags: z.array(BenchmarkFlagFixtureSchema).default([]),
- // Keys a task is asked to create. The seeder soft-deletes each one, so the create
- // path is exercised rather than a second run hitting an already-taken key.
- absent_feature_flags: z.array(FixtureFlagKeySchema).default([]),
- })
- .strict()
-
-export const BenchmarkFileSchema = z
- .object({
- version: z.literal(2),
- // Entities `runner/seed.ts` creates and resets before an agent-mode run, so tasks that
- // mutate state start from a known place and score the same on a second run.
- fixtures: BenchmarkFixturesSchema.default({ feature_flags: [], absent_feature_flags: [] }),
- tasks: z.array(BenchmarkTaskSchema).min(1),
- })
- .strict()
-
-export type BenchmarkProbe = z.infer<typeof BenchmarkProbeSchema>
-export type BenchmarkTask = z.infer<typeof BenchmarkTaskSchema>
-export type BenchmarkFlagFixture = z.infer<typeof BenchmarkFlagFixtureSchema>
-export type BenchmarkFixtures = z.infer<typeof BenchmarkFixturesSchema>
-export type BenchmarkFile = z.infer<typeof BenchmarkFileSchema>
-
-export const DEFAULT_BENCHMARK_PATH = fileURLToPath(new URL('./tasks.yaml', import.meta.url))
-
-export function loadBenchmark(path: string = DEFAULT_BENCHMARK_PATH): BenchmarkFile {
- return BenchmarkFileSchema.parse(parseYaml(readFileSync(path, 'utf-8')))
-}
-
-/**
- * Fixture flag keys a task's intent names. Task independence rides on this: two tasks that
- * mutate the same fixture flag make the run order-dependent and stop a second run from
- * starting where the first one did, so the fixture test asserts no key is named twice.
- */
-export function fixtureFlagKeysInIntent(task: BenchmarkTask, fixtures: BenchmarkFixtures): string[] {
- return fixtureFlagKeys(fixtures).filter((key) => task.intent.includes(key))
-}
-
-/** Every flag key the seeder owns — the ones it writes and the ones it clears. */
-export function fixtureFlagKeys(fixtures: BenchmarkFixtures): string[] {
- return [...fixtures.feature_flags.map((fixture) => fixture.key), ...fixtures.absent_feature_flags]
-}
-
-export function referencedTools(task: BenchmarkTask): string[] {
- return [...new Set([...task.expected_tools, ...task.acceptable_tools, ...(task.probe ? [task.probe.tool] : [])])]
-}
-
-/**
- * Tools that MUST be advertised for a task to be runnable: the expected path
- * plus any deterministic probe tool. `acceptable_tools` are optional
- * alternatives — often feature-gated — so a missing one is not a benchmark
- * failure and is deliberately excluded here.
- */
-export function requiredTools(task: BenchmarkTask): string[] {
- return [...new Set([...task.expected_tools, ...(task.probe ? [task.probe.tool] : [])])]
-}
diff --git a/services/mcp/evals/benchmark/tasks.yaml b/services/mcp/evals/benchmark/tasks.yaml
deleted file mode 100644
index f822d33af8d4..000000000000
--- a/services/mcp/evals/benchmark/tasks.yaml
+++ /dev/null
@@ -1,518 +0,0 @@
-# MCP agent-experience benchmark v2.
-#
-# Task intents are paraphrased from real agent usage (top tool families by
-# production volume) — never verbatim copies of customer text. Keep probes
-# read-only; the runner refuses non-read-only probe tools.
-#
-# v2 gave every feature-flag task its own fixture flag and added the fixtures
-# below, so flag scores are no longer order-dependent. Scores are comparable
-# within a version, not across one.
-version: 2
-
-# Entities `runner/seed.ts` writes to the target project before an agent-mode run.
-# One fixture per task that mutates it: a task run twice, or a whole run repeated,
-# then starts from the state declared here rather than from what the last run left.
-fixtures:
- feature_flags:
- - key: mcp-eval-enable-me
- name: MCP eval — enable target
- active: false
- archived: false
- filters:
- groups:
- - properties: []
- rollout_percentage: 100
-
- - key: mcp-eval-disable-me
- name: MCP eval — disable target
- active: true
- archived: false
- filters:
- groups:
- - properties: []
- rollout_percentage: 100
-
- - key: mcp-eval-archive-me
- name: MCP eval — archive target
- # Enabled on purpose: archiving has to turn the flag off too, and the task
- # scores whether the agent says so.
- active: true
- archived: false
- filters:
- groups:
- - properties: []
- rollout_percentage: 100
-
- - key: mcp-eval-unarchive-me
- name: MCP eval — unarchive target
- active: false
- archived: true
- filters:
- groups:
- - properties: []
- rollout_percentage: 100
-
- - key: mcp-eval-retarget-me
- name: MCP eval — targeting-edit target
- active: true
- archived: false
- # Two groups, variants and a payload, so a targeting edit through the
- # generic update tool has unrelated configuration it can drop.
- filters:
- groups:
- - properties:
- - key: email
- type: person
- value: ['[email protected]']
- operator: exact
- rollout_percentage: 100
- - properties: []
- rollout_percentage: 10
- multivariate:
- variants:
- - key: control
- name: Control
- rollout_percentage: 50
- - key: test
- name: Test
- rollout_percentage: 50
- payloads:
- control: '{"layout":"list"}'
- test: '{"layout":"grid"}'
-
- - key: mcp-eval-rollout-conditions
- name: MCP eval — rollout-conditions read target
- active: true
- archived: false
- filters:
- groups:
- - properties:
- - key: '$geoip_country_code'
- type: person
- value: ['GB', 'IE']
- operator: exact
- rollout_percentage: 25
-
- # `flag-create-killswitch-no-experiment` creates this key, so the seeder clears it
- # first. `flag-create-routes-to-experiment` has no counterpart here: the agent picks
- # the key of the flag the experiment manages, so the seeder cannot name it. That
- # task's leftovers still accumulate across runs.
- absent_feature_flags:
- - mcp-eval-billing-killswitch
-
-tasks:
- # --- sql ---------------------------------------------------------------
- - id: sql-daily-event-volume
- category: sql
- intent: 'How many events did we ingest per day over the last 7 days?'
- expected_tools: [execute-sql]
- acceptable_tools: [query-trends]
- success_criteria: 'Returns a per-day event count covering roughly the last seven days.'
- probe:
- tool: execute-sql
- args:
- query: 'SELECT toDate(timestamp) AS day, count() AS events FROM events WHERE timestamp > now() - INTERVAL 7 DAY GROUP BY day ORDER BY day'
-
- - id: sql-top-events
- category: sql
- intent: 'List the 10 most common event names this month with their counts.'
- expected_tools: [execute-sql]
- acceptable_tools: [read-data-schema, query-trends]
- success_criteria: 'Returns up to ten event names ranked by count for the last ~30 days.'
- probe:
- tool: execute-sql
- args:
- query: 'SELECT event, count() AS c FROM events WHERE timestamp > now() - INTERVAL 30 DAY GROUP BY event ORDER BY c DESC LIMIT 10'
-
- - id: sql-find-saved-insights
- category: sql
- intent: 'Do we already have any saved insights about signup conversion?'
- expected_tools: [execute-sql]
- acceptable_tools: [insight-get]
- success_criteria: 'Searches saved insights (system tables) for signup/conversion matches and reports what exists, checking the table schema before querying it.'
-
- # --- data-schema --------------------------------------------------------
- - id: schema-discover-onboarding-events
- category: data-schema
- intent: "What events does this project capture? I'm especially looking for anything related to onboarding."
- expected_tools: [read-data-schema]
- success_criteria: 'Lists project events and calls out onboarding-related ones, without inventing event names.'
- probe:
- tool: read-data-schema
- args:
- query:
- kind: events
-
- - id: schema-property-values
- category: data-schema
- intent: 'What browsers do our users use? Check the actual property values rather than guessing.'
- expected_tools: [read-data-schema]
- acceptable_tools: [execute-sql]
- success_criteria: 'Reads browser property values from the schema tool (or SQL) and reports observed values.'
-
- - id: warehouse-schema-tables
- category: data-schema
- intent: 'What tables are available in our data warehouse, and what columns do they have?'
- expected_tools: [execute-sql]
- success_criteria: 'Enumerates warehouse tables with column details via system.information_schema.'
-
- # --- product-analytics ---------------------------------------------------
- - id: trends-weekly-active-users
- category: product-analytics
- intent: 'Chart weekly active users for the past 8 weeks.'
- expected_tools: [query-trends]
- acceptable_tools: [execute-sql]
- success_criteria: 'Produces a weekly unique-user series covering roughly eight weeks, after confirming the event exists in the schema.'
-
- - id: funnel-signup-conversion
- category: product-analytics
- intent: "What's our conversion rate from viewing the site to signing up, this month?"
- expected_tools: [query-funnel]
- acceptable_tools: [execute-sql]
- success_criteria: 'Builds a two-step funnel from real events and reports a conversion percentage.'
-
- - id: retention-weekly-return
- category: product-analytics
- intent: 'Do users who signed up last month keep coming back week over week?'
- expected_tools: [query-retention]
- success_criteria: 'Returns a weekly retention breakdown for the signup cohort.'
-
- - id: paths-after-login
- category: product-analytics
- intent: 'What do users typically do right after logging in?'
- expected_tools: [query-paths]
- acceptable_tools: [execute-sql]
- success_criteria: 'Shows the most common event sequences starting from a login-like event.'
-
- - id: lifecycle-user-composition
- category: product-analytics
- intent: "Of this week's active users, how many are new versus returning versus resurrecting?"
- expected_tools: [query-lifecycle]
- success_criteria: 'Reports the lifecycle composition buckets for the current week.'
-
- - id: stickiness-engagement
- category: product-analytics
- intent: 'How many days per week do our most engaged users actually use the product?'
- expected_tools: [query-stickiness]
- success_criteria: 'Returns a stickiness distribution (users by number of active days).'
-
- - id: trends-actors-spike
- category: product-analytics
- intent: 'Something spiked yesterday in our exception events — which users are behind it?'
- expected_tools: [query-trends-actors]
- acceptable_tools: [execute-sql, query-error-tracking-issues-list]
- success_criteria: 'Identifies the people behind the spike data point rather than just re-plotting the trend.'
-
- - id: funnel-dropoff-actors
- category: product-analytics
- intent: 'Which users dropped off between the first and second step of our signup funnel this month?'
- expected_tools: [query-funnel-actors]
- acceptable_tools: [query-funnel, execute-sql]
- success_criteria: 'Lists the actual people who dropped off entering the second step (drop-off step semantics, not a converted-through step), rather than just re-plotting the funnel conversion rate.'
- probe:
- # Repeated-pageview steps keep the probe runnable in any seeded demo
- # project; funnelStep -2 = dropped off entering step 2. Probe mode
- # only guards arg acceptance, tool errors, and latency — an empty
- # actor list still passes. Step SEMANTICS (returning the right
- # people) are judged in agent mode via success_criteria.
- tool: query-funnel-actors
- args:
- funnelStep: -2
- includeRecordings: false
- source:
- kind: FunnelsQuery
- dateRange:
- date_from: '-30d'
- series:
- - kind: EventsNode
- event: '$pageview'
- - kind: EventsNode
- event: '$pageview'
-
- # --- insights -------------------------------------------------------------
- - id: insight-create-daily-signups
- category: insights
- intent: 'Save a trends insight tracking daily signups so the team can find it later.'
- expected_tools: [insight-create]
- acceptable_tools: [query-trends, read-data-schema]
- success_criteria: 'Creates a saved insight with a sensible name and returns its link.'
-
- - id: insight-rename
- category: insights
- intent: "Rename my 'Daily signups' insight to 'Signups (daily)'."
- expected_tools: [insight-update]
- acceptable_tools: [execute-sql, insight-get]
- success_criteria: 'Finds the existing insight by name and updates only its name.'
-
- - id: insight-run-existing
- category: insights
- intent: 'Run our saved weekly revenue insight and tell me the latest value.'
- expected_tools: [insight-query]
- acceptable_tools: [insight-get, execute-sql]
- success_criteria: 'Locates the saved insight, executes it, and reports the most recent data point.'
-
- # Multi-step chains below target the observed pain in this family: losing
- # the created insight's identifiers between calls, over-long descriptions,
- # and saving without ever verifying the query runs. Create/update are
- # mutating tools, so these tasks are agent-mode only (no probes).
- - id: insight-id-roundtrip
- category: insights
- intent: 'Create an insight tracking daily pageviews, then rename it, then run it and tell me the latest value.'
- expected_tools: [insight-create, insight-update, insight-query]
- acceptable_tools: [query-trends, read-data-schema, insight-get]
- success_criteria: 'Completes the whole chain reusing the identifiers returned by insight-create for the update and query calls — no schema-rejection retries from passing the wrong identifier field — and reports the latest data point.'
-
- - id: insight-create-with-description
- category: insights
- intent: "Save a trends insight for weekly signups, with a description explaining how it's calculated and any caveats, so teammates can trust it."
- expected_tools: [insight-create]
- acceptable_tools: [query-trends, read-data-schema]
- success_criteria: 'Creates the insight with a substantive non-empty description on the first attempt — no rejected-then-truncated retry loop on the description field.'
-
- - id: insight-create-verified
- category: insights
- intent: "Save a funnel from homepage visit to signup, and confirm it actually computes results before telling me it's done."
- expected_tools: [insight-create, insight-query]
- acceptable_tools: [query-funnel, read-data-schema]
- success_criteria: 'Verifies the saved funnel actually runs — via insight-query after saving or by pre-testing the query before saving — and reports computed results instead of claiming success without ever executing it.'
-
- # --- error-tracking --------------------------------------------------------
- - id: errors-top-issues
- category: error-tracking
- intent: 'What are our top error issues by volume this week?'
- expected_tools: [query-error-tracking-issues-list]
- acceptable_tools: [execute-sql]
- success_criteria: 'Lists the highest-volume error issues for the last seven days.'
-
- - id: errors-stack-traces
- category: error-tracking
- intent: 'Show me recent stack traces for our most common error.'
- expected_tools: [query-error-tracking-issue-events]
- acceptable_tools: [query-error-tracking-issues-list]
- success_criteria: 'Finds the top issue first, then fetches sample events with stack traces for it.'
-
- # --- feature-flags ----------------------------------------------------------
- - id: flags-list-active
- category: feature-flags
- intent: 'List all our active feature flags.'
- expected_tools: [feature-flag-get-all]
- success_criteria: "Returns the project's active flags by key."
- probe:
- tool: feature-flag-get-all
- args: {}
-
- - id: flag-rollout-conditions
- category: feature-flags
- intent: 'What are the rollout conditions for our mcp-eval-rollout-conditions feature flag?'
- expected_tools: [feature-flag-get-definition-by-key]
- acceptable_tools: [feature-flag-get-all, feature-flag-get-definition]
- success_criteria: >
- Resolves the flag by its key ('mcp-eval-rollout-conditions') directly — does not guess a numeric id — and
- reports the seeded country condition and its 25% rollout.
-
- # Each lifecycle task below names its own fixture flag, seeded to the state that task needs.
- # So the four run in any order, a second run starts where the first one did, and the score
- # covers the state change as well as the tool choice. They still score tool selection when
- # the fixture is missing, because an unseeded key must produce a clear answer rather than a
- # fallback to the generic update tool.
- - id: flag-enable-by-key
- category: feature-flags
- intent: 'Turn on the mcp-eval-enable-me feature flag.'
- expected_tools: [feature-flag-enable]
- acceptable_tools: [feature-flag-get-definition-by-key, feature-flag-get-all, feature-flag-get-definition]
- success_criteria: >
- Calls feature-flag-enable rather than a generic update, and sends no filters payload in any call. If the
- flag does not exist, says so rather than reaching for update-feature-flag.
-
- - id: flag-disable-reports-impact
- category: feature-flags
- intent: "We're seeing errors from the mcp-eval-disable-me flag. Switch it off."
- expected_tools: [feature-flag-disable]
- acceptable_tools:
- [
- feature-flag-get-definition-by-key,
- feature-flag-get-all,
- feature-flag-get-definition,
- feature-flags-dependent-flags-retrieve,
- ]
- success_criteria: >
- Resolves the key to an id and calls feature-flag-disable with no filters payload. Reads the definition first
- and says what else the flag is wired into (linked experiments, surveys, replay settings), or that nothing
- is. If the flag does not exist, says so rather than reaching for update-feature-flag.
-
- - id: flag-archive-completed
- category: feature-flags
- intent: 'The mcp-eval-archive-me rollout is finished. Archive the flag so it stops cluttering the list.'
- expected_tools: [feature-flag-archive]
- acceptable_tools: [feature-flag-get-definition-by-key, feature-flag-get-all, feature-flag-get-definition]
- success_criteria: >
- Calls feature-flag-archive with no filters payload, and tells the user archiving also turns the flag off.
- If the flag does not exist, says so rather than reaching for update-feature-flag.
-
- - id: flag-unarchive-stays-off
- category: feature-flags
- intent: 'Bring the mcp-eval-unarchive-me flag back into the flag list, I need to look at it again.'
- expected_tools: [feature-flag-unarchive]
- acceptable_tools: [feature-flag-get-all, feature-flag-get-definition-by-key, feature-flag-get-definition]
- success_criteria: >
- Calls feature-flag-unarchive with no filters payload and does not enable it, saying the flag is still turned
- off. Finds the archived flag with feature-flag-get-definition-by-key, which returns it with archived: true, or
- by passing {"archived":"true"} to feature-flag-get-all.
-
- - id: flag-targeting-edit-uses-update
- category: feature-flags
- intent: 'Take the mcp-eval-retarget-me flag from a 10% rollout to 25%.'
- expected_tools: [update-feature-flag]
- acceptable_tools: [feature-flag-get-definition-by-key, feature-flag-get-definition, feature-flag-get-all]
- success_criteria: >
- Reads the current definition first, then edits targeting through update-feature-flag. Does not reach for a
- lifecycle tool, which cannot change targeting. Sending `filters` replaces the whole object, so the written
- filters must keep the seeded email condition, both variants and both payloads, and change only the 10%
- group to 25%. If the flag does not exist, says so.
-
- # The description asks the agent to weigh an experiment before it creates a flag. The
- # pair below scores both directions: a user-visible change should route to an experiment,
- # an operational flag should not. A description that pushes experiments too hard fails the
- # second task. Both intents settle the choice up front, so a correct agent has a mutation
- # to make and tool selection scores cleanly.
- - id: flag-create-routes-to-experiment
- category: feature-flags
- intent: "We're putting a new version of our checkout page in front of 20% of users. Set that up — and if an
- experiment is the better way to tell whether it helped, set that up instead of a plain flag."
- expected_tools: [experiment-create]
- acceptable_tools: [feature-flag-get-all, experiment-list]
- success_criteria: >
- Creates an experiment rather than a bare feature flag, and says the experiment manages its own flag. Does
- not leave a separate feature flag behind alongside it.
-
- - id: flag-create-killswitch-no-experiment
- category: feature-flags
- intent:
- 'Add a kill switch flag called mcp-eval-billing-killswitch so we can turn off the billing sync if it starts
- failing.'
- expected_tools: [create-feature-flag]
- acceptable_tools: [feature-flag-get-definition-by-key, feature-flag-get-all]
- success_criteria: >
- Creates the flag and does not raise an experiment. An operational kill switch has no user-visible change
- to measure. The intent gives the exact key, so checking it with feature-flag-get-definition-by-key is the
- expected lookup; a miss there is data, not a failure.
-
- # --- session-replay ----------------------------------------------------------
- - id: replay-rage-clicks
- category: session-replay
- intent: 'Find session recordings from this week where users rage-clicked.'
- expected_tools: [query-session-recordings-list]
- success_criteria: 'Returns recordings filtered to the last week with rage-click activity.'
-
- # --- llm-analytics -------------------------------------------------------------
- - id: llma-recent-traces
- category: llm-analytics
- intent: 'Show me our latest LLM traces and how many tokens they used.'
- expected_tools: [query-llm-traces-list]
- success_criteria: 'Lists recent LLM traces including token usage.'
-
- # --- logs -----------------------------------------------------------------------
- - id: logs-error-severity
- category: logs
- intent: 'Are there any error-severity logs from our API service in the last hour?'
- expected_tools: [query-logs]
- success_criteria: 'Queries logs filtered by severity and service for the last hour.'
-
- # --- dashboards --------------------------------------------------------------------
- - id: dashboard-team-health
- category: dashboards
- intent: "Create a dashboard called 'Team health' from our existing activation insights."
- expected_tools: [dashboard-create]
- acceptable_tools: [execute-sql, insight-get]
- success_criteria: 'Reuses existing insights where they exist and creates the dashboard, returning its link.'
-
- # --- docs -------------------------------------------------------------------------
- - id: docs-behavioral-cohort
- category: docs
- intent: 'How do I create a behavioral cohort in PostHog?'
- expected_tools: [docs-search]
- success_criteria: 'Answers from documentation with source links, not from memory.'
-
- # --- project ------------------------------------------------------------------------
- - id: project-switch-then-flags
- category: project
- intent: 'Switch to our other project and list its feature flags.'
- expected_tools: [switch-project, feature-flag-get-all]
- success_criteria: 'Switches the active project before listing flags, and says which project the flags came from.'
-
- # --- mcp-analytics (dogfood discoverability) -------------------------------------------
- - id: mcp-tool-reliability
- category: mcp-analytics
- intent: 'How reliable has our execute-sql MCP tool been this week? I want error rate and p95 latency.'
- expected_tools: [query-mcp-tool-stats]
- acceptable_tools: [execute-sql, query-mcp-tool-daily-stats]
- success_criteria: 'Uses the MCP analytics tooling (not raw SQL) to report calls, error rate, and p95 for the tool.'
-
- - id: mcp-session-intents
- category: mcp-analytics
- intent: 'What have agents been trying to do with our MCP server lately?'
- expected_tools: [query-mcp-tool-sample-intents]
- acceptable_tools: [execute-sql, mcp-analytics-sessions-list]
- success_criteria: 'Surfaces recent agent intents from MCP analytics rather than guessing.'
-
- # --- metrics ----------------------------------------------------------------------------
- - id: metrics-api-error-rate
- category: metrics
- intent: 'What does the error-rate metric for our API service look like over the last day?'
- expected_tools: [metric-names-list, query-metrics]
- acceptable_tools: [execute-sql]
- success_criteria: 'Resolves the exact metric name first (names match exactly), then queries a rate over the last 24h — does not guess the metric name.'
-
- # --- data-catalog -----------------------------------------------------------------------
- # MCP-surface probe/agent tasks over the catalog. The end-to-end "does the agent trust the
- # catalog" behavioral suite (seeded approved/proposed/drifted/empty arms, scorers + LLM judge)
- # lives in products/data_catalog/evals/ (eval_governed_metrics, eval_semantic_discovery).
- - id: data-catalog-use-canonical-mrr
- category: data-catalog
- intent: 'What was our MRR last month?'
- expected_tools: [execute-sql]
- acceptable_tools: [data-catalog-metric-run]
- success_criteria: >
- Checks system.information_schema.metrics for an approved, non-drifted MRR metric before
- deriving the number. If one exists, runs it and cites it as the canonical definition; never
- presents a proposed or drifted metric as canonical.
- probe:
- tool: execute-sql
- args:
- query: "SELECT name, status, is_drifted, definition_kind FROM system.information_schema.metrics WHERE name ILIKE '%mrr%' OR description ILIKE '%recurring revenue%'"
-
- - id: data-catalog-check-before-deriving
- category: data-catalog
- intent: "What's our activation rate?"
- expected_tools: [execute-sql]
- success_criteria: >
- Looks in the catalog (system.information_schema.metrics) for a canonical activation metric
- before deriving one from scratch, and reports whether an approved one exists.
- probe:
- tool: execute-sql
- args:
- query: "SELECT name, status, is_drifted FROM system.information_schema.metrics WHERE name ILIKE '%activation%' OR description ILIKE '%activation%'"
-
- - id: data-catalog-prefer-certified-source
- category: data-catalog
- intent: 'Which table should I use to report revenue?'
- expected_tools: [execute-sql]
- success_criteria: >
- Uses the certification column on system.information_schema.tables to prefer a certified
- revenue source and avoid deprecated ones, rather than picking an arbitrary similar table.
- probe:
- tool: execute-sql
- args:
- query: 'SELECT table_name, certification, description FROM system.information_schema.tables WHERE certification IS NOT NULL'
-
- - id: data-catalog-never-cite-proposed
- category: data-catalog
- intent: 'Is there an official definition of churn I should use?'
- expected_tools: [execute-sql]
- success_criteria: >
- Reports whether an approved (not proposed or drifted) churn metric exists in the catalog, and
- does not present a proposed metric as if it were official.
- probe:
- tool: execute-sql
- args:
- query: "SELECT name, status, is_drifted FROM system.information_schema.metrics WHERE name ILIKE '%churn%' OR description ILIKE '%churn%'"
diff --git a/services/mcp/evals/runner/probe.ts b/services/mcp/evals/runner/probe.ts
deleted file mode 100644
index 561b79318165..000000000000
--- a/services/mcp/evals/runner/probe.ts
+++ /dev/null
@@ -1,137 +0,0 @@
-/**
- * Deterministic probe runner — no LLM. Connects to a LIVE MCP server,
- * verifies every required tool (the expected path + probe tool) is advertised,
- * executes each task's read-only probe call, and prints/writes a score summary.
- * Acceptable tools are optional alternatives (often feature-gated), so a missing
- * one is not a failure.
- *
- * Usage:
- * LIVE_MCP_URL=http://localhost:9876 LIVE_MCP_TOKEN=phx_... \
- * pnpm exec tsx evals/runner/probe.ts [--out score.json]
- *
- * Exit code is non-zero when any required tool is missing or any probe
- * fails, so the campaign (or CI) can gate on it directly.
- */
-
-import { Client } from '@modelcontextprotocol/sdk/client/index.js'
-import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js'
-import { ErrorCode, McpError } from '@modelcontextprotocol/sdk/types.js'
-import { writeFileSync } from 'node:fs'
-import process from 'node:process'
-
-import { loadBenchmark, referencedTools, requiredTools, type BenchmarkTask } from '../benchmark/schema'
-import { formatSummary, summarize, type ProbeResult, type ToolMiss } from './results'
-
-interface AdvertisedTool {
- name: string
- annotations?: { readOnlyHint?: boolean }
-}
-
-async function runProbe(
- client: Client,
- task: BenchmarkTask,
- advertised: Map<string, AdvertisedTool>
-): Promise<ProbeResult> {
- const probe = task.probe!
- const base = { task_id: task.id, tool: probe.tool }
- const tool = advertised.get(probe.tool)
- if (!tool) {
- return { ...base, status: 'skipped_not_advertised' }
- }
- // Enforce read-only from the LIVE advertisement, not just the fixture —
- // an annotation regression must fail the probe rather than mutate data.
- if (tool.annotations?.readOnlyHint !== true) {
- return { ...base, status: 'refused_not_read_only' }
- }
-
- const startedAt = Date.now()
- try {
- const result = await client.callTool({ name: probe.tool, arguments: probe.args }, undefined, {
- timeout: probe.max_ms,
- })
- const latency = Date.now() - startedAt
- if (result.isError) {
- const content = Array.isArray(result.content) ? result.content : []
- const text = content
- .map((item) => (item && typeof item === 'object' && 'text' in item ? String(item.text) : ''))
- .join(' ')
- return { ...base, status: 'tool_error', latency_ms: latency, error_snippet: text.slice(0, 200) }
- }
- return { ...base, status: 'ok', latency_ms: latency }
- } catch (error) {
- const latency = Date.now() - startedAt
- const message = error instanceof Error ? error.message : String(error)
- // The SDK raises a typed timeout when RequestOptions.timeout elapses —
- // classify on that, not on wall-clock heuristics.
- const isTimeout = error instanceof McpError && error.code === ErrorCode.RequestTimeout
- return {
- ...base,
- status: isTimeout ? 'timeout' : 'transport_error',
- latency_ms: latency,
- error_snippet: message.slice(0, 200),
- }
- }
-}
-
-async function main(): Promise<void> {
- const url = process.env.LIVE_MCP_URL ?? 'http://localhost:9876'
- const token = process.env.LIVE_MCP_TOKEN
- if (!token) {
- console.error('LIVE_MCP_TOKEN is required (a personal API key for the target instance)')
- process.exit(2)
- }
- const outFlagIndex = process.argv.indexOf('--out')
- const outPath = outFlagIndex === -1 ? null : (process.argv[outFlagIndex + 1] ?? null)
- if (outFlagIndex !== -1 && outPath === null) {
- console.error('--out requires a file path')
- process.exit(2)
- }
-
- const benchmark = loadBenchmark()
- const transport = new StreamableHTTPClientTransport(new URL('/mcp', url), {
- // Pin tools mode: the probe audits the full per-tool roster, and the
- // server's auto-detection would otherwise resolve this client to the
- // single-exec CLI default.
- requestInit: { headers: { Authorization: `Bearer ${token}`, 'x-posthog-mcp-mode': 'tools' } },
- })
- const client = new Client({ name: 'mcp-eval-probe', version: '0.0.0' }, { capabilities: {} })
- await client.connect(transport)
-
- try {
- const listed = await client.listTools()
- const advertised = new Map<string, AdvertisedTool>(
- listed.tools.map((tool) => [tool.name, tool as AdvertisedTool])
- )
-
- const toolMisses: ToolMiss[] = benchmark.tasks.flatMap((task) =>
- requiredTools(task)
- .filter((tool) => !advertised.has(tool))
- .map((tool) => ({ task_id: task.id, tool }))
- )
-
- const probes: ProbeResult[] = []
- for (const task of benchmark.tasks.filter((candidate) => candidate.probe)) {
- probes.push(await runProbe(client, task, advertised))
- }
-
- const summary = summarize({
- benchmarkVersion: benchmark.version,
- tasksTotal: benchmark.tasks.length,
- toolsReferenced: new Set(benchmark.tasks.flatMap(referencedTools)).size,
- toolMisses,
- probes,
- })
-
- // stdout directly: the summary IS the program output (oxlint strips console.log).
- process.stdout.write(formatSummary(summary) + '\n')
- if (outPath) {
- writeFileSync(outPath, JSON.stringify(summary, null, 2))
- process.stdout.write(`wrote ${outPath}\n`)
- }
- process.exit(summary.tool_misses.length > 0 || summary.probes_failed > 0 ? 1 : 0)
- } finally {
- await client.close().catch(() => undefined)
- }
-}
-
-void main()
diff --git a/services/mcp/evals/runner/results.ts b/services/mcp/evals/runner/results.ts
deleted file mode 100644
index 7dd15da4cd61..000000000000
--- a/services/mcp/evals/runner/results.ts
+++ /dev/null
@@ -1,94 +0,0 @@
-/**
- * Pure result types + aggregation for eval runs. Kept free of I/O so the
- * scoring math is unit-testable — campaign keep/discard decisions ride on
- * these numbers.
- */
-
-export type ProbeStatus =
- | 'ok'
- | 'tool_error' // server answered with isError: true
- | 'transport_error' // request failed or threw
- | 'timeout'
- | 'refused_not_read_only' // advertised annotations don't mark the tool read-only
- | 'skipped_not_advertised'
-
-export interface ToolMiss {
- task_id: string
- tool: string
-}
-
-export interface ProbeResult {
- task_id: string
- tool: string
- status: ProbeStatus
- latency_ms?: number
- error_snippet?: string
-}
-
-export interface ProbeRunSummary {
- benchmark_version: number
- tasks_total: number
- tools_referenced: number
- /** Referenced tools the server did not advertise — discoverability misses. */
- tool_misses: ToolMiss[]
- probes_total: number
- probes_ok: number
- probes_failed: number
- latency_p50_ms: number | null
- latency_p95_ms: number | null
- probes: ProbeResult[]
-}
-
-export function percentile(sortedValues: number[], p: number): number | null {
- if (sortedValues.length === 0) {
- return null
- }
- const rank = Math.min(sortedValues.length - 1, Math.max(0, Math.ceil((p / 100) * sortedValues.length) - 1))
- return sortedValues[rank] ?? null
-}
-
-export function summarize(input: {
- benchmarkVersion: number
- tasksTotal: number
- toolsReferenced: number
- toolMisses: ToolMiss[]
- probes: ProbeResult[]
-}): ProbeRunSummary {
- const latencies = input.probes
- .map((probe) => probe.latency_ms)
- .filter((value): value is number => typeof value === 'number')
- .sort((a, b) => a - b)
- const ok = input.probes.filter((probe) => probe.status === 'ok').length
-
- return {
- benchmark_version: input.benchmarkVersion,
- tasks_total: input.tasksTotal,
- tools_referenced: input.toolsReferenced,
- tool_misses: input.toolMisses,
- probes_total: input.probes.length,
- probes_ok: ok,
- probes_failed: input.probes.length - ok,
- latency_p50_ms: percentile(latencies, 50),
- latency_p95_ms: percentile(latencies, 95),
- probes: input.probes,
- }
-}
-
-export function formatSummary(summary: ProbeRunSummary): string {
- const lines = [
- `benchmark v${summary.benchmark_version}: ${summary.tasks_total} tasks, ${summary.tools_referenced} referenced tools`,
- `tool presence: ${summary.tool_misses.length === 0 ? 'all advertised' : `${summary.tool_misses.length} MISSING`}`,
- ...summary.tool_misses.map((miss) => ` MISSING ${miss.tool} (task ${miss.task_id})`),
- `probes: ${summary.probes_ok}/${summary.probes_total} ok` +
- (summary.latency_p50_ms !== null
- ? ` (p50 ${summary.latency_p50_ms}ms, p95 ${summary.latency_p95_ms}ms)`
- : ''),
- ...summary.probes
- .filter((probe) => probe.status !== 'ok')
- .map(
- (probe) =>
- ` ${probe.status.toUpperCase()} ${probe.tool} (task ${probe.task_id})${probe.error_snippet ? `: ${probe.error_snippet}` : ''}`
- ),
- ]
- return lines.join('\n')
-}
diff --git a/services/mcp/evals/runner/seed.ts b/services/mcp/evals/runner/seed.ts
deleted file mode 100644
index 7a1a9692ca3f..000000000000
--- a/services/mcp/evals/runner/seed.ts
+++ /dev/null
@@ -1,235 +0,0 @@
-/**
- * Fixture seeder — no LLM, no MCP. Writes the benchmark's `fixtures` block straight
- * to the PostHog API so an agent-mode run starts from the state its tasks describe.
- *
- * Run it before every agent-mode run. Without it the feature-flag tasks score tool
- * selection only: the flags they name do not exist, so the state change half of each
- * `success_criteria` cannot be judged. Run it again after a run to reset. It is
- * idempotent over the keys it owns: every flag in `fixtures.feature_flags` is rewritten
- * and every key in `fixtures.absent_feature_flags` is cleared, so those tasks start a
- * second run where they started the first.
- *
- * `flag-create-routes-to-experiment` is the exception. The agent picks the key of the
- * flag its experiment manages, so the seeder cannot name that key in
- * `absent_feature_flags`, and that task's experiment and flag accumulate across runs.
- *
- * It talks to the REST API rather than to the MCP server on purpose: the tools under
- * test must not also be the thing that builds the state they are measured against.
- *
- * Usage:
- * LIVE_POSTHOG_URL=https://us.posthog.com LIVE_MCP_TOKEN=phx_... \
- * pnpm exec tsx evals/runner/seed.ts [--project 12345]
- *
- * `LIVE_MCP_TOKEN` is the same personal API key the probe uses; it needs
- * `feature_flag:write`, plus `user:read` when `--project` is omitted, because the project
- * is then resolved through `/api/users/@me/`. Without `--project` the project comes from the token's current
- * project — pass it explicitly when the MCP session points somewhere else, because
- * seeding one project and scoring another looks exactly like a tool-selection
- * regression.
- *
- * Exit code is non-zero when any fixture could not be written, so CI can gate on it.
- */
-
-import process from 'node:process'
-import { parseArgs } from 'node:util'
-
-import { type BenchmarkFlagFixture, FIXTURE_TAG, loadBenchmark } from '../benchmark/schema'
-
-/** A flag as the list endpoint returns it, narrowed to what the seeder reads. */
-interface ListedFlag {
- id: number
- key: string
-}
-
-interface SeedOutcome {
- key: string
- action: 'created' | 'updated' | 'cleared' | 'already-absent'
-}
-
-class SeedError extends Error {
- constructor(
- public readonly key: string,
- message: string
- ) {
- super(message)
- this.name = 'SeedError'
- }
-}
-
-class PostHogApi {
- constructor(
- private readonly baseUrl: string,
- private readonly token: string
- ) {}
-
- async request<T>(method: string, path: string, body?: unknown): Promise<T> {
- const response = await fetch(new URL(path, this.baseUrl), {
- method,
- headers: {
- Authorization: `Bearer ${this.token}`,
- ...(body === undefined ? {} : { 'Content-Type': 'application/json' }),
- },
- ...(body === undefined ? {} : { body: JSON.stringify(body) }),
- })
- const text = await response.text()
- if (!response.ok) {
- throw new Error(`${method} ${path} → ${response.status} ${response.statusText}: ${text.slice(0, 400)}`)
- }
- return (text ? JSON.parse(text) : {}) as T
- }
-}
-
-/** The flag state a fixture declares, in the shape the feature-flags API accepts. */
-function fixtureBody(fixture: BenchmarkFlagFixture): Record<string, unknown> {
- return {
- key: fixture.key,
- name: fixture.name,
- active: fixture.active,
- archived: fixture.archived,
- filters: fixture.filters,
- // A project can require a tag on every new flag, which would otherwise reject
- // the create. The tag also marks the flag as eval-owned in the flag list.
- tags: [FIXTURE_TAG],
- }
-}
-
-/**
- * The fixture flag with this key, whatever state it is in.
- *
- * Two calls because the list endpoint hides archived flags unless `archived` is passed,
- * and passing it narrows to archived flags only — neither call alone sees both.
- */
-async function findFlag(api: PostHogApi, projectId: number, key: string): Promise<ListedFlag | undefined> {
- const search = `search=${encodeURIComponent(key)}`
- for (const query of [search, `${search}&archived=true`]) {
- const page = await api.request<{ results: ListedFlag[] }>(
- 'GET',
- `/api/projects/${projectId}/feature_flags/?${query}`
- )
- // `search` is a substring match, so pin the exact key rather than taking the first hit.
- const exact = page.results.find((flag) => flag.key === key)
- if (exact) {
- return exact
- }
- }
- return undefined
-}
-
-async function seedFlag(api: PostHogApi, projectId: number, fixture: BenchmarkFlagFixture): Promise<SeedOutcome> {
- const existing = await findFlag(api, projectId, fixture.key)
- const body = fixtureBody(fixture)
- if (existing) {
- await api.request('PATCH', `/api/projects/${projectId}/feature_flags/${existing.id}/`, body)
- return { key: fixture.key, action: 'updated' }
- }
- await api.request('POST', `/api/projects/${projectId}/feature_flags/`, body)
- return { key: fixture.key, action: 'created' }
-}
-
-async function clearFlag(api: PostHogApi, projectId: number, key: string): Promise<SeedOutcome> {
- const existing = await findFlag(api, projectId, key)
- if (!existing) {
- return { key, action: 'already-absent' }
- }
- // Soft delete, the same write the `delete-feature-flag` tool makes.
- await api.request('PATCH', `/api/projects/${projectId}/feature_flags/${existing.id}/`, { deleted: true })
- return { key, action: 'cleared' }
-}
-
-async function resolveProjectId(api: PostHogApi, override: number | null): Promise<number> {
- if (override !== null) {
- return override
- }
- const me = await api.request<{ team?: { id?: number } }>('GET', '/api/users/@me/')
- const id = me.team?.id
- if (typeof id !== 'number') {
- throw new Error('could not resolve a current project from /api/users/@me/ — pass --project <id>')
- }
- return id
-}
-
-const PROJECT_FLAG = '--project'
-
-/** Reads `--project <id>` or `--project=<id>`, and rejects anything else.
- *
- * Strict because this command writes. An argument it did not understand would fall
- * through to the token's current project, so a typo would seed a project nobody named.
- */
-function parseProjectFlag(args: string[]): number | null {
- // `strict` rejects an unknown argument, a stray positional and a missing value, so a
- // typo stops the run instead of falling through to the token's current project. The
- // seeder writes six flags and soft-deletes one, so the wrong project is not
- // recoverable by rerunning it.
- const { values } = parseArgs({
- args,
- options: { project: { type: 'string', multiple: true } },
- strict: true,
- allowPositionals: false,
- })
- const given = values.project ?? []
- if (given.length === 0) {
- return null
- }
- if (given.length > 1) {
- throw new Error(`${PROJECT_FLAG} was given ${given.length} times`)
- }
- const id = Number(given[0])
- if (!given[0] || !Number.isInteger(id) || id <= 0) {
- throw new Error(`${PROJECT_FLAG} requires a positive integer project id`)
- }
- return id
-}
-
-async function main(): Promise<void> {
- const token = process.env.LIVE_MCP_TOKEN
- if (!token) {
- console.error(
- 'LIVE_MCP_TOKEN is required (a personal API key with feature_flag:write, and user:read unless you pass --project)'
- )
- process.exit(2)
- }
-
- let projectOverride: number | null
- try {
- projectOverride = parseProjectFlag(process.argv.slice(2))
- } catch (error) {
- console.error(error instanceof Error ? error.message : String(error))
- process.exit(2)
- return
- }
-
- const api = new PostHogApi(process.env.LIVE_POSTHOG_URL ?? 'http://localhost:8000', token)
- const { fixtures } = loadBenchmark()
- const projectId = await resolveProjectId(api, projectOverride)
-
- const outcomes: SeedOutcome[] = []
- const failures: SeedError[] = []
- // Sequential: the fixture set is small, and one flag at a time keeps a failure
- // attributable to its own key instead of to whichever request lost a race.
- for (const fixture of fixtures.feature_flags) {
- try {
- outcomes.push(await seedFlag(api, projectId, fixture))
- } catch (error) {
- failures.push(new SeedError(fixture.key, error instanceof Error ? error.message : String(error)))
- }
- }
- for (const key of fixtures.absent_feature_flags) {
- try {
- outcomes.push(await clearFlag(api, projectId, key))
- } catch (error) {
- failures.push(new SeedError(key, error instanceof Error ? error.message : String(error)))
- }
- }
-
- // stdout directly: the summary IS the program output (oxlint strips console.log).
- process.stdout.write(`seeded project ${projectId}\n`)
- for (const outcome of outcomes) {
- process.stdout.write(` ${outcome.action} ${outcome.key}\n`)
- }
- for (const failure of failures) {
- process.stdout.write(` FAILED ${failure.key}: ${failure.message}\n`)
- }
- process.exit(failures.length > 0 ? 1 : 0)
-}
-
-void main()
diff --git a/services/mcp/tests/evals/benchmark.test.ts b/services/mcp/tests/evals/benchmark.test.ts
deleted file mode 100644
index fa28b04b5c75..000000000000
--- a/services/mcp/tests/evals/benchmark.test.ts
+++ /dev/null
@@ -1,132 +0,0 @@
-import { describe, expect, it } from 'vitest'
-
-import { getToolDefinitions } from '@/tools/toolDefinitions'
-
-import {
- BenchmarkFileSchema,
- TASK_CATEGORIES,
- fixtureFlagKeys,
- fixtureFlagKeysInIntent,
- loadBenchmark,
- referencedTools,
-} from '../../evals/benchmark/schema'
-
-const FLAG_FIXTURE = {
- key: 'mcp-eval-enable-me',
- name: 'Enable target',
- active: false,
- archived: false,
-}
-
-const FIXTURES_BLOCK = { feature_flags: [FLAG_FIXTURE], absent_feature_flags: ['mcp-eval-create-me'] }
-
-const TASK = {
- id: 'enable-a-flag',
- category: 'feature-flags',
- intent: 'Turn the mcp-eval-enable-me flag on.',
- expected_tools: ['feature-flag-enable'],
- success_criteria: 'The flag is enabled and the reply says so.',
-}
-
-describe('MCP eval benchmark fixtures', () => {
- // Loaded inside each test (not at describe scope) so a broken fixture or
- // catalog fails the owning test with a real error instead of blowing up
- // vitest's collection phase.
- it('parses against the schema with unique task ids', () => {
- const benchmark = loadBenchmark()
- const ids = benchmark.tasks.map((task) => task.id)
- expect(new Set(ids).size).toBe(ids.length)
- })
-
- it('only references tools that exist in the catalog', () => {
- const known = new Set(Object.keys(getToolDefinitions()))
- const unknown = loadBenchmark().tasks.flatMap((task) =>
- referencedTools(task)
- .filter((tool) => !known.has(tool))
- .map((tool) => `${task.id} → ${tool}`)
- )
- expect(unknown).toEqual([])
- })
-
- it('uses every declared task category', () => {
- const used = new Set(loadBenchmark().tasks.map((task) => task.category))
- const unused = TASK_CATEGORIES.filter((category) => !used.has(category))
- expect(unused).toEqual([])
- })
-
- it('only probes read-only tools', () => {
- const catalog = getToolDefinitions()
- const unsafe = loadBenchmark()
- .tasks.filter((task) => task.probe)
- .filter((task) => catalog[task.probe!.tool]?.annotations?.readOnlyHint !== true)
- .map((task) => `${task.id} → ${task.probe!.tool}`)
- expect(unsafe).toEqual([])
- })
-
- // Both halves of the seeded-fixture contract in one assertion. Zero tasks means the
- // seeder writes state nothing reads, which is how a renamed key silently stops being
- // exercised. Two or more means those tasks share a flag, which is the order-dependence
- // v2 removed: whichever ran first decides what the next one starts from.
- it('names every seeded flag fixture in exactly one task intent', () => {
- const benchmark = loadBenchmark()
- const tasksByKey = new Map(fixtureFlagKeys(benchmark.fixtures).map((key) => [key, [] as string[]]))
- for (const task of benchmark.tasks) {
- for (const key of fixtureFlagKeysInIntent(task, benchmark.fixtures)) {
- tasksByKey.get(key)!.push(task.id)
- }
- }
- const wrong = [...tasksByKey.entries()]
- .filter(([, taskIds]) => taskIds.length !== 1)
- .map(([key, taskIds]) => `${key} → ${taskIds.length ? taskIds.join(', ') : 'no task'}`)
- expect(wrong).toEqual([])
- })
-
- // Every fixture container defaults to empty, so a stripped key used to read as "no
- // fixtures" rather than as a typo: the seeder wrote nothing, exited 0, and the test
- // above passed over an empty map while the next run scored the lifecycle tasks
- // against flags that did not exist.
- it.each([
- { name: 'the fixtures block spelled correctly', fixtures: FIXTURES_BLOCK, parses: true },
- { name: 'a misspelled fixture list', fixtures: { feature_flag: [FLAG_FIXTURE] }, parses: false },
- {
- name: 'a misspelled field on a fixture flag',
- fixtures: { feature_flags: [{ ...FLAG_FIXTURE, activee: true }] },
- parses: false,
- },
- // `clearFlag` soft-deletes whatever key it is handed, so a fixture key without
- // the prefix would delete one of the project's real flags rather than a fixture.
- {
- name: 'a fixture key missing the mcp-eval- prefix',
- fixtures: { feature_flags: [{ ...FLAG_FIXTURE, key: 'billing-killswitch' }] },
- parses: false,
- },
- {
- name: 'an absent-flag key missing the mcp-eval- prefix',
- fixtures: { ...FIXTURES_BLOCK, absent_feature_flags: ['billing-killswitch'] },
- parses: false,
- },
- // The API disables a flag on the way to archiving it, so the seeder cannot leave
- // a flag in both states and a fixture asking for both describes nothing.
- {
- name: 'a fixture that is both archived and active',
- fixtures: { feature_flags: [{ ...FLAG_FIXTURE, active: true, archived: true }] },
- parses: false,
- },
- ])('the schema accepts $name: $parses', ({ fixtures, parses }) => {
- expect(BenchmarkFileSchema.safeParse({ version: 2, fixtures, tasks: [TASK] }).success).toBe(parses)
- })
-
- it('rejects a misspelled fixtures block', () => {
- const parsed = BenchmarkFileSchema.safeParse({ version: 2, fixtuers: FIXTURES_BLOCK, tasks: [TASK] })
-
- expect(parsed.success).toBe(false)
- })
-
- it('probes exercise a tool the task expects', () => {
- const mismatched = loadBenchmark()
- .tasks.filter((task) => task.probe)
- .filter((task) => ![...task.expected_tools, ...task.acceptable_tools].includes(task.probe!.tool))
- .map((task) => task.id)
- expect(mismatched).toEqual([])
- })
-})
diff --git a/services/mcp/tests/evals/results.test.ts b/services/mcp/tests/evals/results.test.ts
deleted file mode 100644
index 23136d0aa0a2..000000000000
--- a/services/mcp/tests/evals/results.test.ts
+++ /dev/null
@@ -1,40 +0,0 @@
-import { describe, expect, it } from 'vitest'
-
-import { percentile, summarize, type ProbeResult } from '../../evals/runner/results'
-
-describe('eval result aggregation', () => {
- it.each([
- { values: [], p: 95, expected: null },
- { values: [100], p: 50, expected: 100 },
- { values: [100], p: 95, expected: 100 },
- { values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], p: 50, expected: 5 },
- { values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], p: 95, expected: 10 },
- { values: [1, 2, 3, 4], p: 95, expected: 4 },
- ])('percentile(p$p) of $values is $expected', ({ values, p, expected }) => {
- expect(percentile(values, p)).toBe(expected)
- })
-
- it('counts every non-ok status as failed and keeps latency math to ok+errored calls only', () => {
- const probes: ProbeResult[] = [
- { task_id: 'a', tool: 't1', status: 'ok', latency_ms: 100 },
- { task_id: 'b', tool: 't2', status: 'tool_error', latency_ms: 300 },
- { task_id: 'c', tool: 't3', status: 'refused_not_read_only' },
- { task_id: 'd', tool: 't4', status: 'skipped_not_advertised' },
- ]
-
- const summary = summarize({
- benchmarkVersion: 0,
- tasksTotal: 4,
- toolsReferenced: 4,
- toolMisses: [{ task_id: 'd', tool: 't4' }],
- probes,
- })
-
- expect(summary.probes_total).toBe(4)
- expect(summary.probes_ok).toBe(1)
- expect(summary.probes_failed).toBe(3)
- expect(summary.latency_p50_ms).toBe(100)
- expect(summary.latency_p95_ms).toBe(300)
- expect(summary.tool_misses).toHaveLength(1)
- })
-})