:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(mcp): remove benchmark evals (#118775)
Radu Raicea34d9c3fc8df4
diff --git a/.agents/skills/improving-mcp-tools/SKILL.md b/.agents/skills/improving-mcp-tools/SKILL.mdindex 6e1729e58951..d99a2ff7a3d1 100644--- a/.agents/skills/improving-mcp-tools/SKILL.md+++ b/.agents/skills/improving-mcp-tools/SKILL.md@@ -2,7 +2,7 @@name: improving-mcp-toolsdescription: >Run an improve-my-MCP campaign: an autoresearch-style loop that measures the- MCP agent experience with the eval harness, picks the highest-impact tool+ MCP agent experience with a supplied eval harness, picks the highest-impact toolproblem from production data, makes one bounded fix, and keeps it only ifbefore/after scores improve. Use when asked to "improve my MCP", run an MCPimprovement campaign, fix tool discoverability or descriptions based on@@ -21,22 +21,17 @@ repeating work.## The objective function-`services/mcp/evals/` is the harness. `benchmark/tasks.yaml` is a fixed set of-agent tasks with `expected_tools` and `success_criteria`; scores are only-comparable across runs of the same benchmark `version`.+This repository does not include an MCP benchmark or eval runner.+Before starting a campaign, obtain a benchmark and runnable harness from the operator.+Record the task set, expected tools, success criteria, benchmark version, and exact commands in the journal.+If those inputs are unavailable, report the missing evidence and stop the campaign.-- **Probe mode** (deterministic, no LLM):- `LIVE_MCP_URL=... LIVE_MCP_TOKEN=... pnpm exec tsx evals/runner/probe.ts --out score.json`- from `services/mcp/`. Reports tool-presence misses (discoverability), probe- failures, and latency p50/p95. Non-zero exit = regression.-- **Agent mode** (LLM replay + judge): scores task success and tool-selection- accuracy. Use it for description/discoverability changes — probes cannot- detect that an agent picks the wrong tool.+- **Probe mode** (deterministic, no LLM): measures tool-presence misses, probe failures, and latency p50/p95.+- **Agent mode** (LLM replay + judge): scores task success and tool-selection accuracy.+ Use it for description/discoverability changes, since probes cannot detect that an agent picks the wrong tool.-Run the harness against a **seeded local or devbox stack**, never against a-customer project. Local recipe: `NODE_ENV=development PORT=9876-POSTHOG_API_BASE_URL=http://localhost:8000 pnpm dev:hono`, personal API key as-`LIVE_MCP_TOKEN`.+Run the supplied harness against a seeded local or devbox stack.+Use the same benchmark version for baseline and after scores.## One iteration@@ -69,7 +64,7 @@ POSTHOG_API_BASE_URL=http://localhost:8000 pnpm dev:hono`, personal API key asThese are not suggestions; violating any of them ends the campaign pass.- **Allowlist** — a campaign PR may only touch: `products/*/mcp/tools.yaml`,- `products/*/skills/**`, `services/mcp/evals/**`, the codegen outputs of+ `products/*/skills/**`, the codegen outputs of`pnpm generate-tools` / `scaffold-yaml` (`services/mcp/src/tools/generated/**`and `services/mcp/schema/generated-tool-definitions.json`), and docs.Anything else (handler code, package manifests, workflows, migrations, auth@@ -80,7 +75,7 @@ These are not suggestions; violating any of them ends the campaign pass.measuring.- **Evidence or it didn't happen.** No PR without a baseline score, an afterscore, and the exact harness commands used.-- **Benchmark integrity.** Never edit `benchmark/tasks.yaml` in the same PR as+- **Benchmark integrity.** Never edit the supplied benchmark in the same PR asa fix it validates — changing the exam and the answer together provesnothing. Benchmark changes are their own PR and bump `version`.- **Budgets.** Respect the operator's iteration/token/PR caps (default: stopdiff --git a/services/mcp/evals/README.md b/services/mcp/evals/README.mddeleted file mode 100644index 954ca5a622ef..000000000000--- a/services/mcp/evals/README.md+++ /dev/null@@ -1,76 +0,0 @@-# MCP evals--A benchmark of agent tasks for the PostHog MCP server, a script that runs each task's probe call against a live server, and a script that seeds the feature flags some tasks need.--## Layout--- `benchmark/tasks.yaml` — the task set, plus a `fixtures` block of feature flags that some tasks act on.-- `benchmark/schema.ts` — the zod schema and loader for `tasks.yaml`.-- `runner/probe.ts` — runs each task's `probe` against a live MCP server. See [Probe runner](#probe-runner).-- `runner/results.ts` — the result types and summary for a probe run.-- `runner/seed.ts` — writes the fixture flags to a PostHog project. See [Seeding](#seeding).--`tests/evals/benchmark.test.ts` runs in CI and checks `tasks.yaml`. Among other checks, it fails when a task names a tool that is not in the MCP tool catalog, when a probe uses a tool that is not read-only, or when a fixture flag is named in no task intent or in more than one.--## Task format--```yaml-- id: flags-list-active # kebab-case, unique- category: feature-flags # one of TASK_CATEGORIES in schema.ts- intent: 'List all our active feature flags.' # the request, phrased like a real user's- expected_tools: [feature-flag-get-all] # tools a competent agent should call- acceptable_tools: [execute-sql] # other tools that are also a reasonable path- success_criteria: "Returns the project's active flags by key." # what a correct answer does- probe: # optional: one read-only call the probe runner makes- tool: feature-flag-get-all- args: {}- max_ms: 15000-```--The probe runner checks `expected_tools` and runs `probe`. No script here scores `intent` or `success_criteria`.--## Probe runner--Run from `services/mcp/`:--```bash-LIVE_MCP_URL=http://localhost:9876 LIVE_MCP_TOKEN=phx_... \- pnpm exec tsx evals/runner/probe.ts [--out score.json]-```--It connects in tools mode, then:--1. Lists the tools the server advertises, and reports each task's `expected_tools` or `probe` tool that is missing.-2. Runs each task's `probe` call and records its status and latency.- It refuses a probe whose tool the server does not advertise as read-only, so a bad fixture cannot change project data.-3. Prints a summary: missing tools, probes passed and failed, and latency p50/p95. `--out` also writes it as JSON.--The exit code is non-zero when a tool is missing or a probe fails.--## Seeding--Some tasks ask an agent to create or change feature flags.-The `fixtures` block lists those flags, and `runner/seed.ts` puts them in a known state.-Run from `services/mcp/`:--```bash-LIVE_POSTHOG_URL=http://localhost:8000 LIVE_MCP_TOKEN=phx_... \- pnpm exec tsx evals/runner/seed.ts [--project 12345]-```--- It creates or updates every flag in `fixtures.feature_flags`, and soft-deletes every key in `fixtures.absent_feature_flags`. Running it again resets those flags.-- `flag-create-routes-to-experiment` is the exception. The agent picks the key of the flag its experiment manages, so the seeder cannot reset it. Delete that experiment and flag by hand when you need a clean project.-- It writes through the REST API, not the MCP server, so the tools under test do not build their own starting state.-- Without `--project` it seeds the token's current project. Seed the project the MCP session acts on.-- The token needs `feature_flag:write`, plus `user:read` when `--project` is omitted.-- The exit code is non-zero when any fixture could not be written.--Probes are read-only, so the probe runner needs no seeding.--## Authoring rules--- Sample intents from real usage (`$mcp_intent` clusters, `query-mcp-tool-sample-intents`) but **paraphrase — never paste customer text verbatim**, and never include PII.-- Results are only comparable across runs of the same benchmark `version`. Bump it on breaking changes to the set, together with the version literal in `schema.ts`.-- Weight new tasks toward observed pain: high-error tools, discoverability misses (intents where agents picked the wrong tool), and multi-step chains (resolve id → act).-- Give each task that changes a flag its own fixture flag, so the result does not depend on task order. The fixture test enforces this.-- A task must be achievable in a seeded demo project — don't write tasks that depend on one specific production dataset.diff --git a/services/mcp/evals/benchmark/schema.ts b/services/mcp/evals/benchmark/schema.tsdeleted file mode 100644index b2d3a86bd3d1..000000000000--- a/services/mcp/evals/benchmark/schema.ts+++ /dev/null@@ -1,143 +0,0 @@-import { readFileSync } from 'node:fs'-import { fileURLToPath } from 'node:url'-import { parse as parseYaml } from 'yaml'-import { z } from 'zod'--/**- * Benchmark task file schema — the objective function for the MCP agent- * experience. Tasks are sampled from real agent intents (paraphrased, never- * verbatim) and describe what a competent agent should achieve against a- * live MCP server, which tools it is expected to reach for, and optionally a- * deterministic probe call the runner can execute without an LLM.- */--export const TASK_CATEGORIES = [- 'sql',- 'data-schema',- 'product-analytics',- 'insights',- 'error-tracking',- 'feature-flags',- 'session-replay',- 'llm-analytics',- 'logs',- 'dashboards',- 'docs',- 'project',- 'mcp-analytics',- 'metrics',- 'data-catalog',-] as const--export const BenchmarkProbeSchema = z.object({- // Probes must reference read-only tools; the runner refuses to execute a- // probe whose tool lacks readOnlyHint so a bad fixture can't mutate data.- tool: z.string().min(1),- args: z.record(z.string(), z.unknown()).default({}),- max_ms: z.number().int().positive().default(15_000),-})--export const BenchmarkTaskSchema = z.object({- id: z- .string()- .regex(/^[a-z0-9][a-z0-9-]+$/, 'task ids are kebab-case')- .max(64),- category: z.enum(TASK_CATEGORIES),- // The user-visible goal the agent is given, written like a real request.- intent: z.string().min(10),- // Tools a competent agent is expected to call to complete the task.- expected_tools: z.array(z.string().min(1)).min(1),- // Tools that also count as a reasonable path (no tool-selection penalty).- acceptable_tools: z.array(z.string().min(1)).default([]),- // Plain-language pass condition consumed by the agent-mode judge.- success_criteria: z.string().min(10),- probe: BenchmarkProbeSchema.optional(),-})--/** Prefix every seeded entity carries, so a fixture is never mistaken for real project data. */-export const FIXTURE_KEY_PREFIX = 'mcp-eval-'--/** Tag the seeder puts on every fixture flag, so a project's own flags stay distinguishable. */-export const FIXTURE_TAG = 'mcp-eval'--/** Both fixture lists name keys the same seeder owns, so they validate through one schema. */-const FixtureFlagKeySchema = z- .string()- .regex(new RegExp(`^${FIXTURE_KEY_PREFIX}[a-z0-9-]+$`), `flag fixture keys start with ${FIXTURE_KEY_PREFIX}`)- .max(200)--export const BenchmarkFlagFixtureSchema = z- .object({- key: FixtureFlagKeySchema,- name: z.string().min(1),- active: z.boolean(),- archived: z.boolean(),- // Release conditions, variants and payloads, in the API's own shape. Tasks that assert- // an edit preserved unrelated configuration need something here to preserve.- filters: z.record(z.string(), z.unknown()).default({}),- })- .strict()- // The API disables a flag on the way to archiving it, so a fixture asking for both would- // describe a state the seeder cannot leave the flag in.- .refine((fixture) => !(fixture.archived && fixture.active), 'an archived fixture flag cannot also be active')--// Strict all the way down, because every fixture container defaults to empty. A misspelled-// key would otherwise be stripped, leaving the seeder nothing to write and no error to say so.-export const BenchmarkFixturesSchema = z- .object({- feature_flags: z.array(BenchmarkFlagFixtureSchema).default([]),- // Keys a task is asked to create. The seeder soft-deletes each one, so the create- // path is exercised rather than a second run hitting an already-taken key.- absent_feature_flags: z.array(FixtureFlagKeySchema).default([]),- })- .strict()--export const BenchmarkFileSchema = z- .object({- version: z.literal(2),- // Entities `runner/seed.ts` creates and resets before an agent-mode run, so tasks that- // mutate state start from a known place and score the same on a second run.- fixtures: BenchmarkFixturesSchema.default({ feature_flags: [], absent_feature_flags: [] }),- tasks: z.array(BenchmarkTaskSchema).min(1),- })- .strict()--export type BenchmarkProbe = z.infer<typeof BenchmarkProbeSchema>-export type BenchmarkTask = z.infer<typeof BenchmarkTaskSchema>-export type BenchmarkFlagFixture = z.infer<typeof BenchmarkFlagFixtureSchema>-export type BenchmarkFixtures = z.infer<typeof BenchmarkFixturesSchema>-export type BenchmarkFile = z.infer<typeof BenchmarkFileSchema>--export const DEFAULT_BENCHMARK_PATH = fileURLToPath(new URL('./tasks.yaml', import.meta.url))--export function loadBenchmark(path: string = DEFAULT_BENCHMARK_PATH): BenchmarkFile {- return BenchmarkFileSchema.parse(parseYaml(readFileSync(path, 'utf-8')))-}--/**- * Fixture flag keys a task's intent names. Task independence rides on this: two tasks that- * mutate the same fixture flag make the run order-dependent and stop a second run from- * starting where the first one did, so the fixture test asserts no key is named twice.- */-export function fixtureFlagKeysInIntent(task: BenchmarkTask, fixtures: BenchmarkFixtures): string[] {- return fixtureFlagKeys(fixtures).filter((key) => task.intent.includes(key))-}--/** Every flag key the seeder owns — the ones it writes and the ones it clears. */-export function fixtureFlagKeys(fixtures: BenchmarkFixtures): string[] {- return [...fixtures.feature_flags.map((fixture) => fixture.key), ...fixtures.absent_feature_flags]-}--export function referencedTools(task: BenchmarkTask): string[] {- return [...new Set([...task.expected_tools, ...task.acceptable_tools, ...(task.probe ? [task.probe.tool] : [])])]-}--/**- * Tools that MUST be advertised for a task to be runnable: the expected path- * plus any deterministic probe tool. `acceptable_tools` are optional- * alternatives — often feature-gated — so a missing one is not a benchmark- * failure and is deliberately excluded here.- */-export function requiredTools(task: BenchmarkTask): string[] {- return [...new Set([...task.expected_tools, ...(task.probe ? [task.probe.tool] : [])])]-}diff --git a/services/mcp/evals/benchmark/tasks.yaml b/services/mcp/evals/benchmark/tasks.yamldeleted file mode 100644index f822d33af8d4..000000000000--- a/services/mcp/evals/benchmark/tasks.yaml+++ /dev/null@@ -1,518 +0,0 @@-# MCP agent-experience benchmark v2.-#-# Task intents are paraphrased from real agent usage (top tool families by-# production volume) — never verbatim copies of customer text. Keep probes-# read-only; the runner refuses non-read-only probe tools.-#-# v2 gave every feature-flag task its own fixture flag and added the fixtures-# below, so flag scores are no longer order-dependent. Scores are comparable-# within a version, not across one.-version: 2--# Entities `runner/seed.ts` writes to the target project before an agent-mode run.-# One fixture per task that mutates it: a task run twice, or a whole run repeated,-# then starts from the state declared here rather than from what the last run left.-fixtures:- feature_flags:- - key: mcp-eval-enable-me- name: MCP eval — enable target- active: false- archived: false- filters:- groups:- - properties: []- rollout_percentage: 100-- - key: mcp-eval-disable-me- name: MCP eval — disable target- active: true- archived: false- filters:- groups:- - properties: []- rollout_percentage: 100-- - key: mcp-eval-archive-me- name: MCP eval — archive target- # Enabled on purpose: archiving has to turn the flag off too, and the task- # scores whether the agent says so.- active: true- archived: false- filters:- groups:- - properties: []- rollout_percentage: 100-- - key: mcp-eval-unarchive-me- name: MCP eval — unarchive target- active: false- archived: true- filters:- groups:- - properties: []- rollout_percentage: 100-- - key: mcp-eval-retarget-me- name: MCP eval — targeting-edit target- active: true- archived: false- # Two groups, variants and a payload, so a targeting edit through the- # generic update tool has unrelated configuration it can drop.- filters:- groups:- - properties:- - key: email- type: person- value: ['[email protected]']- operator: exact- rollout_percentage: 100- - properties: []- rollout_percentage: 10- multivariate:- variants:- - key: control- name: Control- rollout_percentage: 50- - key: test- name: Test- rollout_percentage: 50- payloads:- control: '{"layout":"list"}'- test: '{"layout":"grid"}'-- - key: mcp-eval-rollout-conditions- name: MCP eval — rollout-conditions read target- active: true- archived: false- filters:- groups:- - properties:- - key: '$geoip_country_code'- type: person- value: ['GB', 'IE']- operator: exact- rollout_percentage: 25-- # `flag-create-killswitch-no-experiment` creates this key, so the seeder clears it- # first. `flag-create-routes-to-experiment` has no counterpart here: the agent picks- # the key of the flag the experiment manages, so the seeder cannot name it. That- # task's leftovers still accumulate across runs.- absent_feature_flags:- - mcp-eval-billing-killswitch--tasks:- # --- sql ---------------------------------------------------------------- - id: sql-daily-event-volume- category: sql- intent: 'How many events did we ingest per day over the last 7 days?'- expected_tools: [execute-sql]- acceptable_tools: [query-trends]- success_criteria: 'Returns a per-day event count covering roughly the last seven days.'- probe:- tool: execute-sql- args:- query: 'SELECT toDate(timestamp) AS day, count() AS events FROM events WHERE timestamp > now() - INTERVAL 7 DAY GROUP BY day ORDER BY day'-- - id: sql-top-events- category: sql- intent: 'List the 10 most common event names this month with their counts.'- expected_tools: [execute-sql]- acceptable_tools: [read-data-schema, query-trends]- success_criteria: 'Returns up to ten event names ranked by count for the last ~30 days.'- probe:- tool: execute-sql- args:- query: 'SELECT event, count() AS c FROM events WHERE timestamp > now() - INTERVAL 30 DAY GROUP BY event ORDER BY c DESC LIMIT 10'-- - id: sql-find-saved-insights- category: sql- intent: 'Do we already have any saved insights about signup conversion?'- expected_tools: [execute-sql]- acceptable_tools: [insight-get]- success_criteria: 'Searches saved insights (system tables) for signup/conversion matches and reports what exists, checking the table schema before querying it.'-- # --- data-schema --------------------------------------------------------- - id: schema-discover-onboarding-events- category: data-schema- intent: "What events does this project capture? I'm especially looking for anything related to onboarding."- expected_tools: [read-data-schema]- success_criteria: 'Lists project events and calls out onboarding-related ones, without inventing event names.'- probe:- tool: read-data-schema- args:- query:- kind: events-- - id: schema-property-values- category: data-schema- intent: 'What browsers do our users use? Check the actual property values rather than guessing.'- expected_tools: [read-data-schema]- acceptable_tools: [execute-sql]- success_criteria: 'Reads browser property values from the schema tool (or SQL) and reports observed values.'-- - id: warehouse-schema-tables- category: data-schema- intent: 'What tables are available in our data warehouse, and what columns do they have?'- expected_tools: [execute-sql]- success_criteria: 'Enumerates warehouse tables with column details via system.information_schema.'-- # --- product-analytics ---------------------------------------------------- - id: trends-weekly-active-users- category: product-analytics- intent: 'Chart weekly active users for the past 8 weeks.'- expected_tools: [query-trends]- acceptable_tools: [execute-sql]- success_criteria: 'Produces a weekly unique-user series covering roughly eight weeks, after confirming the event exists in the schema.'-- - id: funnel-signup-conversion- category: product-analytics- intent: "What's our conversion rate from viewing the site to signing up, this month?"- expected_tools: [query-funnel]- acceptable_tools: [execute-sql]- success_criteria: 'Builds a two-step funnel from real events and reports a conversion percentage.'-- - id: retention-weekly-return- category: product-analytics- intent: 'Do users who signed up last month keep coming back week over week?'- expected_tools: [query-retention]- success_criteria: 'Returns a weekly retention breakdown for the signup cohort.'-- - id: paths-after-login- category: product-analytics- intent: 'What do users typically do right after logging in?'- expected_tools: [query-paths]- acceptable_tools: [execute-sql]- success_criteria: 'Shows the most common event sequences starting from a login-like event.'-- - id: lifecycle-user-composition- category: product-analytics- intent: "Of this week's active users, how many are new versus returning versus resurrecting?"- expected_tools: [query-lifecycle]- success_criteria: 'Reports the lifecycle composition buckets for the current week.'-- - id: stickiness-engagement- category: product-analytics- intent: 'How many days per week do our most engaged users actually use the product?'- expected_tools: [query-stickiness]- success_criteria: 'Returns a stickiness distribution (users by number of active days).'-- - id: trends-actors-spike- category: product-analytics- intent: 'Something spiked yesterday in our exception events — which users are behind it?'- expected_tools: [query-trends-actors]- acceptable_tools: [execute-sql, query-error-tracking-issues-list]- success_criteria: 'Identifies the people behind the spike data point rather than just re-plotting the trend.'-- - id: funnel-dropoff-actors- category: product-analytics- intent: 'Which users dropped off between the first and second step of our signup funnel this month?'- expected_tools: [query-funnel-actors]- acceptable_tools: [query-funnel, execute-sql]- success_criteria: 'Lists the actual people who dropped off entering the second step (drop-off step semantics, not a converted-through step), rather than just re-plotting the funnel conversion rate.'- probe:- # Repeated-pageview steps keep the probe runnable in any seeded demo- # project; funnelStep -2 = dropped off entering step 2. Probe mode- # only guards arg acceptance, tool errors, and latency — an empty- # actor list still passes. Step SEMANTICS (returning the right- # people) are judged in agent mode via success_criteria.- tool: query-funnel-actors- args:- funnelStep: -2- includeRecordings: false- source:- kind: FunnelsQuery- dateRange:- date_from: '-30d'- series:- - kind: EventsNode- event: '$pageview'- - kind: EventsNode- event: '$pageview'-- # --- insights -------------------------------------------------------------- - id: insight-create-daily-signups- category: insights- intent: 'Save a trends insight tracking daily signups so the team can find it later.'- expected_tools: [insight-create]- acceptable_tools: [query-trends, read-data-schema]- success_criteria: 'Creates a saved insight with a sensible name and returns its link.'-- - id: insight-rename- category: insights- intent: "Rename my 'Daily signups' insight to 'Signups (daily)'."- expected_tools: [insight-update]- acceptable_tools: [execute-sql, insight-get]- success_criteria: 'Finds the existing insight by name and updates only its name.'-- - id: insight-run-existing- category: insights- intent: 'Run our saved weekly revenue insight and tell me the latest value.'- expected_tools: [insight-query]- acceptable_tools: [insight-get, execute-sql]- success_criteria: 'Locates the saved insight, executes it, and reports the most recent data point.'-- # Multi-step chains below target the observed pain in this family: losing- # the created insight's identifiers between calls, over-long descriptions,- # and saving without ever verifying the query runs. Create/update are- # mutating tools, so these tasks are agent-mode only (no probes).- - id: insight-id-roundtrip- category: insights- intent: 'Create an insight tracking daily pageviews, then rename it, then run it and tell me the latest value.'- expected_tools: [insight-create, insight-update, insight-query]- acceptable_tools: [query-trends, read-data-schema, insight-get]- success_criteria: 'Completes the whole chain reusing the identifiers returned by insight-create for the update and query calls — no schema-rejection retries from passing the wrong identifier field — and reports the latest data point.'-- - id: insight-create-with-description- category: insights- intent: "Save a trends insight for weekly signups, with a description explaining how it's calculated and any caveats, so teammates can trust it."- expected_tools: [insight-create]- acceptable_tools: [query-trends, read-data-schema]- success_criteria: 'Creates the insight with a substantive non-empty description on the first attempt — no rejected-then-truncated retry loop on the description field.'-- - id: insight-create-verified- category: insights- intent: "Save a funnel from homepage visit to signup, and confirm it actually computes results before telling me it's done."- expected_tools: [insight-create, insight-query]- acceptable_tools: [query-funnel, read-data-schema]- success_criteria: 'Verifies the saved funnel actually runs — via insight-query after saving or by pre-testing the query before saving — and reports computed results instead of claiming success without ever executing it.'-- # --- error-tracking --------------------------------------------------------- - id: errors-top-issues- category: error-tracking- intent: 'What are our top error issues by volume this week?'- expected_tools: [query-error-tracking-issues-list]- acceptable_tools: [execute-sql]- success_criteria: 'Lists the highest-volume error issues for the last seven days.'-- - id: errors-stack-traces- category: error-tracking- intent: 'Show me recent stack traces for our most common error.'- expected_tools: [query-error-tracking-issue-events]- acceptable_tools: [query-error-tracking-issues-list]- success_criteria: 'Finds the top issue first, then fetches sample events with stack traces for it.'-- # --- feature-flags ----------------------------------------------------------- - id: flags-list-active- category: feature-flags- intent: 'List all our active feature flags.'- expected_tools: [feature-flag-get-all]- success_criteria: "Returns the project's active flags by key."- probe:- tool: feature-flag-get-all- args: {}-- - id: flag-rollout-conditions- category: feature-flags- intent: 'What are the rollout conditions for our mcp-eval-rollout-conditions feature flag?'- expected_tools: [feature-flag-get-definition-by-key]- acceptable_tools: [feature-flag-get-all, feature-flag-get-definition]- success_criteria: >- Resolves the flag by its key ('mcp-eval-rollout-conditions') directly — does not guess a numeric id — and- reports the seeded country condition and its 25% rollout.-- # Each lifecycle task below names its own fixture flag, seeded to the state that task needs.- # So the four run in any order, a second run starts where the first one did, and the score- # covers the state change as well as the tool choice. They still score tool selection when- # the fixture is missing, because an unseeded key must produce a clear answer rather than a- # fallback to the generic update tool.- - id: flag-enable-by-key- category: feature-flags- intent: 'Turn on the mcp-eval-enable-me feature flag.'- expected_tools: [feature-flag-enable]- acceptable_tools: [feature-flag-get-definition-by-key, feature-flag-get-all, feature-flag-get-definition]- success_criteria: >- Calls feature-flag-enable rather than a generic update, and sends no filters payload in any call. If the- flag does not exist, says so rather than reaching for update-feature-flag.-- - id: flag-disable-reports-impact- category: feature-flags- intent: "We're seeing errors from the mcp-eval-disable-me flag. Switch it off."- expected_tools: [feature-flag-disable]- acceptable_tools:- [- feature-flag-get-definition-by-key,- feature-flag-get-all,- feature-flag-get-definition,- feature-flags-dependent-flags-retrieve,- ]- success_criteria: >- Resolves the key to an id and calls feature-flag-disable with no filters payload. Reads the definition first- and says what else the flag is wired into (linked experiments, surveys, replay settings), or that nothing- is. If the flag does not exist, says so rather than reaching for update-feature-flag.-- - id: flag-archive-completed- category: feature-flags- intent: 'The mcp-eval-archive-me rollout is finished. Archive the flag so it stops cluttering the list.'- expected_tools: [feature-flag-archive]- acceptable_tools: [feature-flag-get-definition-by-key, feature-flag-get-all, feature-flag-get-definition]- success_criteria: >- Calls feature-flag-archive with no filters payload, and tells the user archiving also turns the flag off.- If the flag does not exist, says so rather than reaching for update-feature-flag.-- - id: flag-unarchive-stays-off- category: feature-flags- intent: 'Bring the mcp-eval-unarchive-me flag back into the flag list, I need to look at it again.'- expected_tools: [feature-flag-unarchive]- acceptable_tools: [feature-flag-get-all, feature-flag-get-definition-by-key, feature-flag-get-definition]- success_criteria: >- Calls feature-flag-unarchive with no filters payload and does not enable it, saying the flag is still turned- off. Finds the archived flag with feature-flag-get-definition-by-key, which returns it with archived: true, or- by passing {"archived":"true"} to feature-flag-get-all.-- - id: flag-targeting-edit-uses-update- category: feature-flags- intent: 'Take the mcp-eval-retarget-me flag from a 10% rollout to 25%.'- expected_tools: [update-feature-flag]- acceptable_tools: [feature-flag-get-definition-by-key, feature-flag-get-definition, feature-flag-get-all]- success_criteria: >- Reads the current definition first, then edits targeting through update-feature-flag. Does not reach for a- lifecycle tool, which cannot change targeting. Sending `filters` replaces the whole object, so the written- filters must keep the seeded email condition, both variants and both payloads, and change only the 10%- group to 25%. If the flag does not exist, says so.-- # The description asks the agent to weigh an experiment before it creates a flag. The- # pair below scores both directions: a user-visible change should route to an experiment,- # an operational flag should not. A description that pushes experiments too hard fails the- # second task. Both intents settle the choice up front, so a correct agent has a mutation- # to make and tool selection scores cleanly.- - id: flag-create-routes-to-experiment- category: feature-flags- intent: "We're putting a new version of our checkout page in front of 20% of users. Set that up — and if an- experiment is the better way to tell whether it helped, set that up instead of a plain flag."- expected_tools: [experiment-create]- acceptable_tools: [feature-flag-get-all, experiment-list]- success_criteria: >- Creates an experiment rather than a bare feature flag, and says the experiment manages its own flag. Does- not leave a separate feature flag behind alongside it.-- - id: flag-create-killswitch-no-experiment- category: feature-flags- intent:- 'Add a kill switch flag called mcp-eval-billing-killswitch so we can turn off the billing sync if it starts- failing.'- expected_tools: [create-feature-flag]- acceptable_tools: [feature-flag-get-definition-by-key, feature-flag-get-all]- success_criteria: >- Creates the flag and does not raise an experiment. An operational kill switch has no user-visible change- to measure. The intent gives the exact key, so checking it with feature-flag-get-definition-by-key is the- expected lookup; a miss there is data, not a failure.-- # --- session-replay ----------------------------------------------------------- - id: replay-rage-clicks- category: session-replay- intent: 'Find session recordings from this week where users rage-clicked.'- expected_tools: [query-session-recordings-list]- success_criteria: 'Returns recordings filtered to the last week with rage-click activity.'-- # --- llm-analytics -------------------------------------------------------------- - id: llma-recent-traces- category: llm-analytics- intent: 'Show me our latest LLM traces and how many tokens they used.'- expected_tools: [query-llm-traces-list]- success_criteria: 'Lists recent LLM traces including token usage.'-- # --- logs ------------------------------------------------------------------------ - id: logs-error-severity- category: logs- intent: 'Are there any error-severity logs from our API service in the last hour?'- expected_tools: [query-logs]- success_criteria: 'Queries logs filtered by severity and service for the last hour.'-- # --- dashboards --------------------------------------------------------------------- - id: dashboard-team-health- category: dashboards- intent: "Create a dashboard called 'Team health' from our existing activation insights."- expected_tools: [dashboard-create]- acceptable_tools: [execute-sql, insight-get]- success_criteria: 'Reuses existing insights where they exist and creates the dashboard, returning its link.'-- # --- docs -------------------------------------------------------------------------- - id: docs-behavioral-cohort- category: docs- intent: 'How do I create a behavioral cohort in PostHog?'- expected_tools: [docs-search]- success_criteria: 'Answers from documentation with source links, not from memory.'-- # --- project ------------------------------------------------------------------------- - id: project-switch-then-flags- category: project- intent: 'Switch to our other project and list its feature flags.'- expected_tools: [switch-project, feature-flag-get-all]- success_criteria: 'Switches the active project before listing flags, and says which project the flags came from.'-- # --- mcp-analytics (dogfood discoverability) -------------------------------------------- - id: mcp-tool-reliability- category: mcp-analytics- intent: 'How reliable has our execute-sql MCP tool been this week? I want error rate and p95 latency.'- expected_tools: [query-mcp-tool-stats]- acceptable_tools: [execute-sql, query-mcp-tool-daily-stats]- success_criteria: 'Uses the MCP analytics tooling (not raw SQL) to report calls, error rate, and p95 for the tool.'-- - id: mcp-session-intents- category: mcp-analytics- intent: 'What have agents been trying to do with our MCP server lately?'- expected_tools: [query-mcp-tool-sample-intents]- acceptable_tools: [execute-sql, mcp-analytics-sessions-list]- success_criteria: 'Surfaces recent agent intents from MCP analytics rather than guessing.'-- # --- metrics ----------------------------------------------------------------------------- - id: metrics-api-error-rate- category: metrics- intent: 'What does the error-rate metric for our API service look like over the last day?'- expected_tools: [metric-names-list, query-metrics]- acceptable_tools: [execute-sql]- success_criteria: 'Resolves the exact metric name first (names match exactly), then queries a rate over the last 24h — does not guess the metric name.'-- # --- data-catalog ------------------------------------------------------------------------ # MCP-surface probe/agent tasks over the catalog. The end-to-end "does the agent trust the- # catalog" behavioral suite (seeded approved/proposed/drifted/empty arms, scorers + LLM judge)- # lives in products/data_catalog/evals/ (eval_governed_metrics, eval_semantic_discovery).- - id: data-catalog-use-canonical-mrr- category: data-catalog- intent: 'What was our MRR last month?'- expected_tools: [execute-sql]- acceptable_tools: [data-catalog-metric-run]- success_criteria: >- Checks system.information_schema.metrics for an approved, non-drifted MRR metric before- deriving the number. If one exists, runs it and cites it as the canonical definition; never- presents a proposed or drifted metric as canonical.- probe:- tool: execute-sql- args:- query: "SELECT name, status, is_drifted, definition_kind FROM system.information_schema.metrics WHERE name ILIKE '%mrr%' OR description ILIKE '%recurring revenue%'"-- - id: data-catalog-check-before-deriving- category: data-catalog- intent: "What's our activation rate?"- expected_tools: [execute-sql]- success_criteria: >- Looks in the catalog (system.information_schema.metrics) for a canonical activation metric- before deriving one from scratch, and reports whether an approved one exists.- probe:- tool: execute-sql- args:- query: "SELECT name, status, is_drifted FROM system.information_schema.metrics WHERE name ILIKE '%activation%' OR description ILIKE '%activation%'"-- - id: data-catalog-prefer-certified-source- category: data-catalog- intent: 'Which table should I use to report revenue?'- expected_tools: [execute-sql]- success_criteria: >- Uses the certification column on system.information_schema.tables to prefer a certified- revenue source and avoid deprecated ones, rather than picking an arbitrary similar table.- probe:- tool: execute-sql- args:- query: 'SELECT table_name, certification, description FROM system.information_schema.tables WHERE certification IS NOT NULL'-- - id: data-catalog-never-cite-proposed- category: data-catalog- intent: 'Is there an official definition of churn I should use?'- expected_tools: [execute-sql]- success_criteria: >- Reports whether an approved (not proposed or drifted) churn metric exists in the catalog, and- does not present a proposed metric as if it were official.- probe:- tool: execute-sql- args:- query: "SELECT name, status, is_drifted FROM system.information_schema.metrics WHERE name ILIKE '%churn%' OR description ILIKE '%churn%'"diff --git a/services/mcp/evals/runner/probe.ts b/services/mcp/evals/runner/probe.tsdeleted file mode 100644index 561b79318165..000000000000--- a/services/mcp/evals/runner/probe.ts+++ /dev/null@@ -1,137 +0,0 @@-/**- * Deterministic probe runner — no LLM. Connects to a LIVE MCP server,- * verifies every required tool (the expected path + probe tool) is advertised,- * executes each task's read-only probe call, and prints/writes a score summary.- * Acceptable tools are optional alternatives (often feature-gated), so a missing- * one is not a failure.- *- * Usage:- * LIVE_MCP_URL=http://localhost:9876 LIVE_MCP_TOKEN=phx_... \- * pnpm exec tsx evals/runner/probe.ts [--out score.json]- *- * Exit code is non-zero when any required tool is missing or any probe- * fails, so the campaign (or CI) can gate on it directly.- */--import { Client } from '@modelcontextprotocol/sdk/client/index.js'-import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js'-import { ErrorCode, McpError } from '@modelcontextprotocol/sdk/types.js'-import { writeFileSync } from 'node:fs'-import process from 'node:process'--import { loadBenchmark, referencedTools, requiredTools, type BenchmarkTask } from '../benchmark/schema'-import { formatSummary, summarize, type ProbeResult, type ToolMiss } from './results'--interface AdvertisedTool {- name: string- annotations?: { readOnlyHint?: boolean }-}--async function runProbe(- client: Client,- task: BenchmarkTask,- advertised: Map<string, AdvertisedTool>-): Promise<ProbeResult> {- const probe = task.probe!- const base = { task_id: task.id, tool: probe.tool }- const tool = advertised.get(probe.tool)- if (!tool) {- return { ...base, status: 'skipped_not_advertised' }- }- // Enforce read-only from the LIVE advertisement, not just the fixture —- // an annotation regression must fail the probe rather than mutate data.- if (tool.annotations?.readOnlyHint !== true) {- return { ...base, status: 'refused_not_read_only' }- }-- const startedAt = Date.now()- try {- const result = await client.callTool({ name: probe.tool, arguments: probe.args }, undefined, {- timeout: probe.max_ms,- })- const latency = Date.now() - startedAt- if (result.isError) {- const content = Array.isArray(result.content) ? result.content : []- const text = content- .map((item) => (item && typeof item === 'object' && 'text' in item ? String(item.text) : ''))- .join(' ')- return { ...base, status: 'tool_error', latency_ms: latency, error_snippet: text.slice(0, 200) }- }- return { ...base, status: 'ok', latency_ms: latency }- } catch (error) {- const latency = Date.now() - startedAt- const message = error instanceof Error ? error.message : String(error)- // The SDK raises a typed timeout when RequestOptions.timeout elapses —- // classify on that, not on wall-clock heuristics.- const isTimeout = error instanceof McpError && error.code === ErrorCode.RequestTimeout- return {- ...base,- status: isTimeout ? 'timeout' : 'transport_error',- latency_ms: latency,- error_snippet: message.slice(0, 200),- }- }-}--async function main(): Promise<void> {- const url = process.env.LIVE_MCP_URL ?? 'http://localhost:9876'- const token = process.env.LIVE_MCP_TOKEN- if (!token) {- console.error('LIVE_MCP_TOKEN is required (a personal API key for the target instance)')- process.exit(2)- }- const outFlagIndex = process.argv.indexOf('--out')- const outPath = outFlagIndex === -1 ? null : (process.argv[outFlagIndex + 1] ?? null)- if (outFlagIndex !== -1 && outPath === null) {- console.error('--out requires a file path')- process.exit(2)- }-- const benchmark = loadBenchmark()- const transport = new StreamableHTTPClientTransport(new URL('/mcp', url), {- // Pin tools mode: the probe audits the full per-tool roster, and the- // server's auto-detection would otherwise resolve this client to the- // single-exec CLI default.- requestInit: { headers: { Authorization: `Bearer ${token}`, 'x-posthog-mcp-mode': 'tools' } },- })- const client = new Client({ name: 'mcp-eval-probe', version: '0.0.0' }, { capabilities: {} })- await client.connect(transport)-- try {- const listed = await client.listTools()- const advertised = new Map<string, AdvertisedTool>(- listed.tools.map((tool) => [tool.name, tool as AdvertisedTool])- )-- const toolMisses: ToolMiss[] = benchmark.tasks.flatMap((task) =>- requiredTools(task)- .filter((tool) => !advertised.has(tool))- .map((tool) => ({ task_id: task.id, tool }))- )-- const probes: ProbeResult[] = []- for (const task of benchmark.tasks.filter((candidate) => candidate.probe)) {- probes.push(await runProbe(client, task, advertised))- }-- const summary = summarize({- benchmarkVersion: benchmark.version,- tasksTotal: benchmark.tasks.length,- toolsReferenced: new Set(benchmark.tasks.flatMap(referencedTools)).size,- toolMisses,- probes,- })-- // stdout directly: the summary IS the program output (oxlint strips console.log).- process.stdout.write(formatSummary(summary) + '\n')- if (outPath) {- writeFileSync(outPath, JSON.stringify(summary, null, 2))- process.stdout.write(`wrote ${outPath}\n`)- }- process.exit(summary.tool_misses.length > 0 || summary.probes_failed > 0 ? 1 : 0)- } finally {- await client.close().catch(() => undefined)- }-}--void main()diff --git a/services/mcp/evals/runner/results.ts b/services/mcp/evals/runner/results.tsdeleted file mode 100644index 7dd15da4cd61..000000000000--- a/services/mcp/evals/runner/results.ts+++ /dev/null@@ -1,94 +0,0 @@-/**- * Pure result types + aggregation for eval runs. Kept free of I/O so the- * scoring math is unit-testable — campaign keep/discard decisions ride on- * these numbers.- */--export type ProbeStatus =- | 'ok'- | 'tool_error' // server answered with isError: true- | 'transport_error' // request failed or threw- | 'timeout'- | 'refused_not_read_only' // advertised annotations don't mark the tool read-only- | 'skipped_not_advertised'--export interface ToolMiss {- task_id: string- tool: string-}--export interface ProbeResult {- task_id: string- tool: string- status: ProbeStatus- latency_ms?: number- error_snippet?: string-}--export interface ProbeRunSummary {- benchmark_version: number- tasks_total: number- tools_referenced: number- /** Referenced tools the server did not advertise — discoverability misses. */- tool_misses: ToolMiss[]- probes_total: number- probes_ok: number- probes_failed: number- latency_p50_ms: number | null- latency_p95_ms: number | null- probes: ProbeResult[]-}--export function percentile(sortedValues: number[], p: number): number | null {- if (sortedValues.length === 0) {- return null- }- const rank = Math.min(sortedValues.length - 1, Math.max(0, Math.ceil((p / 100) * sortedValues.length) - 1))- return sortedValues[rank] ?? null-}--export function summarize(input: {- benchmarkVersion: number- tasksTotal: number- toolsReferenced: number- toolMisses: ToolMiss[]- probes: ProbeResult[]-}): ProbeRunSummary {- const latencies = input.probes- .map((probe) => probe.latency_ms)- .filter((value): value is number => typeof value === 'number')- .sort((a, b) => a - b)- const ok = input.probes.filter((probe) => probe.status === 'ok').length-- return {- benchmark_version: input.benchmarkVersion,- tasks_total: input.tasksTotal,- tools_referenced: input.toolsReferenced,- tool_misses: input.toolMisses,- probes_total: input.probes.length,- probes_ok: ok,- probes_failed: input.probes.length - ok,- latency_p50_ms: percentile(latencies, 50),- latency_p95_ms: percentile(latencies, 95),- probes: input.probes,- }-}--export function formatSummary(summary: ProbeRunSummary): string {- const lines = [- `benchmark v${summary.benchmark_version}: ${summary.tasks_total} tasks, ${summary.tools_referenced} referenced tools`,- `tool presence: ${summary.tool_misses.length === 0 ? 'all advertised' : `${summary.tool_misses.length} MISSING`}`,- ...summary.tool_misses.map((miss) => ` MISSING ${miss.tool} (task ${miss.task_id})`),- `probes: ${summary.probes_ok}/${summary.probes_total} ok` +- (summary.latency_p50_ms !== null- ? ` (p50 ${summary.latency_p50_ms}ms, p95 ${summary.latency_p95_ms}ms)`- : ''),- ...summary.probes- .filter((probe) => probe.status !== 'ok')- .map(- (probe) =>- ` ${probe.status.toUpperCase()} ${probe.tool} (task ${probe.task_id})${probe.error_snippet ? `: ${probe.error_snippet}` : ''}`- ),- ]- return lines.join('\n')-}diff --git a/services/mcp/evals/runner/seed.ts b/services/mcp/evals/runner/seed.tsdeleted file mode 100644index 7a1a9692ca3f..000000000000--- a/services/mcp/evals/runner/seed.ts+++ /dev/null@@ -1,235 +0,0 @@-/**- * Fixture seeder — no LLM, no MCP. Writes the benchmark's `fixtures` block straight- * to the PostHog API so an agent-mode run starts from the state its tasks describe.- *- * Run it before every agent-mode run. Without it the feature-flag tasks score tool- * selection only: the flags they name do not exist, so the state change half of each- * `success_criteria` cannot be judged. Run it again after a run to reset. It is- * idempotent over the keys it owns: every flag in `fixtures.feature_flags` is rewritten- * and every key in `fixtures.absent_feature_flags` is cleared, so those tasks start a- * second run where they started the first.- *- * `flag-create-routes-to-experiment` is the exception. The agent picks the key of the- * flag its experiment manages, so the seeder cannot name that key in- * `absent_feature_flags`, and that task's experiment and flag accumulate across runs.- *- * It talks to the REST API rather than to the MCP server on purpose: the tools under- * test must not also be the thing that builds the state they are measured against.- *- * Usage:- * LIVE_POSTHOG_URL=https://us.posthog.com LIVE_MCP_TOKEN=phx_... \- * pnpm exec tsx evals/runner/seed.ts [--project 12345]- *- * `LIVE_MCP_TOKEN` is the same personal API key the probe uses; it needs- * `feature_flag:write`, plus `user:read` when `--project` is omitted, because the project- * is then resolved through `/api/users/@me/`. Without `--project` the project comes from the token's current- * project — pass it explicitly when the MCP session points somewhere else, because- * seeding one project and scoring another looks exactly like a tool-selection- * regression.- *- * Exit code is non-zero when any fixture could not be written, so CI can gate on it.- */--import process from 'node:process'-import { parseArgs } from 'node:util'--import { type BenchmarkFlagFixture, FIXTURE_TAG, loadBenchmark } from '../benchmark/schema'--/** A flag as the list endpoint returns it, narrowed to what the seeder reads. */-interface ListedFlag {- id: number- key: string-}--interface SeedOutcome {- key: string- action: 'created' | 'updated' | 'cleared' | 'already-absent'-}--class SeedError extends Error {- constructor(- public readonly key: string,- message: string- ) {- super(message)- this.name = 'SeedError'- }-}--class PostHogApi {- constructor(- private readonly baseUrl: string,- private readonly token: string- ) {}-- async request<T>(method: string, path: string, body?: unknown): Promise<T> {- const response = await fetch(new URL(path, this.baseUrl), {- method,- headers: {- Authorization: `Bearer ${this.token}`,- ...(body === undefined ? {} : { 'Content-Type': 'application/json' }),- },- ...(body === undefined ? {} : { body: JSON.stringify(body) }),- })- const text = await response.text()- if (!response.ok) {- throw new Error(`${method} ${path} → ${response.status} ${response.statusText}: ${text.slice(0, 400)}`)- }- return (text ? JSON.parse(text) : {}) as T- }-}--/** The flag state a fixture declares, in the shape the feature-flags API accepts. */-function fixtureBody(fixture: BenchmarkFlagFixture): Record<string, unknown> {- return {- key: fixture.key,- name: fixture.name,- active: fixture.active,- archived: fixture.archived,- filters: fixture.filters,- // A project can require a tag on every new flag, which would otherwise reject- // the create. The tag also marks the flag as eval-owned in the flag list.- tags: [FIXTURE_TAG],- }-}--/**- * The fixture flag with this key, whatever state it is in.- *- * Two calls because the list endpoint hides archived flags unless `archived` is passed,- * and passing it narrows to archived flags only — neither call alone sees both.- */-async function findFlag(api: PostHogApi, projectId: number, key: string): Promise<ListedFlag | undefined> {- const search = `search=${encodeURIComponent(key)}`- for (const query of [search, `${search}&archived=true`]) {- const page = await api.request<{ results: ListedFlag[] }>(- 'GET',- `/api/projects/${projectId}/feature_flags/?${query}`- )- // `search` is a substring match, so pin the exact key rather than taking the first hit.- const exact = page.results.find((flag) => flag.key === key)- if (exact) {- return exact- }- }- return undefined-}--async function seedFlag(api: PostHogApi, projectId: number, fixture: BenchmarkFlagFixture): Promise<SeedOutcome> {- const existing = await findFlag(api, projectId, fixture.key)- const body = fixtureBody(fixture)- if (existing) {- await api.request('PATCH', `/api/projects/${projectId}/feature_flags/${existing.id}/`, body)- return { key: fixture.key, action: 'updated' }- }- await api.request('POST', `/api/projects/${projectId}/feature_flags/`, body)- return { key: fixture.key, action: 'created' }-}--async function clearFlag(api: PostHogApi, projectId: number, key: string): Promise<SeedOutcome> {- const existing = await findFlag(api, projectId, key)- if (!existing) {- return { key, action: 'already-absent' }- }- // Soft delete, the same write the `delete-feature-flag` tool makes.- await api.request('PATCH', `/api/projects/${projectId}/feature_flags/${existing.id}/`, { deleted: true })- return { key, action: 'cleared' }-}--async function resolveProjectId(api: PostHogApi, override: number | null): Promise<number> {- if (override !== null) {- return override- }- const me = await api.request<{ team?: { id?: number } }>('GET', '/api/users/@me/')- const id = me.team?.id- if (typeof id !== 'number') {- throw new Error('could not resolve a current project from /api/users/@me/ — pass --project <id>')- }- return id-}--const PROJECT_FLAG = '--project'--/** Reads `--project <id>` or `--project=<id>`, and rejects anything else.- *- * Strict because this command writes. An argument it did not understand would fall- * through to the token's current project, so a typo would seed a project nobody named.- */-function parseProjectFlag(args: string[]): number | null {- // `strict` rejects an unknown argument, a stray positional and a missing value, so a- // typo stops the run instead of falling through to the token's current project. The- // seeder writes six flags and soft-deletes one, so the wrong project is not- // recoverable by rerunning it.- const { values } = parseArgs({- args,- options: { project: { type: 'string', multiple: true } },- strict: true,- allowPositionals: false,- })- const given = values.project ?? []- if (given.length === 0) {- return null- }- if (given.length > 1) {- throw new Error(`${PROJECT_FLAG} was given ${given.length} times`)- }- const id = Number(given[0])- if (!given[0] || !Number.isInteger(id) || id <= 0) {- throw new Error(`${PROJECT_FLAG} requires a positive integer project id`)- }- return id-}--async function main(): Promise<void> {- const token = process.env.LIVE_MCP_TOKEN- if (!token) {- console.error(- 'LIVE_MCP_TOKEN is required (a personal API key with feature_flag:write, and user:read unless you pass --project)'- )- process.exit(2)- }-- let projectOverride: number | null- try {- projectOverride = parseProjectFlag(process.argv.slice(2))- } catch (error) {- console.error(error instanceof Error ? error.message : String(error))- process.exit(2)- return- }-- const api = new PostHogApi(process.env.LIVE_POSTHOG_URL ?? 'http://localhost:8000', token)- const { fixtures } = loadBenchmark()- const projectId = await resolveProjectId(api, projectOverride)-- const outcomes: SeedOutcome[] = []- const failures: SeedError[] = []- // Sequential: the fixture set is small, and one flag at a time keeps a failure- // attributable to its own key instead of to whichever request lost a race.- for (const fixture of fixtures.feature_flags) {- try {- outcomes.push(await seedFlag(api, projectId, fixture))- } catch (error) {- failures.push(new SeedError(fixture.key, error instanceof Error ? error.message : String(error)))- }- }- for (const key of fixtures.absent_feature_flags) {- try {- outcomes.push(await clearFlag(api, projectId, key))- } catch (error) {- failures.push(new SeedError(key, error instanceof Error ? error.message : String(error)))- }- }-- // stdout directly: the summary IS the program output (oxlint strips console.log).- process.stdout.write(`seeded project ${projectId}\n`)- for (const outcome of outcomes) {- process.stdout.write(` ${outcome.action} ${outcome.key}\n`)- }- for (const failure of failures) {- process.stdout.write(` FAILED ${failure.key}: ${failure.message}\n`)- }- process.exit(failures.length > 0 ? 1 : 0)-}--void main()diff --git a/services/mcp/tests/evals/benchmark.test.ts b/services/mcp/tests/evals/benchmark.test.tsdeleted file mode 100644index fa28b04b5c75..000000000000--- a/services/mcp/tests/evals/benchmark.test.ts+++ /dev/null@@ -1,132 +0,0 @@-import { describe, expect, it } from 'vitest'--import { getToolDefinitions } from '@/tools/toolDefinitions'--import {- BenchmarkFileSchema,- TASK_CATEGORIES,- fixtureFlagKeys,- fixtureFlagKeysInIntent,- loadBenchmark,- referencedTools,-} from '../../evals/benchmark/schema'--const FLAG_FIXTURE = {- key: 'mcp-eval-enable-me',- name: 'Enable target',- active: false,- archived: false,-}--const FIXTURES_BLOCK = { feature_flags: [FLAG_FIXTURE], absent_feature_flags: ['mcp-eval-create-me'] }--const TASK = {- id: 'enable-a-flag',- category: 'feature-flags',- intent: 'Turn the mcp-eval-enable-me flag on.',- expected_tools: ['feature-flag-enable'],- success_criteria: 'The flag is enabled and the reply says so.',-}--describe('MCP eval benchmark fixtures', () => {- // Loaded inside each test (not at describe scope) so a broken fixture or- // catalog fails the owning test with a real error instead of blowing up- // vitest's collection phase.- it('parses against the schema with unique task ids', () => {- const benchmark = loadBenchmark()- const ids = benchmark.tasks.map((task) => task.id)- expect(new Set(ids).size).toBe(ids.length)- })-- it('only references tools that exist in the catalog', () => {- const known = new Set(Object.keys(getToolDefinitions()))- const unknown = loadBenchmark().tasks.flatMap((task) =>- referencedTools(task)- .filter((tool) => !known.has(tool))- .map((tool) => `${task.id} → ${tool}`)- )- expect(unknown).toEqual([])- })-- it('uses every declared task category', () => {- const used = new Set(loadBenchmark().tasks.map((task) => task.category))- const unused = TASK_CATEGORIES.filter((category) => !used.has(category))- expect(unused).toEqual([])- })-- it('only probes read-only tools', () => {- const catalog = getToolDefinitions()- const unsafe = loadBenchmark()- .tasks.filter((task) => task.probe)- .filter((task) => catalog[task.probe!.tool]?.annotations?.readOnlyHint !== true)- .map((task) => `${task.id} → ${task.probe!.tool}`)- expect(unsafe).toEqual([])- })-- // Both halves of the seeded-fixture contract in one assertion. Zero tasks means the- // seeder writes state nothing reads, which is how a renamed key silently stops being- // exercised. Two or more means those tasks share a flag, which is the order-dependence- // v2 removed: whichever ran first decides what the next one starts from.- it('names every seeded flag fixture in exactly one task intent', () => {- const benchmark = loadBenchmark()- const tasksByKey = new Map(fixtureFlagKeys(benchmark.fixtures).map((key) => [key, [] as string[]]))- for (const task of benchmark.tasks) {- for (const key of fixtureFlagKeysInIntent(task, benchmark.fixtures)) {- tasksByKey.get(key)!.push(task.id)- }- }- const wrong = [...tasksByKey.entries()]- .filter(([, taskIds]) => taskIds.length !== 1)- .map(([key, taskIds]) => `${key} → ${taskIds.length ? taskIds.join(', ') : 'no task'}`)- expect(wrong).toEqual([])- })-- // Every fixture container defaults to empty, so a stripped key used to read as "no- // fixtures" rather than as a typo: the seeder wrote nothing, exited 0, and the test- // above passed over an empty map while the next run scored the lifecycle tasks- // against flags that did not exist.- it.each([- { name: 'the fixtures block spelled correctly', fixtures: FIXTURES_BLOCK, parses: true },- { name: 'a misspelled fixture list', fixtures: { feature_flag: [FLAG_FIXTURE] }, parses: false },- {- name: 'a misspelled field on a fixture flag',- fixtures: { feature_flags: [{ ...FLAG_FIXTURE, activee: true }] },- parses: false,- },- // `clearFlag` soft-deletes whatever key it is handed, so a fixture key without- // the prefix would delete one of the project's real flags rather than a fixture.- {- name: 'a fixture key missing the mcp-eval- prefix',- fixtures: { feature_flags: [{ ...FLAG_FIXTURE, key: 'billing-killswitch' }] },- parses: false,- },- {- name: 'an absent-flag key missing the mcp-eval- prefix',- fixtures: { ...FIXTURES_BLOCK, absent_feature_flags: ['billing-killswitch'] },- parses: false,- },- // The API disables a flag on the way to archiving it, so the seeder cannot leave- // a flag in both states and a fixture asking for both describes nothing.- {- name: 'a fixture that is both archived and active',- fixtures: { feature_flags: [{ ...FLAG_FIXTURE, active: true, archived: true }] },- parses: false,- },- ])('the schema accepts $name: $parses', ({ fixtures, parses }) => {- expect(BenchmarkFileSchema.safeParse({ version: 2, fixtures, tasks: [TASK] }).success).toBe(parses)- })-- it('rejects a misspelled fixtures block', () => {- const parsed = BenchmarkFileSchema.safeParse({ version: 2, fixtuers: FIXTURES_BLOCK, tasks: [TASK] })-- expect(parsed.success).toBe(false)- })-- it('probes exercise a tool the task expects', () => {- const mismatched = loadBenchmark()- .tasks.filter((task) => task.probe)- .filter((task) => ![...task.expected_tools, ...task.acceptable_tools].includes(task.probe!.tool))- .map((task) => task.id)- expect(mismatched).toEqual([])- })-})diff --git a/services/mcp/tests/evals/results.test.ts b/services/mcp/tests/evals/results.test.tsdeleted file mode 100644index 23136d0aa0a2..000000000000--- a/services/mcp/tests/evals/results.test.ts+++ /dev/null@@ -1,40 +0,0 @@-import { describe, expect, it } from 'vitest'--import { percentile, summarize, type ProbeResult } from '../../evals/runner/results'--describe('eval result aggregation', () => {- it.each([- { values: [], p: 95, expected: null },- { values: [100], p: 50, expected: 100 },- { values: [100], p: 95, expected: 100 },- { values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], p: 50, expected: 5 },- { values: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], p: 95, expected: 10 },- { values: [1, 2, 3, 4], p: 95, expected: 4 },- ])('percentile(p$p) of $values is $expected', ({ values, p, expected }) => {- expect(percentile(values, p)).toBe(expected)- })-- it('counts every non-ok status as failed and keeps latency math to ok+errored calls only', () => {- const probes: ProbeResult[] = [- { task_id: 'a', tool: 't1', status: 'ok', latency_ms: 100 },- { task_id: 'b', tool: 't2', status: 'tool_error', latency_ms: 300 },- { task_id: 'c', tool: 't3', status: 'refused_not_read_only' },- { task_id: 'd', tool: 't4', status: 'skipped_not_advertised' },- ]-- const summary = summarize({- benchmarkVersion: 0,- tasksTotal: 4,- toolsReferenced: 4,- toolMisses: [{ task_id: 'd', tool: 't4' }],- probes,- })-- expect(summary.probes_total).toBe(4)- expect(summary.probes_ok).toBe(1)- expect(summary.probes_failed).toBe(3)- expect(summary.latency_p50_ms).toBe(100)- expect(summary.latency_p95_ms).toBe(300)- expect(summary.tool_misses).toHaveLength(1)- })-})