Skip to content

Open-source SEO, GEO, and marketing skills for AI agents.

chore(evals): scaffold 3-case golden eval for CMO seed prompt

Yuting Zhongcd4195682c2d
diff --git a/CLAUDE.md b/CLAUDE.md
index c442c789..06c91a1d 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -40,7 +40,7 @@ If you touch any of these files, an eval pass is required before shipping:
- `src/server/agent-chat.tsx` chat orchestration if it adds new tool-call patterns
- Any new file under `src/server/onboarding/` that constructs prompt context (e.g., `audit.ts` writing `FIRST_TURN.md`)
-Light eval harness lives at `tests/evals/` (to be added). Pattern: golden scenario JSON + expected-shape assertions. Run via `pnpm eval` (script to be added).
+Light eval harness lives at `tests/evals/`. Pattern: golden scenario JSON + expected-shape assertions. Run via `pnpm eval`. Skips entirely when `OPENAI_API_KEY` is absent. See `tests/evals/README.md` for details.
## Project structure conventions
diff --git a/package.json b/package.json
index ef958213..7e2fd1ff 100644
--- a/package.json
+++ b/package.json
@@ -21,7 +21,8 @@
"lint": "next lint",
"test": "vitest run",
"test:watch": "vitest",
- "test:ui": "vitest --ui"
+ "test:ui": "vitest --ui",
+ "eval": "vitest run --config vitest.evals.config.ts"
},
"dependencies": {
"@ai-sdk/anthropic": "^1.0.0",
diff --git a/tests/evals/README.md b/tests/evals/README.md
new file mode 100644
index 00000000..7cf79b97
--- /dev/null
+++ b/tests/evals/README.md
@@ -0,0 +1,38 @@
+# Evals
+
+Golden-case prompt evaluations for prompts that matter. Per the V1 ship gate in PLAN.md ("eval harness, light"), these are critical for surfaces where a silent prompt regression would degrade product behavior without any test failing.
+
+## Running
+
+```bash
+pnpm eval
+```
+
+Requires `OPENAI_API_KEY` in env (or `~/.gstack/openai.json`). If absent, eval suites skip automatically — they never fail-by-not-running. The default `pnpm test` does NOT run evals; they live under `tests/evals/` and use `*.eval.ts` instead of `*.test.ts`.
+
+## What's covered
+
+| Eval file | What it checks | Decision |
+|---|---|---|
+| `cmo-first-turn.eval.ts` | When the CMO opens a chat session with `FIRST_TURN.md` present in its workspace, the first reply must reference a specific audit finding by name + dollar figure. With an empty-account `FIRST_TURN.md`, the reply must offer launch help. With no `FIRST_TURN.md`, the reply must NOT fabricate audit findings. | D15 + D19 |
+
+## How the eval works
+
+The CMO's production system prompt (from `src/server/agent-templates.ts`) is used unchanged. For each golden case:
+
+1. The case JSON specifies whether `FIRST_TURN.md` is present + its content.
+2. The eval simulates the workspace by prepending the file content to the system prompt with a clear marker. (In production the agent reads it via `exec cat FIRST_TURN.md`; the eval approximates by injecting the read result.)
+3. The LLM is called with the system prompt and a minimal opening user message ("Hi") to elicit the first assistant turn.
+4. The reply is checked against `must_contain_any_of` (regex/string match — permissive enough to absorb wording variation) and `must_not_contain` (strict — used to catch fabrication when no audit context is present).
+
+If a case fails, the eval prints the full reply for inspection. Iterate on the system prompt in `agent-templates.ts` until the case passes again.
+
+## When to add a new eval case
+
+- A new product surface introduces a prompt that, if regressed, would silently degrade behavior.
+- A bug report turns out to be a prompt-following failure — capture it as a golden case so the regression doesn't recur.
+
+## Caveats
+
+- Evals are inherently non-deterministic. Use generous `must_contain_any_of` (alternatives are forgiving) and strict `must_not_contain` (only block clear violations).
+- Cost: each case is ~1 LLM call. The full suite at V1 is ~3 calls = single-digit cents per run.
diff --git a/tests/evals/cases/empty-account.json b/tests/evals/cases/empty-account.json
new file mode 100644
index 00000000..58621b7c
--- /dev/null
+++ b/tests/evals/cases/empty-account.json
@@ -0,0 +1,10 @@
+{
+ "name": "empty-account",
+ "description": "Audit found no campaigns/spend; CMO first reply must offer launch help via the hybrid roadmap.",
+ "first_turn_md": "# First-turn context for the CMO\n\nYou have just connected the user's Google Ads account and run a baseline audit.\nThis is a one-shot context: on your FIRST chat turn after the user opens this\nproject's CMO chat, weave the highlights into your greeting, then move this\nfile to MEMORY/last-first-turn-<YYYY-MM-DD>.md so you don't repeat it.\n\n## Audit metadata\n- Project: acme\n- Audit run at: 2026-05-19T14:00:00Z\n- Account state: empty\n- Findings count: 0\n\n## Top suggestion (new account — empty audit)\n- Set a daily budget for your first campaign — most B2B starts at $50-100/day to gather signal.\n\n## Other roadmap items\n- Decide your first goal: leads vs traffic vs brand. I can help you pick.\n- Talk it through with me — 30 minutes and you'll have a campaign brief.\n\n## Suggested opener\nGreet the user by acknowledging they're just getting started with Google Ads.\nReference the budget recommendation by name. End with an open question that\ninvites them to talk through their first campaign. Keep it under 3 sentences.\n",
+ "user_first_message": "Hi",
+ "expected_assistant_first_reply": {
+ "must_contain_any_of": ["daily budget", "first campaign", "\\$50", "\\$100", "getting started", "leads vs"],
+ "must_not_contain": ["\\$42", "Brand-US", "wasted spend", "wasted-spend"]
+ }
+}
diff --git a/tests/evals/cases/no-context.json b/tests/evals/cases/no-context.json
new file mode 100644
index 00000000..b7492e97
--- /dev/null
+++ b/tests/evals/cases/no-context.json
@@ -0,0 +1,23 @@
+{
+ "name": "no-context",
+ "description": "No FIRST_TURN.md present; CMO must NOT fabricate audit findings.",
+ "first_turn_md": null,
+ "user_first_message": "Hi",
+ "expected_assistant_first_reply": {
+ "must_contain_any_of": [],
+ "must_not_contain": [
+ "Brand-US",
+ "\\$42",
+ "\\$1,260",
+ "WASTED_SPEND",
+ "wasted spend",
+ "wasted-spend",
+ "burning \\$",
+ "zero-conv",
+ "0 conversions in",
+ "LOW_QS",
+ "Quality Score 5",
+ "8 converting search terms"
+ ]
+ }
+}
diff --git a/tests/evals/cases/normal-account.json b/tests/evals/cases/normal-account.json
new file mode 100644
index 00000000..a1481d6c
--- /dev/null
+++ b/tests/evals/cases/normal-account.json
@@ -0,0 +1,10 @@
+{
+ "name": "normal-account",
+ "description": "Audit found a wasted-spend top finding; CMO first reply must reference it concretely.",
+ "first_turn_md": "# First-turn context for the CMO\n\nYou have just connected the user's Google Ads account and run a baseline audit.\nThis is a one-shot context: on your FIRST chat turn after the user opens this\nproject's CMO chat, weave the highlights into your greeting, then move this\nfile to MEMORY/last-first-turn-<YYYY-MM-DD>.md so you don't repeat it.\n\n## Audit metadata\n- Project: acme\n- Audit run at: 2026-05-19T14:00:00Z\n- Account state: normal\n- Findings count: 3\n\n## Top finding\n- Category: WASTED_SPEND\n- Headline: Brand-US is burning $42/day on 4 zero-conv keywords\n- Evidence: Last 30 days · 1,200 clicks · $1,260 spent · 0 conv\n- Suggested action: Pause those 4 keywords; expected savings $1,260/mo\n\n## Other findings\n- LOW_QS — 3 keywords below Quality Score 5\n- SEARCH_TERM_GAP — 8 converting search terms not in your keyword list\n\n## Suggested opener\nGreet the user by referencing the top finding by name with the dollar figure.\nMake it personal and specific. End with an open question that invites them to\neither fix the top finding now or talk through the broader audit. Keep it under\n3 sentences.\n",
+ "user_first_message": "Hi",
+ "expected_assistant_first_reply": {
+ "must_contain_any_of": ["Brand-US", "\\$42", "\\$1,260", "wasted spend", "wasted-spend"],
+ "must_not_contain": []
+ }
+}
diff --git a/tests/evals/cmo-first-turn.eval.ts b/tests/evals/cmo-first-turn.eval.ts
new file mode 100644
index 00000000..804c27fb
--- /dev/null
+++ b/tests/evals/cmo-first-turn.eval.ts
@@ -0,0 +1,114 @@
+import { readFileSync, readdirSync } from "node:fs";
+import { join } from "node:path";
+import { describe, expect, it } from "vitest";
+import { generateText } from "ai";
+import { openai } from "@ai-sdk/openai";
+
+import { TEMPLATES } from "@/server/agent-templates";
+
+/**
+ * Goldens for D15: CMO first-turn greeting MUST honor FIRST_TURN.md when
+ * present and MUST NOT fabricate audit findings when absent.
+ *
+ * Each case JSON specifies whether FIRST_TURN.md exists + its content. The
+ * production system prompt (from agent-templates.ts) is used unchanged. The
+ * eval injects file content via a system-prompt suffix that simulates the
+ * agent reading its workspace at session start. The LLM is called with the
+ * minimal user "Hi" to elicit the first assistant turn.
+ *
+ * Skips entirely when OPENAI_API_KEY is absent — the user explicitly
+ * declined live evals during the design review (D24). Run with `pnpm eval`
+ * to opt in.
+ */
+
+type EvalCase = {
+ name: string;
+ description: string;
+ first_turn_md: string | null;
+ user_first_message: string;
+ expected_assistant_first_reply: {
+ must_contain_any_of: string[];
+ must_not_contain: string[];
+ };
+};
+
+const HAS_KEY = Boolean(process.env.OPENAI_API_KEY);
+const CASES_DIR = join(__dirname, "cases");
+const cases: EvalCase[] = readdirSync(CASES_DIR)
+ .filter((f) => f.endsWith(".json"))
+ .map((f) => JSON.parse(readFileSync(join(CASES_DIR, f), "utf8")) as EvalCase);
+
+function cmoSystemPrompt(): string {
+ const cmo = TEMPLATES.find((t) => t.key === "cmo");
+ if (!cmo) throw new Error("CMO template not found in agent-templates.ts");
+ return cmo.system_prompt;
+}
+
+function buildSystemMessage(c: EvalCase): string {
+ const base = cmoSystemPrompt();
+ if (!c.first_turn_md) {
+ return `${base}
+
+[Workspace check]
+FIRST_TURN.md does not exist in your workspace.`;
+ }
+ return `${base}
+
+[Workspace check]
+FIRST_TURN.md IS present in your workspace. Its contents:
+
+=== FIRST_TURN.md ===
+${c.first_turn_md}
+=== END FIRST_TURN.md ===
+
+Per your instructions, weave the file's content into your opening greeting
+now. Treat this as your FIRST chat turn.`;
+}
+
+describe.skipIf(!HAS_KEY)("CMO first-turn (eval, D15)", () => {
+ for (const c of cases) {
+ it(
+ `${c.name} — ${c.description}`,
+ async () => {
+ const { text } = await generateText({
+ model: openai("gpt-4o-mini"),
+ system: buildSystemMessage(c),
+ prompt: c.user_first_message,
+ maxRetries: 1,
+ });
+
+ const lower = text.toLowerCase();
+
+ if (c.expected_assistant_first_reply.must_contain_any_of.length > 0) {
+ const matched = c.expected_assistant_first_reply.must_contain_any_of.some(
+ (pattern) => new RegExp(pattern, "i").test(text),
+ );
+ expect(
+ matched,
+ `\nCMO reply did not reference any of the expected anchors.\nExpected one of: ${c.expected_assistant_first_reply.must_contain_any_of.join(", ")}\nGot:\n${text}\n`,
+ ).toBe(true);
+ }
+
+ for (const forbidden of c.expected_assistant_first_reply.must_not_contain) {
+ const re = new RegExp(forbidden, "i");
+ expect(
+ re.test(text),
+ `\nCMO reply contained forbidden phrase \"${forbidden}\".\nGot:\n${text}\n`,
+ ).toBe(false);
+ }
+
+ void lower;
+ },
+ // Allow up to 30s per case (LLM call latency).
+ 30_000,
+ );
+ }
+});
+
+if (!HAS_KEY) {
+ // Surface a non-test diagnostic so the user knows evals were skipped (and why).
+ // Vitest prints stdout when --reporter=verbose; in default reporter it's hidden.
+ console.log(
+ "[evals] OPENAI_API_KEY not set — CMO first-turn evals skipped. Run `pnpm eval` with the key in env to opt in.",
+ );
+}
diff --git a/vitest.evals.config.ts b/vitest.evals.config.ts
new file mode 100644
index 00000000..b685df77
--- /dev/null
+++ b/vitest.evals.config.ts
@@ -0,0 +1,25 @@
+import { defineConfig } from "vitest/config";
+import { fileURLToPath } from "node:url";
+import { dirname, resolve } from "node:path";
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+
+/**
+ * Separate config for prompt evals. Lives under tests/evals/*.eval.ts so the
+ * default `pnpm test` (which globs src/**\/*.test.ts) doesn't trigger them.
+ * Run via `pnpm eval`. Skips entirely when OPENAI_API_KEY is absent — the
+ * user opted out of live evals during the design review (D24).
+ */
+export default defineConfig({
+ resolve: {
+ alias: {
+ "@": resolve(__dirname, "src"),
+ },
+ },
+ test: {
+ environment: "node",
+ globals: false,
+ include: ["tests/evals/**/*.eval.ts"],
+ testTimeout: 60_000,
+ },
+});