Skip to content

OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security

test(evals): measure natural-language security skill routing (#1700)

Michael D'Angelof905ce3ccd0e
diff --git a/.github/workflows/node-ci.yml b/.github/workflows/node-ci.yml
index 433c391b9..1205b27ab 100644
--- a/.github/workflows/node-ci.yml
+++ b/.github/workflows/node-ci.yml
@@ -469,6 +469,9 @@ jobs:
- name: Test secret discovery eval contracts
if: matrix.os == 'ubuntu-latest' && matrix.python == '3.12'
run: node --experimental-strip-types --test evals/secret-discovery/test_*.mts
+ - name: Test workflow routing eval contracts
+ if: matrix.os == 'ubuntu-latest' && matrix.python == '3.12'
+ run: node --experimental-strip-types --test evals/workflow-routing/test_*.mts
required-test:
# Skipped metadata checks must not replace required source check results.
diff --git a/evals/README.md b/evals/README.md
index ae456816e..7a4ad74c2 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -11,7 +11,10 @@ being embedded in its source tree or shipped npm runtime.
- [Secret discovery](secret-discovery/README.md): checks whether the production
core audit finds synthetic credentials in source and keeps them in its final
findings, with deterministic grading and harness checks.
+- [Workflow routing](workflow-routing/README.md): checks whether repository-installed
+ production skills route natural-language requests to the appropriate workflow,
+ stopping before workflow execution.
-Model runs are opt-in. CI runs the deterministic triage and secret-discovery
-helper checks and the real-IPC reducer regression through the normal MCP test
-suite.
+Model runs are opt-in. CI runs the deterministic triage, secret-discovery, and
+workflow-routing helper checks and the real-IPC reducer regression through the
+normal MCP test suite.
diff --git a/evals/tsconfig.json b/evals/tsconfig.json
index 0aed99eab..52d5541c7 100644
--- a/evals/tsconfig.json
+++ b/evals/tsconfig.json
@@ -6,6 +6,7 @@
"include": [
"deep-reducer/*.mts",
"secret-discovery/*.mts",
+ "workflow-routing/*.mts",
"triage-finding/assertions/*.mts",
"triage-finding/scripts/*.mts",
"triage-finding/sastbench/assertions/*.mts",
diff --git a/evals/workflow-routing/.gitignore b/evals/workflow-routing/.gitignore
new file mode 100644
index 000000000..30846059d
--- /dev/null
+++ b/evals/workflow-routing/.gitignore
@@ -0,0 +1 @@
+/reports/
diff --git a/evals/workflow-routing/README.md b/evals/workflow-routing/README.md
new file mode 100644
index 000000000..3710d386f
--- /dev/null
+++ b/evals/workflow-routing/README.md
@@ -0,0 +1,83 @@
+# Repository-installed skill routing eval
+
+This opt-in cohort checks whether a fresh Codex thread loads the appropriate
+production security skill for a natural-language request. The request does not
+name a skill. The eval stops after workflow selection and loading, before source
+review, scan startup, triage, patching, or verification.
+
+| Request | Expected skill |
+| ------------------------------------------ | -------------------- |
+| Ordinary repository security review | `security-scan` |
+| Security review of uncommitted changes | `security-diff-scan` |
+| Explicit exhaustive, multi-pass scan | `deep-security-scan` |
+| Assess a supplied vulnerability claim | `triage-finding` |
+| Implement and test a security fix | `fix-finding` |
+| Explicitly verify an existing security fix | `verify-fix` |
+| Fix an ordinary sorting bug | None |
+| Check a refactor's build and unit tests | None |
+
+The negative controls catch accidental security-workflow activation. In
+particular, testing a fix does not automatically request the standalone
+`verify-fix` workflow, and an ordinary repository scan does not request Deep Scan.
+
+## Run
+
+Use the repository's normal SDK and MCP dependency setup, then:
+
+```sh
+pnpm --dir sdk/typescript run build:evals
+node --experimental-strip-types evals/workflow-routing/run.mts
+```
+
+An optional positional argument selects a model. Without one, Codex selects its
+default; the SDK does not expose that resolved model. Model runs consume usage
+and are not part of CI.
+
+The runner reuses the secret-discovery eval's private-home authentication,
+restricted permission profile, native preflight, cancellation draining, and
+cleanup. See its [runtime notes](../secret-discovery/README.md#run), including
+saved-login refresh behavior and the Windows elevated-sandbox requirement. It
+also isolates `HOME` so ambient user skills do not enter discovery. Each case
+starts a fresh thread with the fixture as its authoritative working directory.
+
+All current production skills and their references are copied unchanged into
+the temporary repository's `.agents` directory. Expected routes and the grader
+remain outside the readable fixture. Plugins, connected apps, network access,
+memories, and subagents remain disabled. No application source or real findings
+are supplied; the invoice claim is synthetic and is not evaluated as a finding.
+
+## Evidence and limits
+
+A routing pass requires a completed turn and a successful shell-tool trace that
+returns the expected skill's production heading and introductory instructions.
+The grader accepts ordinary or line-numbered reads. An agent's claimed choice,
+a filename listing, failed read, or reading every skill does not satisfy the
+positive cases. Loading another security skill fails both positive and negative
+cases. The final response is retained for review, not used as proof of selection.
+
+`report.json` records each route, turn outcome, full-turn token usage, elapsed
+time, failed/repeated commands, repeated skill reads, final response, source
+revision and dirty state, package/SDK/CLI versions, requested model, prompt hash,
+and hashes of all staged skill entrypoints. Per-case JSONL files retain SDK events.
+Artifacts are saved under ignored `reports/`. Cohort status and missing-case IDs
+distinguish a partial or interrupted run from all-case success. A failed or incomplete turn cannot
+pass; unavailable usage remains `null`. No monetary cost or efficiency threshold
+is inferred from token counts.
+
+This checks repository skill discovery and loading, not marketplace installation,
+the packed npm layout, successful workflow execution, target handoff, saved
+artifacts, finding quality, or missing-tool recovery. The harness explicitly
+stops at the routing boundary; a passing case must not be presented as a completed
+security task. It does not parse arbitrary shell programs to prove what they
+read. Unsupported command-output formats can fail the content-based grader and
+should be reviewed in the retained transcript before changing the assertion.
+
+## Deterministic checks
+
+```sh
+node --experimental-strip-types --test evals/workflow-routing/test_*.mts
+```
+
+Normal CI checks unchanged production staging, each expected route, negative
+controls, false-positive trace rejection, failures, cancellation, and SDK thread
+setup. These checks validate the harness; they do not establish model accuracy.
diff --git a/evals/workflow-routing/cases.mts b/evals/workflow-routing/cases.mts
new file mode 100644
index 000000000..520437d23
--- /dev/null
+++ b/evals/workflow-routing/cases.mts
@@ -0,0 +1,58 @@
+export const cases = [
+ {
+ id: "repository-default",
+ request: "Check this repository for security vulnerabilities.",
+ skill: "security-scan",
+ },
+ {
+ id: "working-tree-diff",
+ request: "Review my uncommitted changes for security vulnerabilities.",
+ skill: "security-diff-scan",
+ },
+ {
+ id: "explicit-deep",
+ request: "Do an exhaustive, multi-pass security scan of this repository.",
+ skill: "deep-security-scan",
+ },
+ {
+ id: "existing-finding",
+ request:
+ "Triage this reported vulnerability against this repository: GET /invoices/:id exposes another user's invoice without checking ownership. Assess the claim; do not fix it.",
+ skill: "triage-finding",
+ },
+ {
+ id: "explicit-security-fix",
+ request:
+ "Fix and test this security finding: GET /invoices/:id exposes another user's invoice without checking ownership. Preserve legitimate owner access.",
+ skill: "fix-finding",
+ },
+ {
+ id: "explicit-fix-verification",
+ request:
+ "Verify that the existing security fix closes the reported cross-user invoice access in GET /invoices/:id. Check legitimate owner access too. Do not modify code.",
+ skill: "verify-fix",
+ },
+ {
+ id: "ordinary-bug-fix",
+ request:
+ "Fix the invoice table's incorrect alphabetical sort order and run its tests.",
+ skill: null,
+ },
+ {
+ id: "ordinary-test-run",
+ request: "Verify that this refactor builds and its unit tests pass.",
+ skill: null,
+ },
+] as const;
+
+export type RoutingCase = (typeof cases)[number];
+
+export function routingPrompt(scenario: RoutingCase) {
+ return `${scenario.request}
+
+This is a routing-only evaluation. Use the installed skills as you normally
+would for the request above. Read the applicable workflow's SKILL.md, if any,
+then stop before executing the workflow. Do not inspect application source,
+run tests, edit files, start scans, or contact services. Briefly state the next
+step; do not claim that the requested work has been completed.`;
+}
diff --git a/evals/workflow-routing/harness.mts b/evals/workflow-routing/harness.mts
new file mode 100644
index 000000000..0c69831fb
--- /dev/null
+++ b/evals/workflow-routing/harness.mts
@@ -0,0 +1,222 @@
+import { createHash } from "node:crypto";
+import { cp, mkdir, readFile, readdir } from "node:fs/promises";
+import { join } from "node:path";
+import type {
+ ThreadEvent,
+ ThreadOptions,
+} from "../../sdk/typescript/node_modules/@openai/codex-sdk/dist/index.js";
+import { codexSettings, threadSettings } from "../secret-discovery/harness.mts";
+import { deepScanPermissionProfileFallbackError } from "../secret-discovery/runtime.mts";
+import { cases, routingPrompt, type RoutingCase } from "./cases.mts";
+
+const pluginRoot = join(import.meta.dirname, "../../plugins/codex-security");
+export const sha256 = (text: string) =>
+ createHash("sha256").update(text).digest("hex");
+
+export interface Skill {
+ name: string;
+ sha256: string;
+ heading: string;
+ introduction: string;
+}
+
+/** Install unchanged repository skills; labels and grader stay outside this tree. */
+export async function prepareRoutingEval(root: string) {
+ const repo = join(root, "repository");
+ const runtime = join(repo, ".agents");
+ await mkdir(runtime, { recursive: true });
+ for (const directory of ["skills", "references"]) {
+ await cp(join(pluginRoot, directory), join(runtime, directory), {
+ recursive: true,
+ });
+ }
+ const skills: Skill[] = [];
+ for (const name of (await readdir(join(runtime, "skills"))).sort()) {
+ const source = await readFile(
+ join(runtime, "skills", name, "SKILL.md"),
+ "utf8",
+ );
+ const body = source.replace(/^---\r?\n[\s\S]*?\r?\n---\r?\n/u, "");
+ const paragraphs = body.trim().split(/\r?\n\s*\r?\n/u);
+ const heading = paragraphs[0];
+ const introduction = paragraphs.find(
+ (paragraph) => !paragraph.startsWith("#"),
+ );
+ if (!heading?.startsWith("# ") || !introduction)
+ throw new Error(`Cannot identify production skill content: ${name}`);
+ skills.push({ name, sha256: sha256(source), heading, introduction });
+ }
+ return { repo, runtime, skills };
+}
+
+export function routingSettings(
+ home: string,
+ codexPath: string,
+ environment = process.env,
+) {
+ // Repository skills are the only skill installation in this eval. Keep the
+ // caller's ~/.agents/skills out of discovery as well as their Codex config.
+ return codexSettings(home, codexPath, {
+ ...environment,
+ HOME: home,
+ ...(process.platform === "win32" ? { USERPROFILE: home } : {}),
+ });
+}
+
+function unnumbered(text: string) {
+ return text.replace(/\r\n/gu, "\n").replace(/^\s*\d+\t/gmu, "");
+}
+
+/** Recognize returned production content, not a filename or a claimed choice. */
+export function gradeRoutingTrace(
+ expected: string | null,
+ skills: readonly Skill[],
+ events: readonly ThreadEvent[],
+) {
+ const reads: string[] = [];
+ const commands: string[] = [];
+ let failedCommands = 0;
+ let unexpectedActions = 0;
+ let failed = false;
+ let completed = false;
+ let finalResponse = "";
+ for (const event of events) {
+ if (event.type === "turn.completed") completed = true;
+ if (event.type === "turn.failed" || event.type === "error") failed = true;
+ if (event.type !== "item.completed") continue;
+ const { item } = event;
+ if (item.type === "agent_message") finalResponse = item.text;
+ if (["file_change", "mcp_tool_call", "web_search"].includes(item.type))
+ unexpectedActions++;
+ if (item.type !== "command_execution") continue;
+ commands.push(item.command);
+ if (item.exit_code !== 0 || item.status !== "completed") {
+ failedCommands++;
+ continue;
+ }
+ const output = unnumbered(item.aggregated_output);
+ for (const skill of skills) {
+ if (output.includes(skill.heading) && output.includes(skill.introduction))
+ reads.push(skill.name);
+ }
+ }
+ const loadedSkills = [...new Set(reads)];
+ const expectedSkills = expected === null ? [] : [expected];
+ return {
+ passed:
+ completed &&
+ !failed &&
+ finalResponse.trim().length > 0 &&
+ unexpectedActions === 0 &&
+ JSON.stringify(loadedSkills) === JSON.stringify(expectedSkills),
+ turnCompleted: completed,
+ turnFailed: failed,
+ expectedSkill: expected,
+ loadedSkills,
+ commandCount: commands.length,
+ failedCommands,
+ repeatedCommands: commands.length - new Set(commands).size,
+ repeatedSkillReads: reads.length - loadedSkills.length,
+ unexpectedActions,
+ finalResponse,
+ };
+}
+
+export interface RoutingCodex {
+ startThread(options: ThreadOptions): {
+ runStreamed(
+ prompt: string,
+ options: { signal: AbortSignal },
+ ): Promise<{ events: AsyncIterable<ThreadEvent> }>;
+ };
+}
+
+export async function runRoutingCase(
+ prepared: Awaited<ReturnType<typeof prepareRoutingEval>>,
+ scenario: RoutingCase,
+ codex: RoutingCodex,
+ { model, signal }: { model?: string; signal?: AbortSignal } = {},
+) {
+ const prompt = routingPrompt(scenario);
+ const events: ThreadEvent[] = [];
+ const controller = new AbortController();
+ const combined = signal
+ ? AbortSignal.any([signal, controller.signal])
+ : controller.signal;
+ const started = performance.now();
+ let error: string | undefined;
+ try {
+ const thread = codex.startThread(threadSettings(prepared, model));
+ const stream = await thread.runStreamed(prompt, { signal: combined });
+ for await (const event of stream.events) {
+ events.push(event);
+ const warning =
+ event.type === "error"
+ ? event.message
+ : event.type === "item.completed" && event.item.type === "error"
+ ? event.item.message
+ : undefined;
+ const fallback = deepScanPermissionProfileFallbackError(warning);
+ if (fallback) {
+ error ??= fallback.message;
+ controller.abort(fallback);
+ }
+ // Drain after cancellation before the shared harness removes temp state.
+ }
+ combined.throwIfAborted();
+ } catch (failure) {
+ error ??= failure instanceof Error ? failure.message : String(failure);
+ }
+ const usage = events.findLast(
+ (event) => event.type === "turn.completed",
+ )?.usage;
+ const grade = gradeRoutingTrace(scenario.skill, prepared.skills, events);
+ return {
+ events,
+ report: {
+ caseId: scenario.id,
+ ...grade,
+ passed: grade.passed && error === undefined,
+ ...(error ? { error } : {}),
+ elapsedMilliseconds: Math.round(performance.now() - started),
+ usage: usage ?? null,
+ promptSha256: sha256(prompt),
+ },
+ };
+}
+
+/** Partial or interrupted cohorts cannot be mistaken for all-case success. */
+export function summarizeRoutingCases(
+ results: readonly { caseId: string; passed: boolean }[],
+ {
+ interrupted = false,
+ error,
+ }: { interrupted?: boolean; error?: string } = {},
+) {
+ const missingCases = cases
+ .filter(
+ (scenario) => !results.some((result) => result.caseId === scenario.id),
+ )
+ .map((scenario) => scenario.id);
+ const complete = missingCases.length === 0 && results.length === cases.length;
+ const passed =
+ complete &&
+ !interrupted &&
+ error === undefined &&
+ results.every((result) => result.passed);
+ return {
+ status: interrupted
+ ? "interrupted"
+ : error
+ ? "failed"
+ : !complete
+ ? "incomplete"
+ : passed
+ ? "passed"
+ : "failed",
+ passed,
+ expectedCases: cases.map((scenario) => scenario.id),
+ missingCases,
+ ...(error ? { error } : {}),
+ };
+}
diff --git a/evals/workflow-routing/run.mts b/evals/workflow-routing/run.mts
new file mode 100644
index 000000000..da1fc3330
--- /dev/null
+++ b/evals/workflow-routing/run.mts
@@ -0,0 +1,119 @@
+import { execFileSync } from "node:child_process";
+import {
+ mkdir,
+ mkdtemp,
+ readFile,
+ realpath,
+ writeFile,
+} from "node:fs/promises";
+import { join } from "node:path";
+import { Codex } from "../../sdk/typescript/node_modules/@openai/codex-sdk/dist/index.js";
+import * as auth from "../../sdk/typescript/dist/auth.js";
+import {
+ createIsolatedHome,
+ resolveCodexCommand,
+} from "../../sdk/typescript/dist/runtime.js";
+import { preflightEval } from "../secret-discovery/harness.mts";
+import { withEvalState } from "../secret-discovery/runtime.mts";
+import { cases } from "./cases.mts";
+import {
+ prepareRoutingEval,
+ routingSettings,
+ runRoutingCase,
+ summarizeRoutingCases,
+ type Skill,
+} from "./harness.mts";
+
+const reports = join(import.meta.dirname, "reports");
+await mkdir(reports, { recursive: true });
+const directory = await mkdtemp(join(reports, "run-"));
+const model = process.argv[2];
+const sourceRoot = join(import.meta.dirname, "../..");
+const git = (...args: string[]) =>
+ execFileSync("git", args, { cwd: sourceRoot, encoding: "utf8" }).trim();
+const packageVersion = async (file: string) =>
+ JSON.parse(await readFile(join(sourceRoot, file), "utf8")).version as string;
+const identity = {
+ sourceRevision: git("rev-parse", "HEAD"),
+ sourceDirty: git("status", "--porcelain", "--untracked-files=normal") !== "",
+ packageVersion: await packageVersion("sdk/typescript/package.json"),
+ codexSdkVersion: await packageVersion(
+ "sdk/typescript/node_modules/@openai/codex-sdk/package.json",
+ ),
+ codexCliVersion: await packageVersion(
+ "sdk/typescript/node_modules/@openai/codex/package.json",
+ ),
+ requestedModel: model ?? null,
+ modelSelection: model
+ ? "explicit"
+ : "Codex default; SDK does not expose resolved model",
+ scope:
+ "repository-installed skill routing; workflow execution is not evaluated",
+};
+const results: Awaited<ReturnType<typeof runRoutingCase>>["report"][] = [];
+let skills: Skill[] = [];
+let interrupted = false;
+let error: string | undefined;
+const saveReport = () =>
+ writeFile(
+ join(directory, "report.json"),
+ JSON.stringify(
+ {
+ ...identity,
+ ...summarizeRoutingCases(results, { interrupted, error }),
+ skills,
+ results,
+ },
+ null,
+ 2,
+ ) + "\n",
+ );
+console.log(`Eval artifacts: ${directory}`);
+await saveReport();
+try {
+ await withEvalState(
+ createIsolatedHome,
+ (
+ auth as unknown as typeof import("../../sdk/typescript/src/auth.js")
+ ).configuredCodexHome(process.env),
+ async ({ root, home, signal }) => {
+ try {
+ const prepared = await prepareRoutingEval(root);
+ skills = prepared.skills;
+ const codexPath = await realpath(resolveCodexCommand({}).command);
+ const settings = await preflightEval(
+ prepared,
+ routingSettings(home, codexPath),
+ signal,
+ );
+ const codex = new Codex(settings);
+ for (const scenario of cases) {
+ signal.throwIfAborted();
+ const { events, report } = await runRoutingCase(
+ prepared,
+ scenario,
+ codex,
+ { model, signal },
+ );
+ await writeFile(
+ join(directory, `${scenario.id}.jsonl`),
+ events.map((event) => JSON.stringify(event)).join("\n") + "\n",
+ );
+ results.push(report);
+ await saveReport();
+ console.log(`${scenario.id}: ${report.passed ? "PASS" : "FAIL"}`);
+ }
+ process.exitCode = results.every((result) => result.passed) ? 0 : 1;
+ } finally {
+ interrupted = signal.aborted;
+ }
+ },
+ );
+} catch (failure) {
+ error = failure instanceof Error ? failure.message : String(failure);
+ throw failure;
+} finally {
+ // withEvalState can finish before its callback after an early signal.
+ interrupted ||= process.exitCode === 130 || process.exitCode === 143;
+ await saveReport();
+}
diff --git a/evals/workflow-routing/test_routing.mts b/evals/workflow-routing/test_routing.mts
new file mode 100644
index 000000000..2a4cb2c85
--- /dev/null
+++ b/evals/workflow-routing/test_routing.mts
@@ -0,0 +1,328 @@
+import assert from "node:assert/strict";
+import { mkdtemp, readFile, readdir, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { after, test } from "node:test";
+import type { ThreadEvent } from "../../sdk/typescript/node_modules/@openai/codex-sdk/dist/index.js";
+import { cases, routingPrompt } from "./cases.mts";
+import {
+ gradeRoutingTrace,
+ prepareRoutingEval,
+ routingSettings,
+ runRoutingCase,
+ sha256,
+ summarizeRoutingCases,
+ type Skill,
+} from "./harness.mts";
+
+const root = await mkdtemp(join(tmpdir(), "routing-contract-"));
+after(() => rm(root, { recursive: true, force: true }));
+const prepared = await prepareRoutingEval(root);
+const skill = (name: string) =>
+ prepared.skills.find((entry) => entry.name === name)!;
+const message: ThreadEvent = {
+ type: "item.completed",
+ item: {
+ id: "message",
+ type: "agent_message",
+ text: "Ready for the next step.",
+ },
+};
+const completed: ThreadEvent = {
+ type: "turn.completed",
+ usage: {
+ input_tokens: 10,
+ cached_input_tokens: 0,
+ cache_write_input_tokens: 0,
+ output_tokens: 5,
+ reasoning_output_tokens: 0,
+ },
+};
+function read(
+ entry: Skill,
+ output = `${entry.heading}\n\n${entry.introduction}`,
+): ThreadEvent {
+ return {
+ type: "item.completed",
+ item: {
+ id: "read",
+ type: "command_execution",
+ command: `cat .agents/skills/${entry.name}/SKILL.md`,
+ aggregated_output: output,
+ status: "completed",
+ exit_code: 0,
+ },
+ };
+}
+const grade = (expected: string | null, events: ThreadEvent[]) =>
+ gradeRoutingTrace(expected, prepared.skills, events);
+
+test("stages every production skill unchanged without the labels or harness", async () => {
+ assert.deepEqual(await readdir(prepared.repo), [".agents"]);
+ assert.deepEqual((await readdir(prepared.runtime)).sort(), [
+ "references",
+ "skills",
+ ]);
+ for (const entry of prepared.skills) {
+ const source = await readFile(
+ join(
+ import.meta.dirname,
+ "../../plugins/codex-security/skills",
+ entry.name,
+ "SKILL.md",
+ ),
+ "utf8",
+ );
+ const staged = await readFile(
+ join(prepared.runtime, "skills", entry.name, "SKILL.md"),
+ "utf8",
+ );
+ assert.equal(staged, source);
+ assert.equal(entry.sha256, sha256(source));
+ }
+ for (const scenario of cases) {
+ if (scenario.skill) assert.ok(skill(scenario.skill));
+ assert.ok(!routingPrompt(scenario).includes(scenario.id));
+ assert.ok(!routingPrompt(scenario).includes("$"));
+ }
+});
+
+for (const scenario of cases) {
+ test(`accepts the expected observed route: ${scenario.id}`, () => {
+ const reads = scenario.skill ? [read(skill(scenario.skill))] : [];
+ assert.equal(
+ grade(scenario.skill, [...reads, message, completed]).passed,
+ true,
+ );
+ });
+}
+
+test("a claimed choice or filename listing cannot satisfy a skill read", () => {
+ const expected = skill("security-scan");
+ for (const events of [
+ [message, completed],
+ [
+ read(expected, ".agents/skills/security-scan/SKILL.md"),
+ message,
+ completed,
+ ],
+ [
+ read(expected, `${expected.heading}\nname: security-scan`),
+ message,
+ completed,
+ ],
+ ]) {
+ assert.equal(grade(expected.name, events).passed, false);
+ }
+});
+
+test("accepts numbered source output but rejects failed or unfinished reads", () => {
+ const expected = skill("verify-fix");
+ const output = `${expected.heading}\n\n${expected.introduction}`
+ .split("\n")
+ .map((line, index) => `${index + 1}\t${line}`)
+ .join("\n");
+ assert.equal(
+ grade(expected.name, [read(expected, output), message, completed]).passed,
+ true,
+ );
+ for (const patch of [
+ { exit_code: 1 },
+ { exit_code: undefined },
+ { status: "failed" },
+ { status: "in_progress" },
+ ]) {
+ const event = read(expected);
+ if (event.type === "item.completed") Object.assign(event.item, patch);
+ assert.equal(
+ grade(expected.name, [event, message, completed]).passed,
+ false,
+ );
+ }
+});
+
+test("rejects deep-by-default, implicit verification, and loading every skill", () => {
+ for (const [expected, loaded] of [
+ ["security-scan", ["deep-security-scan"]],
+ ["fix-finding", ["fix-finding", "verify-fix"]],
+ ["security-diff-scan", prepared.skills.map((entry) => entry.name)],
+ [null, ["fix-finding"]],
+ [null, ["verify-fix"]],
+ ] as const) {
+ assert.equal(
+ grade(expected, [
+ ...loaded.map((name) => read(skill(name))),
+ message,
+ completed,
+ ]).passed,
+ false,
+ );
+ }
+});
+
+test("recognizes content returned by glob reads in positive and negative cases", () => {
+ const event = read(skill("security-scan"));
+ if (
+ event.type === "item.completed" &&
+ event.item.type === "command_execution"
+ )
+ event.item.command = "cat .agents/skills/security-scan/*";
+ assert.equal(
+ grade("security-scan", [event, message, completed]).passed,
+ true,
+ );
+ assert.equal(grade(null, [event, message, completed]).passed, false);
+});
+
+test("failed or incomplete turns cannot pass even with the correct skill", () => {
+ const observed = [read(skill("security-scan")), message];
+ assert.equal(grade("security-scan", observed).passed, false);
+ assert.equal(
+ grade("security-scan", [
+ ...observed,
+ { type: "turn.failed", error: { message: "fixture failure" } },
+ completed,
+ ]).passed,
+ false,
+ );
+ assert.equal(
+ grade("security-scan", [read(skill("security-scan")), completed]).passed,
+ false,
+ );
+});
+
+test("records repeated reads and failed commands without pretending to measure completion", () => {
+ const observed = read(skill("security-scan"));
+ const failure = read(skill("security-scan"));
+ if (failure.type === "item.completed")
+ Object.assign(failure.item, { exit_code: 1, status: "failed" });
+ const result = grade("security-scan", [
+ failure,
+ observed,
+ observed,
+ message,
+ completed,
+ ]);
+ assert.equal(result.passed, true);
+ assert.equal(result.failedCommands, 1);
+ assert.equal(result.repeatedCommands, 2);
+ assert.equal(result.repeatedSkillReads, 1);
+});
+
+test("rejects workflow execution visible in non-shell tool traces", () => {
+ const event: ThreadEvent = {
+ type: "item.completed",
+ item: {
+ id: "write",
+ type: "file_change",
+ status: "completed",
+ changes: [{ path: "app.js", kind: "update" }],
+ },
+ };
+ assert.equal(
+ grade("fix-finding", [
+ read(skill("fix-finding")),
+ event,
+ message,
+ completed,
+ ]).passed,
+ false,
+ );
+});
+
+test("uses an isolated home for Codex config and ambient skill discovery", () => {
+ const settings = routingSettings(
+ join(root, "home"),
+ join(root, "bin", "codex"),
+ { HOME: "/ambient", PATH: "/usr/bin" },
+ );
+ assert.equal(settings.env.HOME, join(root, "home"));
+ assert.equal(settings.env.CODEX_HOME, join(root, "home"));
+ assert.ok(settings.configOverrides.includes("features.plugins=false"));
+});
+
+test("starts a fresh thread in the fixture and grades its actual events", async () => {
+ const events = [read(skill("security-scan")), message, completed];
+ const result = await runRoutingCase(
+ prepared,
+ cases[0],
+ {
+ startThread(options) {
+ assert.equal(options.workingDirectory, prepared.repo);
+ assert.equal(options.model, "fixture-model");
+ return {
+ async runStreamed(prompt) {
+ assert.equal(prompt, routingPrompt(cases[0]));
+ return {
+ events: (async function* () {
+ yield* events;
+ })(),
+ };
+ },
+ };
+ },
+ },
+ { model: "fixture-model" },
+ );
+ assert.equal(result.report.passed, true);
+ assert.equal(result.report.usage?.input_tokens, 10);
+ assert.deepEqual(result.events, events);
+});
+
+test("keeps an interrupted result failed while draining its stream", async () => {
+ const controller = new AbortController();
+ let drained = false;
+ const result = await runRoutingCase(
+ prepared,
+ cases[0],
+ {
+ startThread() {
+ return {
+ async runStreamed() {
+ return {
+ events: (async function* () {
+ yield read(skill("security-scan"));
+ controller.abort(new Error("fixture cancellation"));
+ yield message;
+ yield completed;
+ drained = true;
+ })(),
+ };
+ },
+ };
+ },
+ },
+ { signal: controller.signal },
+ );
+ assert.equal(drained, true);
+ assert.equal(result.report.passed, false);
+ assert.equal(result.report.error, "fixture cancellation");
+});
+
+test("cohort reports distinguish partial, interrupted, failed, and complete results", () => {
+ const results = cases.map((scenario) => ({
+ caseId: scenario.id,
+ passed: true,
+ }));
+ const partial = summarizeRoutingCases(results.slice(0, 1));
+ assert.equal(partial.passed, false);
+ assert.equal(partial.status, "incomplete");
+ assert.deepEqual(
+ partial.missingCases,
+ cases.slice(1).map((scenario) => scenario.id),
+ );
+ const interrupted = summarizeRoutingCases(results.slice(0, 1), {
+ interrupted: true,
+ });
+ assert.equal(interrupted.status, "interrupted");
+ assert.equal(interrupted.passed, false);
+ assert.equal(
+ summarizeRoutingCases([], { error: "preflight failed" }).status,
+ "failed",
+ );
+ assert.equal(summarizeRoutingCases(results).passed, true);
+ assert.equal(
+ summarizeRoutingCases(results, { interrupted: true }).passed,
+ false,
+ );
+});
diff --git a/sdk/typescript/package.json b/sdk/typescript/package.json
index dd07ff7d9..2a45bcc74 100644
--- a/sdk/typescript/package.json
+++ b/sdk/typescript/package.json
@@ -65,7 +65,7 @@
"build:plugin": "node scripts/build-plugin.mjs",
"check:plugin-source": "node scripts/check-plugin-source.mjs",
"check:package": "node scripts/check-package.mjs",
- "format": "prettier --check --ignore-path .gitignore --ignore-path .prettierignore \"**/*.{cjs,mjs,js,ts,mts,tsx,json}\" \"../../.github/scripts/*.mts\" \"../../examples/custom-validation/*.mts\" \"../../plugins/codex-security/native/*.mts\" \"../../evals/{deep-reducer,secret-discovery}/*.mts\" \"../../evals/triage-finding/{assertions,scripts,sastbench/assertions,sastbench/scripts}/*.mts\" \"../../evals/triage-finding/types.ts\" \"../../evals/tsconfig.json\" && node scripts/check-markdown.mjs",
+ "format": "prettier --check --ignore-path .gitignore --ignore-path .prettierignore \"**/*.{cjs,mjs,js,ts,mts,tsx,json}\" \"../../.github/scripts/*.mts\" \"../../examples/custom-validation/*.mts\" \"../../plugins/codex-security/native/*.mts\" \"../../evals/{deep-reducer,secret-discovery,workflow-routing}/*.mts\" \"../../evals/triage-finding/{assertions,scripts,sastbench/assertions,sastbench/scripts}/*.mts\" \"../../evals/triage-finding/types.ts\" \"../../evals/tsconfig.json\" && node scripts/check-markdown.mjs",
"generate:models": "node scripts/generate-models.cjs",
"generate:models:check": "node scripts/generate-models.cjs --check",
"lint": "tsc --noEmit",