OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security
TypeScript ai-securityapplication-securityclicode-scanningcodexcodex-securitycybersecuritydevsecopsnodejsnpmopenaisdksecuritytypescriptvulnerability-scanning
test(evals): measure natural-language security skill routing (#1700)
Michael D'Angelof905ce3ccd0e
diff --git a/.github/workflows/node-ci.yml b/.github/workflows/node-ci.ymlindex 433c391b9..1205b27ab 100644--- a/.github/workflows/node-ci.yml+++ b/.github/workflows/node-ci.yml@@ -469,6 +469,9 @@ jobs:- name: Test secret discovery eval contractsif: matrix.os == 'ubuntu-latest' && matrix.python == '3.12'run: node --experimental-strip-types --test evals/secret-discovery/test_*.mts+ - name: Test workflow routing eval contracts+ if: matrix.os == 'ubuntu-latest' && matrix.python == '3.12'+ run: node --experimental-strip-types --test evals/workflow-routing/test_*.mtsrequired-test:# Skipped metadata checks must not replace required source check results.diff --git a/evals/README.md b/evals/README.mdindex ae456816e..7a4ad74c2 100644--- a/evals/README.md+++ b/evals/README.md@@ -11,7 +11,10 @@ being embedded in its source tree or shipped npm runtime.- [Secret discovery](secret-discovery/README.md): checks whether the productioncore audit finds synthetic credentials in source and keeps them in its finalfindings, with deterministic grading and harness checks.+- [Workflow routing](workflow-routing/README.md): checks whether repository-installed+ production skills route natural-language requests to the appropriate workflow,+ stopping before workflow execution.-Model runs are opt-in. CI runs the deterministic triage and secret-discovery-helper checks and the real-IPC reducer regression through the normal MCP test-suite.+Model runs are opt-in. CI runs the deterministic triage, secret-discovery, and+workflow-routing helper checks and the real-IPC reducer regression through the+normal MCP test suite.diff --git a/evals/tsconfig.json b/evals/tsconfig.jsonindex 0aed99eab..52d5541c7 100644--- a/evals/tsconfig.json+++ b/evals/tsconfig.json@@ -6,6 +6,7 @@"include": ["deep-reducer/*.mts","secret-discovery/*.mts",+ "workflow-routing/*.mts","triage-finding/assertions/*.mts","triage-finding/scripts/*.mts","triage-finding/sastbench/assertions/*.mts",diff --git a/evals/workflow-routing/.gitignore b/evals/workflow-routing/.gitignorenew file mode 100644index 000000000..30846059d--- /dev/null+++ b/evals/workflow-routing/.gitignore@@ -0,0 +1 @@+/reports/diff --git a/evals/workflow-routing/README.md b/evals/workflow-routing/README.mdnew file mode 100644index 000000000..3710d386f--- /dev/null+++ b/evals/workflow-routing/README.md@@ -0,0 +1,83 @@+# Repository-installed skill routing eval++This opt-in cohort checks whether a fresh Codex thread loads the appropriate+production security skill for a natural-language request. The request does not+name a skill. The eval stops after workflow selection and loading, before source+review, scan startup, triage, patching, or verification.++| Request | Expected skill |+| ------------------------------------------ | -------------------- |+| Ordinary repository security review | `security-scan` |+| Security review of uncommitted changes | `security-diff-scan` |+| Explicit exhaustive, multi-pass scan | `deep-security-scan` |+| Assess a supplied vulnerability claim | `triage-finding` |+| Implement and test a security fix | `fix-finding` |+| Explicitly verify an existing security fix | `verify-fix` |+| Fix an ordinary sorting bug | None |+| Check a refactor's build and unit tests | None |++The negative controls catch accidental security-workflow activation. In+particular, testing a fix does not automatically request the standalone+`verify-fix` workflow, and an ordinary repository scan does not request Deep Scan.++## Run++Use the repository's normal SDK and MCP dependency setup, then:++```sh+pnpm --dir sdk/typescript run build:evals+node --experimental-strip-types evals/workflow-routing/run.mts+```++An optional positional argument selects a model. Without one, Codex selects its+default; the SDK does not expose that resolved model. Model runs consume usage+and are not part of CI.++The runner reuses the secret-discovery eval's private-home authentication,+restricted permission profile, native preflight, cancellation draining, and+cleanup. See its [runtime notes](../secret-discovery/README.md#run), including+saved-login refresh behavior and the Windows elevated-sandbox requirement. It+also isolates `HOME` so ambient user skills do not enter discovery. Each case+starts a fresh thread with the fixture as its authoritative working directory.++All current production skills and their references are copied unchanged into+the temporary repository's `.agents` directory. Expected routes and the grader+remain outside the readable fixture. Plugins, connected apps, network access,+memories, and subagents remain disabled. No application source or real findings+are supplied; the invoice claim is synthetic and is not evaluated as a finding.++## Evidence and limits++A routing pass requires a completed turn and a successful shell-tool trace that+returns the expected skill's production heading and introductory instructions.+The grader accepts ordinary or line-numbered reads. An agent's claimed choice,+a filename listing, failed read, or reading every skill does not satisfy the+positive cases. Loading another security skill fails both positive and negative+cases. The final response is retained for review, not used as proof of selection.++`report.json` records each route, turn outcome, full-turn token usage, elapsed+time, failed/repeated commands, repeated skill reads, final response, source+revision and dirty state, package/SDK/CLI versions, requested model, prompt hash,+and hashes of all staged skill entrypoints. Per-case JSONL files retain SDK events.+Artifacts are saved under ignored `reports/`. Cohort status and missing-case IDs+distinguish a partial or interrupted run from all-case success. A failed or incomplete turn cannot+pass; unavailable usage remains `null`. No monetary cost or efficiency threshold+is inferred from token counts.++This checks repository skill discovery and loading, not marketplace installation,+the packed npm layout, successful workflow execution, target handoff, saved+artifacts, finding quality, or missing-tool recovery. The harness explicitly+stops at the routing boundary; a passing case must not be presented as a completed+security task. It does not parse arbitrary shell programs to prove what they+read. Unsupported command-output formats can fail the content-based grader and+should be reviewed in the retained transcript before changing the assertion.++## Deterministic checks++```sh+node --experimental-strip-types --test evals/workflow-routing/test_*.mts+```++Normal CI checks unchanged production staging, each expected route, negative+controls, false-positive trace rejection, failures, cancellation, and SDK thread+setup. These checks validate the harness; they do not establish model accuracy.diff --git a/evals/workflow-routing/cases.mts b/evals/workflow-routing/cases.mtsnew file mode 100644index 000000000..520437d23--- /dev/null+++ b/evals/workflow-routing/cases.mts@@ -0,0 +1,58 @@+export const cases = [+ {+ id: "repository-default",+ request: "Check this repository for security vulnerabilities.",+ skill: "security-scan",+ },+ {+ id: "working-tree-diff",+ request: "Review my uncommitted changes for security vulnerabilities.",+ skill: "security-diff-scan",+ },+ {+ id: "explicit-deep",+ request: "Do an exhaustive, multi-pass security scan of this repository.",+ skill: "deep-security-scan",+ },+ {+ id: "existing-finding",+ request:+ "Triage this reported vulnerability against this repository: GET /invoices/:id exposes another user's invoice without checking ownership. Assess the claim; do not fix it.",+ skill: "triage-finding",+ },+ {+ id: "explicit-security-fix",+ request:+ "Fix and test this security finding: GET /invoices/:id exposes another user's invoice without checking ownership. Preserve legitimate owner access.",+ skill: "fix-finding",+ },+ {+ id: "explicit-fix-verification",+ request:+ "Verify that the existing security fix closes the reported cross-user invoice access in GET /invoices/:id. Check legitimate owner access too. Do not modify code.",+ skill: "verify-fix",+ },+ {+ id: "ordinary-bug-fix",+ request:+ "Fix the invoice table's incorrect alphabetical sort order and run its tests.",+ skill: null,+ },+ {+ id: "ordinary-test-run",+ request: "Verify that this refactor builds and its unit tests pass.",+ skill: null,+ },+] as const;++export type RoutingCase = (typeof cases)[number];++export function routingPrompt(scenario: RoutingCase) {+ return `${scenario.request}++This is a routing-only evaluation. Use the installed skills as you normally+would for the request above. Read the applicable workflow's SKILL.md, if any,+then stop before executing the workflow. Do not inspect application source,+run tests, edit files, start scans, or contact services. Briefly state the next+step; do not claim that the requested work has been completed.`;+}diff --git a/evals/workflow-routing/harness.mts b/evals/workflow-routing/harness.mtsnew file mode 100644index 000000000..0c69831fb--- /dev/null+++ b/evals/workflow-routing/harness.mts@@ -0,0 +1,222 @@+import { createHash } from "node:crypto";+import { cp, mkdir, readFile, readdir } from "node:fs/promises";+import { join } from "node:path";+import type {+ ThreadEvent,+ ThreadOptions,+} from "../../sdk/typescript/node_modules/@openai/codex-sdk/dist/index.js";+import { codexSettings, threadSettings } from "../secret-discovery/harness.mts";+import { deepScanPermissionProfileFallbackError } from "../secret-discovery/runtime.mts";+import { cases, routingPrompt, type RoutingCase } from "./cases.mts";++const pluginRoot = join(import.meta.dirname, "../../plugins/codex-security");+export const sha256 = (text: string) =>+ createHash("sha256").update(text).digest("hex");++export interface Skill {+ name: string;+ sha256: string;+ heading: string;+ introduction: string;+}++/** Install unchanged repository skills; labels and grader stay outside this tree. */+export async function prepareRoutingEval(root: string) {+ const repo = join(root, "repository");+ const runtime = join(repo, ".agents");+ await mkdir(runtime, { recursive: true });+ for (const directory of ["skills", "references"]) {+ await cp(join(pluginRoot, directory), join(runtime, directory), {+ recursive: true,+ });+ }+ const skills: Skill[] = [];+ for (const name of (await readdir(join(runtime, "skills"))).sort()) {+ const source = await readFile(+ join(runtime, "skills", name, "SKILL.md"),+ "utf8",+ );+ const body = source.replace(/^---\r?\n[\s\S]*?\r?\n---\r?\n/u, "");+ const paragraphs = body.trim().split(/\r?\n\s*\r?\n/u);+ const heading = paragraphs[0];+ const introduction = paragraphs.find(+ (paragraph) => !paragraph.startsWith("#"),+ );+ if (!heading?.startsWith("# ") || !introduction)+ throw new Error(`Cannot identify production skill content: ${name}`);+ skills.push({ name, sha256: sha256(source), heading, introduction });+ }+ return { repo, runtime, skills };+}++export function routingSettings(+ home: string,+ codexPath: string,+ environment = process.env,+) {+ // Repository skills are the only skill installation in this eval. Keep the+ // caller's ~/.agents/skills out of discovery as well as their Codex config.+ return codexSettings(home, codexPath, {+ ...environment,+ HOME: home,+ ...(process.platform === "win32" ? { USERPROFILE: home } : {}),+ });+}++function unnumbered(text: string) {+ return text.replace(/\r\n/gu, "\n").replace(/^\s*\d+\t/gmu, "");+}++/** Recognize returned production content, not a filename or a claimed choice. */+export function gradeRoutingTrace(+ expected: string | null,+ skills: readonly Skill[],+ events: readonly ThreadEvent[],+) {+ const reads: string[] = [];+ const commands: string[] = [];+ let failedCommands = 0;+ let unexpectedActions = 0;+ let failed = false;+ let completed = false;+ let finalResponse = "";+ for (const event of events) {+ if (event.type === "turn.completed") completed = true;+ if (event.type === "turn.failed" || event.type === "error") failed = true;+ if (event.type !== "item.completed") continue;+ const { item } = event;+ if (item.type === "agent_message") finalResponse = item.text;+ if (["file_change", "mcp_tool_call", "web_search"].includes(item.type))+ unexpectedActions++;+ if (item.type !== "command_execution") continue;+ commands.push(item.command);+ if (item.exit_code !== 0 || item.status !== "completed") {+ failedCommands++;+ continue;+ }+ const output = unnumbered(item.aggregated_output);+ for (const skill of skills) {+ if (output.includes(skill.heading) && output.includes(skill.introduction))+ reads.push(skill.name);+ }+ }+ const loadedSkills = [...new Set(reads)];+ const expectedSkills = expected === null ? [] : [expected];+ return {+ passed:+ completed &&+ !failed &&+ finalResponse.trim().length > 0 &&+ unexpectedActions === 0 &&+ JSON.stringify(loadedSkills) === JSON.stringify(expectedSkills),+ turnCompleted: completed,+ turnFailed: failed,+ expectedSkill: expected,+ loadedSkills,+ commandCount: commands.length,+ failedCommands,+ repeatedCommands: commands.length - new Set(commands).size,+ repeatedSkillReads: reads.length - loadedSkills.length,+ unexpectedActions,+ finalResponse,+ };+}++export interface RoutingCodex {+ startThread(options: ThreadOptions): {+ runStreamed(+ prompt: string,+ options: { signal: AbortSignal },+ ): Promise<{ events: AsyncIterable<ThreadEvent> }>;+ };+}++export async function runRoutingCase(+ prepared: Awaited<ReturnType<typeof prepareRoutingEval>>,+ scenario: RoutingCase,+ codex: RoutingCodex,+ { model, signal }: { model?: string; signal?: AbortSignal } = {},+) {+ const prompt = routingPrompt(scenario);+ const events: ThreadEvent[] = [];+ const controller = new AbortController();+ const combined = signal+ ? AbortSignal.any([signal, controller.signal])+ : controller.signal;+ const started = performance.now();+ let error: string | undefined;+ try {+ const thread = codex.startThread(threadSettings(prepared, model));+ const stream = await thread.runStreamed(prompt, { signal: combined });+ for await (const event of stream.events) {+ events.push(event);+ const warning =+ event.type === "error"+ ? event.message+ : event.type === "item.completed" && event.item.type === "error"+ ? event.item.message+ : undefined;+ const fallback = deepScanPermissionProfileFallbackError(warning);+ if (fallback) {+ error ??= fallback.message;+ controller.abort(fallback);+ }+ // Drain after cancellation before the shared harness removes temp state.+ }+ combined.throwIfAborted();+ } catch (failure) {+ error ??= failure instanceof Error ? failure.message : String(failure);+ }+ const usage = events.findLast(+ (event) => event.type === "turn.completed",+ )?.usage;+ const grade = gradeRoutingTrace(scenario.skill, prepared.skills, events);+ return {+ events,+ report: {+ caseId: scenario.id,+ ...grade,+ passed: grade.passed && error === undefined,+ ...(error ? { error } : {}),+ elapsedMilliseconds: Math.round(performance.now() - started),+ usage: usage ?? null,+ promptSha256: sha256(prompt),+ },+ };+}++/** Partial or interrupted cohorts cannot be mistaken for all-case success. */+export function summarizeRoutingCases(+ results: readonly { caseId: string; passed: boolean }[],+ {+ interrupted = false,+ error,+ }: { interrupted?: boolean; error?: string } = {},+) {+ const missingCases = cases+ .filter(+ (scenario) => !results.some((result) => result.caseId === scenario.id),+ )+ .map((scenario) => scenario.id);+ const complete = missingCases.length === 0 && results.length === cases.length;+ const passed =+ complete &&+ !interrupted &&+ error === undefined &&+ results.every((result) => result.passed);+ return {+ status: interrupted+ ? "interrupted"+ : error+ ? "failed"+ : !complete+ ? "incomplete"+ : passed+ ? "passed"+ : "failed",+ passed,+ expectedCases: cases.map((scenario) => scenario.id),+ missingCases,+ ...(error ? { error } : {}),+ };+}diff --git a/evals/workflow-routing/run.mts b/evals/workflow-routing/run.mtsnew file mode 100644index 000000000..da1fc3330--- /dev/null+++ b/evals/workflow-routing/run.mts@@ -0,0 +1,119 @@+import { execFileSync } from "node:child_process";+import {+ mkdir,+ mkdtemp,+ readFile,+ realpath,+ writeFile,+} from "node:fs/promises";+import { join } from "node:path";+import { Codex } from "../../sdk/typescript/node_modules/@openai/codex-sdk/dist/index.js";+import * as auth from "../../sdk/typescript/dist/auth.js";+import {+ createIsolatedHome,+ resolveCodexCommand,+} from "../../sdk/typescript/dist/runtime.js";+import { preflightEval } from "../secret-discovery/harness.mts";+import { withEvalState } from "../secret-discovery/runtime.mts";+import { cases } from "./cases.mts";+import {+ prepareRoutingEval,+ routingSettings,+ runRoutingCase,+ summarizeRoutingCases,+ type Skill,+} from "./harness.mts";++const reports = join(import.meta.dirname, "reports");+await mkdir(reports, { recursive: true });+const directory = await mkdtemp(join(reports, "run-"));+const model = process.argv[2];+const sourceRoot = join(import.meta.dirname, "../..");+const git = (...args: string[]) =>+ execFileSync("git", args, { cwd: sourceRoot, encoding: "utf8" }).trim();+const packageVersion = async (file: string) =>+ JSON.parse(await readFile(join(sourceRoot, file), "utf8")).version as string;+const identity = {+ sourceRevision: git("rev-parse", "HEAD"),+ sourceDirty: git("status", "--porcelain", "--untracked-files=normal") !== "",+ packageVersion: await packageVersion("sdk/typescript/package.json"),+ codexSdkVersion: await packageVersion(+ "sdk/typescript/node_modules/@openai/codex-sdk/package.json",+ ),+ codexCliVersion: await packageVersion(+ "sdk/typescript/node_modules/@openai/codex/package.json",+ ),+ requestedModel: model ?? null,+ modelSelection: model+ ? "explicit"+ : "Codex default; SDK does not expose resolved model",+ scope:+ "repository-installed skill routing; workflow execution is not evaluated",+};+const results: Awaited<ReturnType<typeof runRoutingCase>>["report"][] = [];+let skills: Skill[] = [];+let interrupted = false;+let error: string | undefined;+const saveReport = () =>+ writeFile(+ join(directory, "report.json"),+ JSON.stringify(+ {+ ...identity,+ ...summarizeRoutingCases(results, { interrupted, error }),+ skills,+ results,+ },+ null,+ 2,+ ) + "\n",+ );+console.log(`Eval artifacts: ${directory}`);+await saveReport();+try {+ await withEvalState(+ createIsolatedHome,+ (+ auth as unknown as typeof import("../../sdk/typescript/src/auth.js")+ ).configuredCodexHome(process.env),+ async ({ root, home, signal }) => {+ try {+ const prepared = await prepareRoutingEval(root);+ skills = prepared.skills;+ const codexPath = await realpath(resolveCodexCommand({}).command);+ const settings = await preflightEval(+ prepared,+ routingSettings(home, codexPath),+ signal,+ );+ const codex = new Codex(settings);+ for (const scenario of cases) {+ signal.throwIfAborted();+ const { events, report } = await runRoutingCase(+ prepared,+ scenario,+ codex,+ { model, signal },+ );+ await writeFile(+ join(directory, `${scenario.id}.jsonl`),+ events.map((event) => JSON.stringify(event)).join("\n") + "\n",+ );+ results.push(report);+ await saveReport();+ console.log(`${scenario.id}: ${report.passed ? "PASS" : "FAIL"}`);+ }+ process.exitCode = results.every((result) => result.passed) ? 0 : 1;+ } finally {+ interrupted = signal.aborted;+ }+ },+ );+} catch (failure) {+ error = failure instanceof Error ? failure.message : String(failure);+ throw failure;+} finally {+ // withEvalState can finish before its callback after an early signal.+ interrupted ||= process.exitCode === 130 || process.exitCode === 143;+ await saveReport();+}diff --git a/evals/workflow-routing/test_routing.mts b/evals/workflow-routing/test_routing.mtsnew file mode 100644index 000000000..2a4cb2c85--- /dev/null+++ b/evals/workflow-routing/test_routing.mts@@ -0,0 +1,328 @@+import assert from "node:assert/strict";+import { mkdtemp, readFile, readdir, rm } from "node:fs/promises";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { after, test } from "node:test";+import type { ThreadEvent } from "../../sdk/typescript/node_modules/@openai/codex-sdk/dist/index.js";+import { cases, routingPrompt } from "./cases.mts";+import {+ gradeRoutingTrace,+ prepareRoutingEval,+ routingSettings,+ runRoutingCase,+ sha256,+ summarizeRoutingCases,+ type Skill,+} from "./harness.mts";++const root = await mkdtemp(join(tmpdir(), "routing-contract-"));+after(() => rm(root, { recursive: true, force: true }));+const prepared = await prepareRoutingEval(root);+const skill = (name: string) =>+ prepared.skills.find((entry) => entry.name === name)!;+const message: ThreadEvent = {+ type: "item.completed",+ item: {+ id: "message",+ type: "agent_message",+ text: "Ready for the next step.",+ },+};+const completed: ThreadEvent = {+ type: "turn.completed",+ usage: {+ input_tokens: 10,+ cached_input_tokens: 0,+ cache_write_input_tokens: 0,+ output_tokens: 5,+ reasoning_output_tokens: 0,+ },+};+function read(+ entry: Skill,+ output = `${entry.heading}\n\n${entry.introduction}`,+): ThreadEvent {+ return {+ type: "item.completed",+ item: {+ id: "read",+ type: "command_execution",+ command: `cat .agents/skills/${entry.name}/SKILL.md`,+ aggregated_output: output,+ status: "completed",+ exit_code: 0,+ },+ };+}+const grade = (expected: string | null, events: ThreadEvent[]) =>+ gradeRoutingTrace(expected, prepared.skills, events);++test("stages every production skill unchanged without the labels or harness", async () => {+ assert.deepEqual(await readdir(prepared.repo), [".agents"]);+ assert.deepEqual((await readdir(prepared.runtime)).sort(), [+ "references",+ "skills",+ ]);+ for (const entry of prepared.skills) {+ const source = await readFile(+ join(+ import.meta.dirname,+ "../../plugins/codex-security/skills",+ entry.name,+ "SKILL.md",+ ),+ "utf8",+ );+ const staged = await readFile(+ join(prepared.runtime, "skills", entry.name, "SKILL.md"),+ "utf8",+ );+ assert.equal(staged, source);+ assert.equal(entry.sha256, sha256(source));+ }+ for (const scenario of cases) {+ if (scenario.skill) assert.ok(skill(scenario.skill));+ assert.ok(!routingPrompt(scenario).includes(scenario.id));+ assert.ok(!routingPrompt(scenario).includes("$"));+ }+});++for (const scenario of cases) {+ test(`accepts the expected observed route: ${scenario.id}`, () => {+ const reads = scenario.skill ? [read(skill(scenario.skill))] : [];+ assert.equal(+ grade(scenario.skill, [...reads, message, completed]).passed,+ true,+ );+ });+}++test("a claimed choice or filename listing cannot satisfy a skill read", () => {+ const expected = skill("security-scan");+ for (const events of [+ [message, completed],+ [+ read(expected, ".agents/skills/security-scan/SKILL.md"),+ message,+ completed,+ ],+ [+ read(expected, `${expected.heading}\nname: security-scan`),+ message,+ completed,+ ],+ ]) {+ assert.equal(grade(expected.name, events).passed, false);+ }+});++test("accepts numbered source output but rejects failed or unfinished reads", () => {+ const expected = skill("verify-fix");+ const output = `${expected.heading}\n\n${expected.introduction}`+ .split("\n")+ .map((line, index) => `${index + 1}\t${line}`)+ .join("\n");+ assert.equal(+ grade(expected.name, [read(expected, output), message, completed]).passed,+ true,+ );+ for (const patch of [+ { exit_code: 1 },+ { exit_code: undefined },+ { status: "failed" },+ { status: "in_progress" },+ ]) {+ const event = read(expected);+ if (event.type === "item.completed") Object.assign(event.item, patch);+ assert.equal(+ grade(expected.name, [event, message, completed]).passed,+ false,+ );+ }+});++test("rejects deep-by-default, implicit verification, and loading every skill", () => {+ for (const [expected, loaded] of [+ ["security-scan", ["deep-security-scan"]],+ ["fix-finding", ["fix-finding", "verify-fix"]],+ ["security-diff-scan", prepared.skills.map((entry) => entry.name)],+ [null, ["fix-finding"]],+ [null, ["verify-fix"]],+ ] as const) {+ assert.equal(+ grade(expected, [+ ...loaded.map((name) => read(skill(name))),+ message,+ completed,+ ]).passed,+ false,+ );+ }+});++test("recognizes content returned by glob reads in positive and negative cases", () => {+ const event = read(skill("security-scan"));+ if (+ event.type === "item.completed" &&+ event.item.type === "command_execution"+ )+ event.item.command = "cat .agents/skills/security-scan/*";+ assert.equal(+ grade("security-scan", [event, message, completed]).passed,+ true,+ );+ assert.equal(grade(null, [event, message, completed]).passed, false);+});++test("failed or incomplete turns cannot pass even with the correct skill", () => {+ const observed = [read(skill("security-scan")), message];+ assert.equal(grade("security-scan", observed).passed, false);+ assert.equal(+ grade("security-scan", [+ ...observed,+ { type: "turn.failed", error: { message: "fixture failure" } },+ completed,+ ]).passed,+ false,+ );+ assert.equal(+ grade("security-scan", [read(skill("security-scan")), completed]).passed,+ false,+ );+});++test("records repeated reads and failed commands without pretending to measure completion", () => {+ const observed = read(skill("security-scan"));+ const failure = read(skill("security-scan"));+ if (failure.type === "item.completed")+ Object.assign(failure.item, { exit_code: 1, status: "failed" });+ const result = grade("security-scan", [+ failure,+ observed,+ observed,+ message,+ completed,+ ]);+ assert.equal(result.passed, true);+ assert.equal(result.failedCommands, 1);+ assert.equal(result.repeatedCommands, 2);+ assert.equal(result.repeatedSkillReads, 1);+});++test("rejects workflow execution visible in non-shell tool traces", () => {+ const event: ThreadEvent = {+ type: "item.completed",+ item: {+ id: "write",+ type: "file_change",+ status: "completed",+ changes: [{ path: "app.js", kind: "update" }],+ },+ };+ assert.equal(+ grade("fix-finding", [+ read(skill("fix-finding")),+ event,+ message,+ completed,+ ]).passed,+ false,+ );+});++test("uses an isolated home for Codex config and ambient skill discovery", () => {+ const settings = routingSettings(+ join(root, "home"),+ join(root, "bin", "codex"),+ { HOME: "/ambient", PATH: "/usr/bin" },+ );+ assert.equal(settings.env.HOME, join(root, "home"));+ assert.equal(settings.env.CODEX_HOME, join(root, "home"));+ assert.ok(settings.configOverrides.includes("features.plugins=false"));+});++test("starts a fresh thread in the fixture and grades its actual events", async () => {+ const events = [read(skill("security-scan")), message, completed];+ const result = await runRoutingCase(+ prepared,+ cases[0],+ {+ startThread(options) {+ assert.equal(options.workingDirectory, prepared.repo);+ assert.equal(options.model, "fixture-model");+ return {+ async runStreamed(prompt) {+ assert.equal(prompt, routingPrompt(cases[0]));+ return {+ events: (async function* () {+ yield* events;+ })(),+ };+ },+ };+ },+ },+ { model: "fixture-model" },+ );+ assert.equal(result.report.passed, true);+ assert.equal(result.report.usage?.input_tokens, 10);+ assert.deepEqual(result.events, events);+});++test("keeps an interrupted result failed while draining its stream", async () => {+ const controller = new AbortController();+ let drained = false;+ const result = await runRoutingCase(+ prepared,+ cases[0],+ {+ startThread() {+ return {+ async runStreamed() {+ return {+ events: (async function* () {+ yield read(skill("security-scan"));+ controller.abort(new Error("fixture cancellation"));+ yield message;+ yield completed;+ drained = true;+ })(),+ };+ },+ };+ },+ },+ { signal: controller.signal },+ );+ assert.equal(drained, true);+ assert.equal(result.report.passed, false);+ assert.equal(result.report.error, "fixture cancellation");+});++test("cohort reports distinguish partial, interrupted, failed, and complete results", () => {+ const results = cases.map((scenario) => ({+ caseId: scenario.id,+ passed: true,+ }));+ const partial = summarizeRoutingCases(results.slice(0, 1));+ assert.equal(partial.passed, false);+ assert.equal(partial.status, "incomplete");+ assert.deepEqual(+ partial.missingCases,+ cases.slice(1).map((scenario) => scenario.id),+ );+ const interrupted = summarizeRoutingCases(results.slice(0, 1), {+ interrupted: true,+ });+ assert.equal(interrupted.status, "interrupted");+ assert.equal(interrupted.passed, false);+ assert.equal(+ summarizeRoutingCases([], { error: "preflight failed" }).status,+ "failed",+ );+ assert.equal(summarizeRoutingCases(results).passed, true);+ assert.equal(+ summarizeRoutingCases(results, { interrupted: true }).passed,+ false,+ );+});diff --git a/sdk/typescript/package.json b/sdk/typescript/package.jsonindex dd07ff7d9..2a45bcc74 100644--- a/sdk/typescript/package.json+++ b/sdk/typescript/package.json@@ -65,7 +65,7 @@"build:plugin": "node scripts/build-plugin.mjs","check:plugin-source": "node scripts/check-plugin-source.mjs","check:package": "node scripts/check-package.mjs",- "format": "prettier --check --ignore-path .gitignore --ignore-path .prettierignore \"**/*.{cjs,mjs,js,ts,mts,tsx,json}\" \"../../.github/scripts/*.mts\" \"../../examples/custom-validation/*.mts\" \"../../plugins/codex-security/native/*.mts\" \"../../evals/{deep-reducer,secret-discovery}/*.mts\" \"../../evals/triage-finding/{assertions,scripts,sastbench/assertions,sastbench/scripts}/*.mts\" \"../../evals/triage-finding/types.ts\" \"../../evals/tsconfig.json\" && node scripts/check-markdown.mjs",+ "format": "prettier --check --ignore-path .gitignore --ignore-path .prettierignore \"**/*.{cjs,mjs,js,ts,mts,tsx,json}\" \"../../.github/scripts/*.mts\" \"../../examples/custom-validation/*.mts\" \"../../plugins/codex-security/native/*.mts\" \"../../evals/{deep-reducer,secret-discovery,workflow-routing}/*.mts\" \"../../evals/triage-finding/{assertions,scripts,sastbench/assertions,sastbench/scripts}/*.mts\" \"../../evals/triage-finding/types.ts\" \"../../evals/tsconfig.json\" && node scripts/check-markdown.mjs","generate:models": "node scripts/generate-models.cjs","generate:models:check": "node scripts/generate-models.cjs --check","lint": "tsc --noEmit",