OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security
Repository-installed skill routing eval
This opt-in cohort checks whether a fresh Codex thread loads the appropriate production security skill for a natural-language request. The request does not name a skill. The eval stops after workflow selection and loading, before source review, scan startup, triage, patching, or verification.
See the routing cases for each prompt and expected skill.
The negative controls catch accidental security-workflow activation. In
particular, testing a fix does not automatically request the standalone
verify-fix workflow, and an ordinary repository scan does not request Deep Scan.
Run
Build the evaluation tools, then run the commands below from the repository root.
Deterministic checks
Run the CI checks for staging, expected routes, false-positive trace rejection, cancellation, and SDK thread setup without making model calls:
node --experimental-strip-types --test evals/workflow-routing/test_*.mts
Run a model evaluation
To measure model routing accuracy, run the following command from the same directory. It consumes model usage. Pass a model name as a positional argument to select one explicitly. If omitted, Codex selects a default and the SDK does not report its name.
node --experimental-strip-types evals/workflow-routing/run.mts
The runner reuses the secret-discovery eval's private-home authentication,
restricted permission profile, native preflight, cancellation draining, and
cleanup. See its runtime notes, including
saved-login refresh behavior and the Windows elevated-sandbox requirement. It
also isolates HOME so ambient user skills do not enter discovery. Each case
starts a fresh thread with the fixture as its authoritative working directory.
All current production skills and their references are copied unchanged into
the temporary repository's .agents directory. Expected routes and the grader
remain outside the readable fixture. Plugins, connected apps, network access,
memories, and subagents remain disabled. No application source or real findings
are supplied; the invoice claim is synthetic and is not evaluated as a finding.
Evidence and limits
A routing pass requires a completed turn and a successful shell-tool trace that returns the expected skill's production heading and introductory instructions. The grader accepts ordinary or line-numbered reads. An agent's claimed choice, a filename listing, failed read, or reading every skill does not satisfy the positive cases. Loading another security skill fails both positive and negative cases. The final response is retained for review, not used as proof of selection.
report.json records each route, turn outcome, full-turn token usage, elapsed
time, failed/repeated commands, repeated skill reads, final response, source
revision and dirty state, package/SDK/CLI versions, requested model, prompt hash,
and hashes of all staged skill entrypoints. Per-case JSONL files retain SDK events.
Artifacts are saved under ignored reports/. Cohort status and missing-case IDs
distinguish a partial or interrupted run from all-case success. A failed or incomplete turn cannot
pass; unavailable usage remains null. No monetary cost or efficiency threshold
is inferred from token counts.
This checks repository skill discovery and loading, not marketplace installation, the packed npm layout, successful workflow execution, target handoff, saved artifacts, finding quality, or missing-tool recovery. The harness explicitly stops at the routing boundary; a passing case must not be presented as a completed security task. It does not parse arbitrary shell programs to prove what they read. Unsupported command-output formats can fail the content-based grader and should be reviewed in the retained transcript before changing the assertion.