OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security
TypeScript ai-securityapplication-securityclicode-scanningcodexcodex-securitycybersecuritydevsecopsnodejsnpmopenaisdksecuritytypescriptvulnerability-scanning
test: relocate triage evals and reducer model entry point (#1056)
mldangelo-oaia3deb87cfc36
diff --git a/.github/dependabot.yml b/.github/dependabot.ymlindex 7c91c7bda..37d4cf2cb 100644--- a/.github/dependabot.yml+++ b/.github/dependabot.yml@@ -4,7 +4,7 @@ updates:directories:- /sdk/typescript- /plugins/codex-security/mcp-app- - /plugins/codex-security/skills/triage-finding/evals+ - /evals/triage-findingschedule:interval: croncronjob: "0 9 * * *"diff --git a/.github/workflows/node-ci.yml b/.github/workflows/node-ci.ymlindex c2edbac72..659528234 100644--- a/.github/workflows/node-ci.yml+++ b/.github/workflows/node-ci.yml@@ -512,7 +512,7 @@ jobs:if-no-files-found: warnretention-days: 14- name: Test deterministic triage eval contracts- working-directory: plugins/codex-security/skills/triage-finding/evals+ working-directory: evals/triage-findingrun: node --run test:deterministicrequired-test:diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 667813f94..34c6cbbe6 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -15,6 +15,10 @@ packages and releases. Do not edit or commit files in that directory. See the[SDK testing guide](sdk/typescript/TESTING.md) for the generation and validationcommands.+Model-based evaluations live in [`evals/`](evals/README.md), outside the plugin+source. Deterministic triage checks and the MCP reducer IPC regression remain+part of normal CI.+Search [existing issues](https://github.com/openai/codex-security/issues)before opening a new one.diff --git a/RELEASING.md b/RELEASING.mdindex d6ca4ae98..199b12e38 100644--- a/RELEASING.md+++ b/RELEASING.md@@ -52,10 +52,10 @@ version-update cooldown.Updates still require review and passing CI; nothing is merged automatically.Keep `@openai/codex` and `@openai/codex-sdk` on the same exact version across the-TypeScript SDK, MCP app, and triage evals. Dependabot groups their updates across-all three projects, and the SDK tests reject mismatched pins or multiple locked-SDK versions. The evals override Promptfoo's transitive Codex SDK to the direct-SDK dependency so it follows the same update.+TypeScript SDK, MCP app, and triage evals in `evals/triage-finding`. Dependabot+groups their updates across all three projects, and the SDK tests reject+mismatched pins or multiple locked SDK versions. The evals override Promptfoo's+transitive Codex SDK to the direct SDK dependency so it follows the same update.Each pnpm project applies the same seven-day age policy to newly resolveddependencies, including transitive packages, with `openai` and `@openai/*` exempt.diff --git a/evals/README.md b/evals/README.mdnew file mode 100644index 000000000..7839a00aa--- /dev/null+++ b/evals/README.md@@ -0,0 +1,13 @@+# Evaluations++Model-based evaluations live here so they can evolve with the plugin without+being embedded in its source tree or shipped npm runtime.++- [Triage finding](triage-finding/README.md): Promptfoo input contracts, OSS+ calibration, and SastBench. This suite owns its private package and lockfile.+- [Deep reducer](deep-reducer/README.md): optional model evaluation of real IPC+ size-limit recovery, pagination, and finding retention. It reuses the MCP+ test helpers and dependencies.++Model runs are opt-in. CI still runs the deterministic triage helper checks and+the real-IPC reducer regression through the normal MCP test suite.diff --git a/evals/deep-reducer/.gitignore b/evals/deep-reducer/.gitignorenew file mode 100644index 000000000..a9a1bd38a--- /dev/null+++ b/evals/deep-reducer/.gitignore@@ -0,0 +1 @@+reports/diff --git a/evals/deep-reducer/README.md b/evals/deep-reducer/README.mdnew file mode 100644index 000000000..15a6a68ee--- /dev/null+++ b/evals/deep-reducer/README.md@@ -0,0 +1,27 @@+# Deep reducer paging eval++This optional model eval runs one reducer against synthetic persisted worker findings and a previous aggregate. It uses the pinned Codex CLI, its external code-mode host, the production reducer prompt, and the production artifact tool handlers. It does not launch a repository scan.++The fixture has a 32 MiB individual semantic field containing quotes, backslashes, newlines, and Unicode. Its old duplicated MCP response is about 114 MiB. The first input read deliberately returns that old response; all subsequent calls use the production paged implementation. The IPC error comes from the real transport, not a mocked error message.++The grader requires a smaller byte budget on the retry with the same cursor and reference, byte-bounded subsequent pages, one successful result submission, every source accounted for exactly once, complete original payloads matching their SHA-256 hashes, and preservation of the previous finding identity and synthesized history.++Install the MCP app's locked dependencies using the repository's normal setup, then run from the repository root:++```sh+node evals/deep-reducer/run.mjs+```++The run uses the caller's normal Codex credentials and configuration and consumes model usage. An optional final argument selects a model; otherwise Codex uses its configured model. This eval is opt-in and does not run in CI. The model receives only the production reducer prompt and must discover and carry out recovery itself. The SDK does not expose every code-mode error event, so the grade relies on the observed oversized response, changed request, and saved result.++The command prints the path to an ignored directory under `evals/deep-reducer/reports/` containing `report.json`, the compact tool-call trace, and the generated artifacts. The fixture is generated at runtime; no large payload files are checked in.++## Deterministic regression test++The same fixture, real transport, and grading helpers live under `plugins/codex-security/mcp-app/tests/support/reducer-paging/`. The normal MCP test suite exercises them with scripted model responses from a local HTTP provider, a private temporary Codex home, and no model credentials or external model requests. It additionally asserts the exact transport diagnostic and checks that the grader rejects missing input pages and collapsed independent findings.++Run that test directly from the repository root:++```sh+node --test plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs+```diff --git a/evals/deep-reducer/run.mjs b/evals/deep-reducer/run.mjsnew file mode 100644index 000000000..81b8c5373--- /dev/null+++ b/evals/deep-reducer/run.mjs@@ -0,0 +1,15 @@+import { mkdir, mkdtemp } from "node:fs/promises";+import path from "node:path";+import { fileURLToPath } from "node:url";+import { runReducerPagingEval } from "../../plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging.mjs";++const reports = fileURLToPath(new URL("./reports/", import.meta.url));+await mkdir(reports, { recursive: true });+const root = await mkdtemp(path.join(reports, "deep-reducer-paging-"));+console.log(`Eval artifacts: ${root}`);+const report = await runReducerPagingEval({+ root,+ mode: "model",+ model: process.argv[2],+});+console.log(JSON.stringify(report, null, 2));diff --git a/plugins/codex-security/skills/triage-finding/evals/.gitignore b/evals/triage-finding/.gitignoresimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/.gitignorerename to evals/triage-finding/.gitignorediff --git a/plugins/codex-security/skills/triage-finding/evals/README.md b/evals/triage-finding/README.mdsimilarity index 80%rename from plugins/codex-security/skills/triage-finding/evals/README.mdrename to evals/triage-finding/README.mdindex 23e567763..e21c3ab23 100644--- a/plugins/codex-security/skills/triage-finding/evals/README.md+++ b/evals/triage-finding/README.md@@ -8,36 +8,36 @@ The suite uses the Promptfoo Codex SDK provider because it only needs final assiUse Node.js 22.22.0 or newer for the eval runner. Run these commands from the repository root to install dependencies under this eval directory.-The eval directory has its own `pnpm-workspace.yaml` so pnpm treats it as a small standalone workspace instead of joining the root OpenAI monorepo workspace.+The eval directory has its own `pnpm-workspace.yaml` so pnpm treats it as a small standalone workspace, independent of the SDK and MCP app packages.Install the local eval runner:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run setup+pnpm --dir evals/triage-finding run setup```Validate the config:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run validate+pnpm --dir evals/triage-finding run validate```Run the full eval:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval+pnpm --dir evals/triage-finding run eval```Run the first case as a quick smoke test:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:smoke+pnpm --dir evals/triage-finding run eval:smoke```Run one case while iterating:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval --filter-metadata case_id=sarif-redirects+pnpm --dir evals/triage-finding run eval --filter-metadata case_id=sarif-redirects```The eval target is `fixtures/repo`, a small synthetic Express app with both true positive and false positive/review cases. Assertions are deterministic:@@ -60,43 +60,43 @@ ELI5: the dataset says "this exact old commit should be affected" and "this exacValidate the dataset structure:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run test:dataset+pnpm --dir evals/triage-finding run test:dataset```Run all deterministic calibration checks:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run test:calibration+pnpm --dir evals/triage-finding run test:calibration```Regenerate the Promptfoo calibration tests:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run calibration:generate+pnpm --dir evals/triage-finding run calibration:generate```Hydrate the local OSS checkouts:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run calibration:hydrate+pnpm --dir evals/triage-finding run calibration:hydrate```Validate the calibration Promptfoo config:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run validate:calibration+pnpm --dir evals/triage-finding run validate:calibration```Run one OSS variant as a smoke eval:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:calibration:smoke+pnpm --dir evals/triage-finding run eval:calibration:smoke```Run the full OSS calibration eval:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:calibration+pnpm --dir evals/triage-finding run eval:calibration```The calibration config is separate from the default synthetic eval:diff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/calibration-evidence.js b/evals/triage-finding/assertions/calibration-evidence.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/assertions/calibration-evidence.jsrename to evals/triage-finding/assertions/calibration-evidence.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/github-rest-intake.js b/evals/triage-finding/assertions/github-rest-intake.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/assertions/github-rest-intake.jsrename to evals/triage-finding/assertions/github-rest-intake.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/missing-input.js b/evals/triage-finding/assertions/missing-input.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/assertions/missing-input.jsrename to evals/triage-finding/assertions/missing-input.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/ticket-intake.js b/evals/triage-finding/assertions/ticket-intake.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/assertions/ticket-intake.jsrename to evals/triage-finding/assertions/ticket-intake.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/triage-io.js b/evals/triage-finding/assertions/triage-io.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/assertions/triage-io.jsrename to evals/triage-finding/assertions/triage-io.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/datasets/README.md b/evals/triage-finding/datasets/README.mdsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/datasets/README.mdrename to evals/triage-finding/datasets/README.mddiff --git a/plugins/codex-security/skills/triage-finding/evals/datasets/triage-calibration-seed.json b/evals/triage-finding/datasets/triage-calibration-seed.jsonsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/datasets/triage-calibration-seed.jsonrename to evals/triage-finding/datasets/triage-calibration-seed.jsondiff --git a/plugins/codex-security/skills/triage-finding/evals/fixtures/repo/package-lock.json b/evals/triage-finding/fixtures/repo/package-lock.jsonsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/fixtures/repo/package-lock.jsonrename to evals/triage-finding/fixtures/repo/package-lock.jsondiff --git a/plugins/codex-security/skills/triage-finding/evals/fixtures/repo/package.json b/evals/triage-finding/fixtures/repo/package.jsonsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/fixtures/repo/package.jsonrename to evals/triage-finding/fixtures/repo/package.jsondiff --git a/plugins/codex-security/skills/triage-finding/evals/fixtures/repo/src/server.js b/evals/triage-finding/fixtures/repo/src/server.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/fixtures/repo/src/server.jsrename to evals/triage-finding/fixtures/repo/src/server.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/fixtures/repo/tests/unsafe-query.test.js b/evals/triage-finding/fixtures/repo/tests/unsafe-query.test.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/fixtures/repo/tests/unsafe-query.test.jsrename to evals/triage-finding/fixtures/repo/tests/unsafe-query.test.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/package.json b/evals/triage-finding/package.jsonsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/package.jsonrename to evals/triage-finding/package.jsondiff --git a/plugins/codex-security/skills/triage-finding/evals/pnpm-lock.yaml b/evals/triage-finding/pnpm-lock.yamlsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/pnpm-lock.yamlrename to evals/triage-finding/pnpm-lock.yamldiff --git a/plugins/codex-security/skills/triage-finding/evals/pnpm-workspace.yaml b/evals/triage-finding/pnpm-workspace.yamlsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/pnpm-workspace.yamlrename to evals/triage-finding/pnpm-workspace.yamldiff --git a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration-smoke.yaml b/evals/triage-finding/promptfooconfig.calibration-smoke.yamlsimilarity index 96%rename from plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration-smoke.yamlrename to evals/triage-finding/promptfooconfig.calibration-smoke.yamlindex 06a794069..7259be208 100644--- a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration-smoke.yaml+++ b/evals/triage-finding/promptfooconfig.calibration-smoke.yaml@@ -8,7 +8,7 @@ providers:- id: openai:codex-sdk:gpt-5.5label: "Codex SDK triage-finding calibration smoke"config:- working_dir: ../../../../..+ working_dir: ../..sandbox_mode: read-onlyapproval_policy: nevernetwork_access_enabled: falsediff --git a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration.yaml b/evals/triage-finding/promptfooconfig.calibration.yamlsimilarity index 96%rename from plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration.yamlrename to evals/triage-finding/promptfooconfig.calibration.yamlindex 18783529b..3ebd971f3 100644--- a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration.yaml+++ b/evals/triage-finding/promptfooconfig.calibration.yaml@@ -8,7 +8,7 @@ providers:- id: openai:codex-sdk:gpt-5.5label: "Codex SDK triage-finding calibration"config:- working_dir: ../../../../..+ working_dir: ../..sandbox_mode: read-onlyapproval_policy: nevernetwork_access_enabled: falsediff --git a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.yaml b/evals/triage-finding/promptfooconfig.yamlsimilarity index 98%rename from plugins/codex-security/skills/triage-finding/evals/promptfooconfig.yamlrename to evals/triage-finding/promptfooconfig.yamlindex 7882244aa..088478f43 100644--- a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.yaml+++ b/evals/triage-finding/promptfooconfig.yaml@@ -8,7 +8,7 @@ providers:- id: openai:codex-sdk:gpt-5.5label: "Codex SDK triage-finding"config:- working_dir: ../../../../..+ working_dir: ../..sandbox_mode: read-onlyapproval_policy: nevernetwork_access_enabled: falsediff --git a/plugins/codex-security/skills/triage-finding/evals/prompts/triage-request.txt b/evals/triage-finding/prompts/triage-request.txtsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/prompts/triage-request.txtrename to evals/triage-finding/prompts/triage-request.txtdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/DESIGN.md b/evals/triage-finding/sastbench/DESIGN.mdsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/DESIGN.mdrename to evals/triage-finding/sastbench/DESIGN.mddiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/README.md b/evals/triage-finding/sastbench/README.mdsimilarity index 93%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/README.mdrename to evals/triage-finding/sastbench/README.mdindex 31860f0e8..31c964fbd 100644--- a/plugins/codex-security/skills/triage-finding/evals/sastbench/README.md+++ b/evals/triage-finding/sastbench/README.md@@ -4,20 +4,20 @@ This Promptfoo lane measures how accurately the source version of `plugins/codexSastBench contains 2,737 findings: 299 labeled true positive and 2,438 labeled false positive. The JavaScript test generator exposes the scanner claim and the exact affected checkout to Codex while keeping SastBench's label-bearing `finding_id` and ground-truth label out of the rendered prompt. The eval runner stages the skill in a throwaway working directory and uses a deny-by-default Codex permission profile so only that directory, hydrated target repos, and their label-free Git cache plus minimal Codex runtime paths are readable to the model.-Run commands from the repository root. The parent `evals/` directory owns the pinned Promptfoo and Codex SDK dependencies and the ignored `artifacts/` tree.+Run commands from the repository root. The parent `evals/triage-finding/` directory owns the pinned Promptfoo and Codex SDK dependencies and the ignored `artifacts/` tree.## Prepare the benchmarkInstall dependencies once:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run setup+pnpm --dir evals/triage-finding run setup```Install the pinned SastBench checkout and hydrate all 275 repository revisions:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run sastbench:prepare+pnpm --dir evals/triage-finding run sastbench:prepare```Preparation verifies the SastBench origin, pinned commit, dataset SHA-256, record counts, target origins, target commits, and clean target worktrees. It must finish before any paid model evaluation begins.@@ -27,14 +27,14 @@ Preparation verifies the SastBench origin, pinned commit, dataset SHA-256, recorRun deterministic tests and ask Promptfoo to load the dynamic test generator:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run test:sastbench-pnpm --dir plugins/codex-security/skills/triage-finding/evals run validate:sastbench+pnpm --dir evals/triage-finding run test:sastbench+pnpm --dir evals/triage-finding run validate:sastbench```## Run an evaluation```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:sastbench \+pnpm --dir evals/triage-finding run eval:sastbench \--output ./artifacts/sastbench-results/canary.promptfoo.jsonl```@@ -51,8 +51,8 @@ The sample reuses the full eval's pinned dataset, hydrated repositories, prompt,Validate and run it from the repository root:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run validate:sastbench:sample-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:sastbench:sample \+pnpm --dir evals/triage-finding run validate:sastbench:sample+pnpm --dir evals/triage-finding run eval:sastbench:sample \--output ./artifacts/sastbench-results/representative-sample.promptfoo.jsonl```@@ -130,7 +130,7 @@ Every denominator uses `max(denominator, 1)`. An empty denominator therefore proOpen the Promptfoo viewer to inspect the completed evaluation:```bash-pnpm --dir plugins/codex-security/skills/triage-finding/evals run pf:view+pnpm --dir evals/triage-finding run pf:view```Promptfoo also records cost, latency, and token use without SastBench-specific code.diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/assertions/sastbench-metrics.js b/evals/triage-finding/sastbench/assertions/sastbench-metrics.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/assertions/sastbench-metrics.jsrename to evals/triage-finding/sastbench/assertions/sastbench-metrics.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/assertions/sastbench-verdict.js b/evals/triage-finding/sastbench/assertions/sastbench-verdict.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/assertions/sastbench-verdict.jsrename to evals/triage-finding/sastbench/assertions/sastbench-verdict.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/promptfooconfig.sastbench-sample.yaml b/evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yamlsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/promptfooconfig.sastbench-sample.yamlrename to evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yamldiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/promptfooconfig.sastbench.yaml b/evals/triage-finding/sastbench/promptfooconfig.sastbench.yamlsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/promptfooconfig.sastbench.yamlrename to evals/triage-finding/sastbench/promptfooconfig.sastbench.yamldiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/prompts/sastbench-triage-request.txt b/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txtsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/prompts/sastbench-triage-request.txtrename to evals/triage-finding/sastbench/prompts/sastbench-triage-request.txtdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/generate-sastbench-tests.js b/evals/triage-finding/sastbench/scripts/generate-sastbench-tests.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/generate-sastbench-tests.jsrename to evals/triage-finding/sastbench/scripts/generate-sastbench-tests.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/hydrate-sastbench-repos.js b/evals/triage-finding/sastbench/scripts/hydrate-sastbench-repos.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/hydrate-sastbench-repos.jsrename to evals/triage-finding/sastbench/scripts/hydrate-sastbench-repos.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/install-sastbench.js b/evals/triage-finding/sastbench/scripts/install-sastbench.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/install-sastbench.jsrename to evals/triage-finding/sastbench/scripts/install-sastbench.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/run-sastbench-promptfoo.js b/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.jssimilarity index 94%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/run-sastbench-promptfoo.jsrename to evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.jsindex 16cd518f2..2f7736751 100644--- a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/run-sastbench-promptfoo.js+++ b/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.js@@ -7,8 +7,8 @@ const os = require("node:os");const path = require("node:path");const EVAL_ROOT = path.resolve(__dirname, "..", "..");-const TRIAGE_SKILL_ROOT = path.resolve(EVAL_ROOT, "..");-const PLUGIN_ROOT = path.resolve(TRIAGE_SKILL_ROOT, "..", "..");+const PLUGIN_ROOT = path.resolve(EVAL_ROOT, "..", "..", "plugins", "codex-security");+const TRIAGE_SKILL_ROOT = path.join(PLUGIN_ROOT, "skills", "triage-finding");const DEFAULT_TARGET_ROOT = path.join(EVAL_ROOT, "artifacts", "sastbench-targets");const DEFAULT_GIT_CACHE_ROOT = path.join(EVAL_ROOT, "artifacts", "sastbench-git-cache");const PROMPTFOO_BIN = path.join(EVAL_ROOT, "node_modules", ".bin", "promptfoo");diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/sastbench-lib.js b/evals/triage-finding/sastbench/scripts/sastbench-lib.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/sastbench-lib.jsrename to evals/triage-finding/sastbench/scripts/sastbench-lib.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/sastbench-result.js b/evals/triage-finding/sastbench/scripts/sastbench-result.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/sastbench-result.jsrename to evals/triage-finding/sastbench/scripts/sastbench-result.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-hydrate-sastbench-repos.js b/evals/triage-finding/sastbench/scripts/test-hydrate-sastbench-repos.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-hydrate-sastbench-repos.jsrename to evals/triage-finding/sastbench/scripts/test-hydrate-sastbench-repos.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-install-sastbench.js b/evals/triage-finding/sastbench/scripts/test-install-sastbench.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-install-sastbench.jsrename to evals/triage-finding/sastbench/scripts/test-install-sastbench.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-promptfoo-native-harness.js b/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-promptfoo-native-harness.jsrename to evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-sastbench-lib.js b/evals/triage-finding/sastbench/scripts/test-sastbench-lib.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-sastbench-lib.jsrename to evals/triage-finding/sastbench/scripts/test-sastbench-lib.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-sastbench-result.js b/evals/triage-finding/sastbench/scripts/test-sastbench-result.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-sastbench-result.jsrename to evals/triage-finding/sastbench/scripts/test-sastbench-result.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/schemas/triage-result-v0.schema.json b/evals/triage-finding/schemas/triage-result-v0.schema.jsonsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/schemas/triage-result-v0.schema.jsonrename to evals/triage-finding/schemas/triage-result-v0.schema.jsondiff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/generate-calibration-tests.js b/evals/triage-finding/scripts/generate-calibration-tests.jssimilarity index 98%rename from plugins/codex-security/skills/triage-finding/evals/scripts/generate-calibration-tests.jsrename to evals/triage-finding/scripts/generate-calibration-tests.jsindex 7bafe1cfc..1ac5f23e3 100644--- a/plugins/codex-security/skills/triage-finding/evals/scripts/generate-calibration-tests.js+++ b/evals/triage-finding/scripts/generate-calibration-tests.js@@ -7,7 +7,7 @@ const path = require("node:path");const DEFAULT_DATASET = path.join(__dirname, "..", "datasets", "triage-calibration-seed.json");const DEFAULT_OUTPUT = path.join(__dirname, "..", "tests", "calibration-oss.yaml");const DEFAULT_REPO_ROOT =- "plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos";+ "evals/triage-finding/artifacts/calibration-repos";function parseArgs(argv) {const args = {diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/hydrate-calibration-repos.js b/evals/triage-finding/scripts/hydrate-calibration-repos.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/scripts/hydrate-calibration-repos.jsrename to evals/triage-finding/scripts/hydrate-calibration-repos.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-app-surface-instructions.js b/evals/triage-finding/scripts/test-app-surface-instructions.jssimilarity index 87%rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-app-surface-instructions.jsrename to evals/triage-finding/scripts/test-app-surface-instructions.jsindex 257428c48..0d51c5fe8 100644--- a/plugins/codex-security/skills/triage-finding/evals/scripts/test-app-surface-instructions.js+++ b/evals/triage-finding/scripts/test-app-surface-instructions.js@@ -2,13 +2,15 @@ const assert = require("node:assert/strict");const fs = require("node:fs");const path = require("node:path");-const skillPath = path.join(__dirname, "..", "..", "SKILL.md");+const pluginRoot = path.resolve(__dirname, "..", "..", "..", "plugins", "codex-security");+const skillRoot = path.join(pluginRoot, "skills", "triage-finding");+const skillPath = path.join(skillRoot, "SKILL.md");const skill = fs.readFileSync(skillPath, "utf8");-const ticketIntakePath = path.join(__dirname, "..", "..", "references", "ticket-intake.md");+const ticketIntakePath = path.join(skillRoot, "references", "ticket-intake.md");const ticketIntake = fs.readFileSync(ticketIntakePath, "utf8");-const agentPath = path.join(__dirname, "..", "..", "agents", "openai.yaml");+const agentPath = path.join(skillRoot, "agents", "openai.yaml");const agent = fs.readFileSync(agentPath, "utf8");-const pluginPath = path.join(__dirname, "..", "..", "..", "..", ".codex-plugin", "plugin.json");+const pluginPath = path.join(pluginRoot, ".codex-plugin", "plugin.json");const plugin = JSON.parse(fs.readFileSync(pluginPath, "utf8"));assert.match(skill, /## Jira and Linear Intake/);diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-calibration-dataset.js b/evals/triage-finding/scripts/test-calibration-dataset.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-calibration-dataset.jsrename to evals/triage-finding/scripts/test-calibration-dataset.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-calibration-evidence.js b/evals/triage-finding/scripts/test-calibration-evidence.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-calibration-evidence.jsrename to evals/triage-finding/scripts/test-calibration-evidence.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-generate-calibration-tests.js b/evals/triage-finding/scripts/test-generate-calibration-tests.jssimilarity index 94%rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-generate-calibration-tests.jsrename to evals/triage-finding/scripts/test-generate-calibration-tests.jsindex 7644f30da..a70d34fa2 100644--- a/plugins/codex-security/skills/triage-finding/evals/scripts/test-generate-calibration-tests.js+++ b/evals/triage-finding/scripts/test-generate-calibration-tests.js@@ -29,7 +29,7 @@ assert.match(generatedYaml, /case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerablassert.match(generatedYaml, /case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed/);assert.match(generatedYaml, /expected_verdicts: confirmed/);assert.match(generatedYaml, /expected_verdicts: not_actionable/);-assert.match(generatedYaml, /target_repo: plugins\/codex-security\/skills\/triage-finding\/evals\/artifacts\/calibration-repos\//);+assert.match(generatedYaml, /target_repo: evals\/triage-finding\/artifacts\/calibration-repos\//);assert.doesNotMatch(generatedYaml,/expected_evidence_terms:\n\s+- /,diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-hydrate-calibration-repos.js b/evals/triage-finding/scripts/test-hydrate-calibration-repos.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-hydrate-calibration-repos.jsrename to evals/triage-finding/scripts/test-hydrate-calibration-repos.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-triage-io.js b/evals/triage-finding/scripts/test-triage-io.jssimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-triage-io.jsrename to evals/triage-finding/scripts/test-triage-io.jsdiff --git a/plugins/codex-security/skills/triage-finding/evals/tests/calibration-oss.yaml b/evals/triage-finding/tests/calibration-oss.yamlsimilarity index 93%rename from plugins/codex-security/skills/triage-finding/evals/tests/calibration-oss.yamlrename to evals/triage-finding/tests/calibration-oss.yamlindex f460edfde..89e3ab0c1 100644--- a/plugins/codex-security/skills/triage-finding/evals/tests/calibration-oss.yaml+++ b/evals/triage-finding/tests/calibration-oss.yaml@@ -8,7 +8,7 @@expected_binary_label: positivevars:case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerable- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/vulnerable+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/vulnerablesource_type_under_test: cveexpected_ids: GHSA-73vx-49mv-v8w5-000-vulnerableexpected_source_types: cve@@ -43,7 +43,7 @@expected_binary_label: negativevars:case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/fixed+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/fixedsource_type_under_test: cveexpected_ids: GHSA-73vx-49mv-v8w5-000-fixedexpected_source_types: cve@@ -78,7 +78,7 @@expected_binary_label: positivevars:case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-vulnerable- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/vulnerable+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/vulnerablesource_type_under_test: cveexpected_ids: GHSA-qpgx-jfcq-r59f-000-vulnerableexpected_source_types: cve@@ -113,7 +113,7 @@expected_binary_label: negativevars:case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-fixed- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/fixed+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/fixedsource_type_under_test: cveexpected_ids: GHSA-qpgx-jfcq-r59f-000-fixedexpected_source_types: cve@@ -148,7 +148,7 @@expected_binary_label: positivevars:case_id: oss-scitokens-ghsa-rh5m-2482-966c-vulnerable- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/vulnerable+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/vulnerablesource_type_under_test: cveexpected_ids: GHSA-rh5m-2482-966c-000-vulnerableexpected_source_types: cve@@ -183,7 +183,7 @@expected_binary_label: negativevars:case_id: oss-scitokens-ghsa-rh5m-2482-966c-fixed- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/fixed+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/fixedsource_type_under_test: cveexpected_ids: GHSA-rh5m-2482-966c-000-fixedexpected_source_types: cve@@ -218,7 +218,7 @@expected_binary_label: positivevars:case_id: oss-emissary-ghsa-3p24-9x7v-7789-vulnerable- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/vulnerable+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/vulnerablesource_type_under_test: cveexpected_ids: GHSA-3p24-9x7v-7789-000-vulnerableexpected_source_types: cve@@ -253,7 +253,7 @@expected_binary_label: negativevars:case_id: oss-emissary-ghsa-3p24-9x7v-7789-fixed- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/fixed+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/fixedsource_type_under_test: cveexpected_ids: GHSA-3p24-9x7v-7789-000-fixedexpected_source_types: cve@@ -288,7 +288,7 @@expected_binary_label: positivevars:case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-vulnerable- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/vulnerable+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/vulnerablesource_type_under_test: cveexpected_ids: GHSA-6RGM-GR97-X3J5-vulnerableexpected_source_types: cve@@ -323,7 +323,7 @@expected_binary_label: negativevars:case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-fixed- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/fixed+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/fixedsource_type_under_test: cveexpected_ids: GHSA-6RGM-GR97-X3J5-fixedexpected_source_types: cve@@ -358,7 +358,7 @@expected_binary_label: positivevars:case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-vulnerable- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/vulnerable+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/vulnerablesource_type_under_test: cveexpected_ids: GHSA-27vg-33gh-4hwg-000-vulnerableexpected_source_types: cve@@ -393,7 +393,7 @@expected_binary_label: negativevars:case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-fixed- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/fixed+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/fixedsource_type_under_test: cveexpected_ids: GHSA-27vg-33gh-4hwg-000-fixedexpected_source_types: cve@@ -428,7 +428,7 @@expected_binary_label: positivevars:case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-vulnerable- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/vulnerable+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/vulnerablesource_type_under_test: cveexpected_ids: GHSA-v8jm-5vwx-cfxm-gap-fill-000-vulnerableexpected_source_types: cve@@ -463,7 +463,7 @@expected_binary_label: negativevars:case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-fixed- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/fixed+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/fixedsource_type_under_test: cveexpected_ids: GHSA-v8jm-5vwx-cfxm-gap-fill-000-fixedexpected_source_types: cve@@ -498,7 +498,7 @@expected_binary_label: positivevars:case_id: oss-monetr-ghsa-29v9-frvh-c426-vulnerable- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/vulnerable+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/vulnerablesource_type_under_test: cveexpected_ids: GHSA-29v9-frvh-c426-000-vulnerableexpected_source_types: cve@@ -533,7 +533,7 @@expected_binary_label: negativevars:case_id: oss-monetr-ghsa-29v9-frvh-c426-fixed- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/fixed+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/fixedsource_type_under_test: cveexpected_ids: GHSA-29v9-frvh-c426-000-fixedexpected_source_types: cvediff --git a/plugins/codex-security/skills/triage-finding/evals/tests/github-rest-intake.yaml b/evals/triage-finding/tests/github-rest-intake.yamlsimilarity index 100%rename from plugins/codex-security/skills/triage-finding/evals/tests/github-rest-intake.yamlrename to evals/triage-finding/tests/github-rest-intake.yamldiff --git a/plugins/codex-security/skills/triage-finding/evals/tests/input-types.yaml b/evals/triage-finding/tests/input-types.yamlsimilarity index 92%rename from plugins/codex-security/skills/triage-finding/evals/tests/input-types.yamlrename to evals/triage-finding/tests/input-types.yamlindex 5f0ad7146..1c950751c 100644--- a/plugins/codex-security/skills/triage-finding/evals/tests/input-types.yaml+++ b/evals/triage-finding/tests/input-types.yaml@@ -4,7 +4,7 @@source_type: sarifvars:case_id: sarif-redirects- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: sarifexpected_ids: SARIF-OPEN-REDIRECT,SARIF-SAFE-REDIRECTexpected_source_types: sarif,sarif@@ -27,7 +27,7 @@source_type: cvevars:case_id: cve-dependencies- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: cveexpected_ids: CVE-2026-0001,CVE-2026-0002expected_source_types: cve,cve@@ -50,7 +50,7 @@source_type: advisoryvars:case_id: advisory-downloads- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: advisoryexpected_ids: ADV-DOWNLOAD-TRAVERSAL,ADV-SAFE-DOWNLOADexpected_source_types: advisory,advisory@@ -70,7 +70,7 @@source_type: scanner_ticketvars:case_id: scanner-query- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketexpected_ids: SCAN-SQL-001,SCAN-SQL-TEST-ONLYexpected_source_types: scanner_ticket,scanner_ticket@@ -92,7 +92,7 @@source_type: bug_bountyvars:case_id: bug-bounty-access- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: bug_bountyexpected_ids: BB-ORDER-IDOR,BB-PARTNER-EXPORTexpected_source_types: bug_bounty,bug_bounty@@ -112,7 +112,7 @@source_type: codex_security_findingvars:case_id: codex-security-findings- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: codex_security_findingexpected_ids: csf_111111111111111111111111,csf_222222222222222222222222expected_source_types: codex_security_finding,codex_security_finding@@ -160,7 +160,7 @@source_type: freeformvars:case_id: freeform-mixed- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: freeformexpected_ids: FREEFORM-REDIRECT,FREEFORM-UNSPECIFIED-AUTHexpected_source_types: freeform,freeform@@ -178,7 +178,7 @@source_type: unknownvars:case_id: unknown-fallback- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: unknownexpected_ids: UNKNOWN-001,UNKNOWN-002expected_source_types: unknown,unknowndiff --git a/plugins/codex-security/skills/triage-finding/evals/tests/invocation-behavior.yaml b/evals/triage-finding/tests/invocation-behavior.yamlsimilarity index 87%rename from plugins/codex-security/skills/triage-finding/evals/tests/invocation-behavior.yamlrename to evals/triage-finding/tests/invocation-behavior.yamlindex 564df02d7..6526d1978 100644--- a/plugins/codex-security/skills/triage-finding/evals/tests/invocation-behavior.yaml+++ b/evals/triage-finding/tests/invocation-behavior.yaml@@ -4,7 +4,7 @@source_type: nonevars:case_id: missing-finding-input- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: nonefinding_input: ""invocation_preamble: "Use $codex-security:triage-finding with no finding supplied."diff --git a/plugins/codex-security/skills/triage-finding/evals/tests/ticket-intake.yaml b/evals/triage-finding/tests/ticket-intake.yamlsimilarity index 91%rename from plugins/codex-security/skills/triage-finding/evals/tests/ticket-intake.yamlrename to evals/triage-finding/tests/ticket-intake.yamlindex acfd45b3a..2a8fd4b2f 100644--- a/plugins/codex-security/skills/triage-finding/evals/tests/ticket-intake.yaml+++ b/evals/triage-finding/tests/ticket-intake.yaml@@ -4,7 +4,7 @@source_type: scanner_ticketvars:case_id: ticket-connector-unavailable- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketfinding_input: "Linear issue SEC-293. Connector observation: the Linear connector is unavailable because it is not connected."invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."@@ -25,7 +25,7 @@source_type: scanner_ticketvars:case_id: ticket-connector-permission- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketfinding_input: "Linear issue SEC-293. Connector observation: access was denied because the authenticated user lacks permission."invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."@@ -46,7 +46,7 @@source_type: scanner_ticketvars:case_id: ticket-not-found- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketfinding_input: "Linear issue SEC-999999. Connector observation: the issue was not found or is inaccessible."invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-999999."@@ -67,7 +67,7 @@source_type: scanner_ticketvars:case_id: ticket-transient-retry-exhausted- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketfinding_input: "Linear issue SEC-293. Connector observation: the first read returned a transient timeout; the identical retry also timed out."invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."@@ -89,7 +89,7 @@source_type: scanner_ticketvars:case_id: linear-direct-subissue-confirmation- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketfinding_input: "Linear parent SEC-293 was fetched. Child listing returned SEC-294 'Auth bypass' and SEC-295 'Unsafe redirect'. Their full bodies have not been fetched."invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."@@ -110,7 +110,7 @@source_type: scanner_ticketvars:case_id: linear-nested-subissue-confirmation- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketfinding_input: "The user approved direct child SEC-294. Its full body was fetched. The next child listing found SEC-296 'Nested token leak'; its full body has not been fetched."invocation_preamble: "Continue $codex-security:triage-finding for the approved Linear issue tree."@@ -131,7 +131,7 @@source_type: scanner_ticketvars:case_id: linear-ambiguous-parent- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketfinding_input: "Linear parent SEC-293 contains collection notes plus a possible standalone authorization claim. The user approved importing its child findings."invocation_preamble: "Continue $codex-security:triage-finding for Linear parent SEC-293."@@ -152,7 +152,7 @@source_type: scanner_ticketvars:case_id: linear-subissue-limit- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo+ target_repo: evals/triage-finding/fixtures/reposource_type_under_test: scanner_ticketfinding_input: "The approved Linear parent and selected sub-issue levels contain 251 triage items. No issue bodies beyond the limit have been imported."invocation_preamble: "Continue $codex-security:triage-finding for this Linear issue tree."diff --git a/plugins/codex-security/mcp-app/evals/README.md b/plugins/codex-security/mcp-app/evals/README.mddeleted file mode 100644index b8ac619b4..000000000--- a/plugins/codex-security/mcp-app/evals/README.md+++ /dev/null@@ -1,25 +0,0 @@-# Deep reducer paging eval--This eval runs one reducer against synthetic persisted worker findings and a previous aggregate. It uses the pinned Codex CLI, its external code-mode host, and the production artifact tool handlers. It does not launch a repository scan.--The fixture has a 32 MiB individual semantic field containing quotes, backslashes, newlines, and Unicode. Its old duplicated MCP response is about 114 MiB. The first input read deliberately returns that old response; all subsequent calls use the production paged implementation. The IPC error comes from the real transport, not a mocked error message.--The grader requires a smaller byte budget on the retry with the same cursor and reference, byte-bounded subsequent pages, one successful result submission, every source accounted for exactly once, complete original payloads matching their SHA-256 hashes, and preservation of the previous finding identity and synthesized history.--Install the MCP app's locked dependencies using the repository's normal setup, then run from the repository root:--```sh-node plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs deterministic-```--The deterministic mode supplies scripted model responses through a local HTTP provider. It uses a private temporary Codex home and no model credentials or external model requests. The script reads every input page, fetches original and prior canonical details by reference, and records the reduction through the real tools. This mode also runs in the normal MCP test suite as `test_deep_reducer_paging_eval.mjs`.--To evaluate a model's recovery decisions using the production reducer prompt:--```sh-node plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs model-```--The model run uses the caller's normal Codex credentials and configuration and consumes model usage. An optional final argument selects a model; otherwise Codex uses its configured model. The fault injection and result grader are the same. This mode is opt-in and does not run in CI. Unlike the deterministic run, it does not supply the recovery code to the model. The SDK does not expose every code-mode error event, so the model-mode grade relies on the observed oversized response, changed request, and saved result; the deterministic mode additionally asserts the exact transport diagnostic.--Each command prints the path to an ignored directory under `mcp-app/reports/` containing `report.json`, the compact tool-call trace, and the generated artifacts. The fixture is generated at runtime; no large payload files are checked in.diff --git a/plugins/codex-security/mcp-app/evals/controlled-code-mode.mjs b/plugins/codex-security/mcp-app/tests/support/reducer-paging/controlled-code-mode.mjssimilarity index 100%rename from plugins/codex-security/mcp-app/evals/controlled-code-mode.mjsrename to plugins/codex-security/mcp-app/tests/support/reducer-paging/controlled-code-mode.mjsdiff --git a/plugins/codex-security/mcp-app/evals/deep-reducer-paging-fixture.mjs b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-fixture.mjssimilarity index 100%rename from plugins/codex-security/mcp-app/evals/deep-reducer-paging-fixture.mjsrename to plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-fixture.mjsdiff --git a/plugins/codex-security/mcp-app/evals/deep-reducer-paging-server.mjs b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-server.mjssimilarity index 92%rename from plugins/codex-security/mcp-app/evals/deep-reducer-paging-server.mjsrename to plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-server.mjsindex 6703ab1a4..97c42aebb 100644--- a/plugins/codex-security/mcp-app/evals/deep-reducer-paging-server.mjs+++ b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-server.mjs@@ -1,8 +1,8 @@import { appendFile, readFile } from "node:fs/promises";import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";-import { getCodexSecurityDeepReducerInputs } from "../src/artifact-deep-reducer.ts";-import { registerCompactWorkerArtifactTools } from "../src/server/compact-artifact-tools.ts";+import { getCodexSecurityDeepReducerInputs } from "../../../src/artifact-deep-reducer.ts";+import { registerCompactWorkerArtifactTools } from "../../../src/server/compact-artifact-tools.ts";async function main() {const { context, tracePath } = JSON.parse(diff --git a/plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging.mjssimilarity index 91%rename from plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjsrename to plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging.mjsindex 505ace7c1..7d22a9485 100644--- a/plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs+++ b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging.mjs@@ -1,5 +1,5 @@import assert from "node:assert/strict";-import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";+import { readFile, writeFile } from "node:fs/promises";import path from "node:path";import { fileURLToPath, pathToFileURL } from "node:url";import { Codex } from "@openai/codex-sdk";@@ -10,7 +10,7 @@ import {gradeReducerPagingResult,} from "./deep-reducer-paging-fixture.mjs";-const evalDirectory = path.dirname(fileURLToPath(import.meta.url));+const supportDirectory = path.dirname(fileURLToPath(import.meta.url));/** Run one reducer through real code-mode IPC and the production artifact tools. */export async function runReducerPagingEval({@@ -31,7 +31,9 @@ export async function runReducerPagingEval({);await Promise.all([build({- entryPoints: [path.join(evalDirectory, "deep-reducer-paging-server.mjs")],+ entryPoints: [+ path.join(supportDirectory, "deep-reducer-paging-server.mjs"),+ ],outfile: serverPath,bundle: true,platform: "node",@@ -39,7 +41,9 @@ export async function runReducerPagingEval({loader: { ".md": "text" },}),build({- entryPoints: [path.join(evalDirectory, "../src/deep-scan/templates.ts")],+ entryPoints: [+ path.join(supportDirectory, "../../../src/deep-scan/templates.ts"),+ ],outfile: promptModulePath,bundle: true,platform: "node",@@ -253,16 +257,3 @@ const result = await tools.mcp__cs_artifacts__record_codex_security_deep_reductiif (result.isError) throw new Error(result.content[0].text);text({reductionRecorded: true});`;--if (process.argv[1] === fileURLToPath(import.meta.url)) {- const reports = path.join(evalDirectory, "../reports");- await mkdir(reports, { recursive: true });- const root = await mkdtemp(path.join(reports, "deep-reducer-paging-"));- console.log(`Eval artifacts: ${root}`);- const report = await runReducerPagingEval({- root,- mode: process.argv[2] ?? "deterministic",- model: process.argv[3],- });- console.log(JSON.stringify(report, null, 2));-}diff --git a/plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs b/plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjsindex e97926009..bf9fb1307 100644--- a/plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs+++ b/plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs@@ -6,11 +6,11 @@ import test from "node:test";import {gradeReducerPagingTrace,runReducerPagingEval,-} from "../evals/deep-reducer-paging.mjs";+} from "./support/reducer-paging/deep-reducer-paging.mjs";import {createReducerPagingFixture,gradeReducerPagingResult,-} from "../evals/deep-reducer-paging-fixture.mjs";+} from "./support/reducer-paging/deep-reducer-paging-fixture.mjs";test("a reducer recovers from the real IPC frame limit and records all sources", async (t) => {const root = await mkdtemp(path.join(tmpdir(), "deep-reducer-ipc-eval-"));diff --git a/sdk/typescript/tests-ts/runtime.test.ts b/sdk/typescript/tests-ts/runtime.test.tsindex fc0d5e957..f47aef5ff 100644--- a/sdk/typescript/tests-ts/runtime.test.ts+++ b/sdk/typescript/tests-ts/runtime.test.ts@@ -4791,9 +4791,12 @@ describe("runtime directories and plugin Python boundary", () => {expect(result["details"]).toHaveLength(5 * 1024 * 1024);});- test.each([false, true])(- "selects workbench Git for the requested target (bound: %s)",- async (bound) => {+ test.each([+ ["unbound", false],+ ["bound", true],+ ] as const)(+ "selects workbench Git for the requested target (%s)",+ async (_label, bound) => {const root = await temporaryDirectory();const repository = join(root, "repository");const untrustedBin = join(repository, "tools");diff --git a/sdk/typescript/tests-ts/scan-logs.test.ts b/sdk/typescript/tests-ts/scan-logs.test.tsindex 972637477..f383cfafc 100644--- a/sdk/typescript/tests-ts/scan-logs.test.ts+++ b/sdk/typescript/tests-ts/scan-logs.test.ts@@ -220,9 +220,12 @@ describe("saved scan logs", () => {expect(JSON.stringify(result)).not.toContain("unrelated");});- test.each([undefined, "missing-owner"])(- "feedback collects known worker descendants without the owner log with continuation %s",- async (continuationThreadId) => {+ test.each([+ ["omitted", undefined],+ ["recorded", "missing-owner"],+ ] as const)(+ "feedback collects known worker descendants without the owner log with %s continuation",+ async (_label, continuationThreadId) => {const home = await temporaryHome();await writeSession(home, "owner-child", [], "missing-owner");await writeSession(home, "worker", [diff --git a/sdk/typescript/tests-ts/skeleton.test.ts b/sdk/typescript/tests-ts/skeleton.test.tsindex 5b6bfaec1..b5439c3cf 100644--- a/sdk/typescript/tests-ts/skeleton.test.ts+++ b/sdk/typescript/tests-ts/skeleton.test.ts@@ -62,7 +62,7 @@ describe("TypeScript package skeleton", () => {const directories = ["sdk/typescript","plugins/codex-security/mcp-app",- "plugins/codex-security/skills/triage-finding/evals",+ "evals/triage-finding",];const manifests = await Promise.all(directories.map(async (directory) =>