Skip to content

OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security

test: relocate triage evals and reducer model entry point (#1056)

mldangelo-oaia3deb87cfc36
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index 7c91c7bda..37d4cf2cb 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -4,7 +4,7 @@ updates:
directories:
- /sdk/typescript
- /plugins/codex-security/mcp-app
- - /plugins/codex-security/skills/triage-finding/evals
+ - /evals/triage-finding
schedule:
interval: cron
cronjob: "0 9 * * *"
diff --git a/.github/workflows/node-ci.yml b/.github/workflows/node-ci.yml
index c2edbac72..659528234 100644
--- a/.github/workflows/node-ci.yml
+++ b/.github/workflows/node-ci.yml
@@ -512,7 +512,7 @@ jobs:
if-no-files-found: warn
retention-days: 14
- name: Test deterministic triage eval contracts
- working-directory: plugins/codex-security/skills/triage-finding/evals
+ working-directory: evals/triage-finding
run: node --run test:deterministic
required-test:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 667813f94..34c6cbbe6 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -15,6 +15,10 @@ packages and releases. Do not edit or commit files in that directory. See the
[SDK testing guide](sdk/typescript/TESTING.md) for the generation and validation
commands.
+Model-based evaluations live in [`evals/`](evals/README.md), outside the plugin
+source. Deterministic triage checks and the MCP reducer IPC regression remain
+part of normal CI.
+
Search [existing issues](https://github.com/openai/codex-security/issues)
before opening a new one.
diff --git a/RELEASING.md b/RELEASING.md
index d6ca4ae98..199b12e38 100644
--- a/RELEASING.md
+++ b/RELEASING.md
@@ -52,10 +52,10 @@ version-update cooldown.
Updates still require review and passing CI; nothing is merged automatically.
Keep `@openai/codex` and `@openai/codex-sdk` on the same exact version across the
-TypeScript SDK, MCP app, and triage evals. Dependabot groups their updates across
-all three projects, and the SDK tests reject mismatched pins or multiple locked
-SDK versions. The evals override Promptfoo's transitive Codex SDK to the direct
-SDK dependency so it follows the same update.
+TypeScript SDK, MCP app, and triage evals in `evals/triage-finding`. Dependabot
+groups their updates across all three projects, and the SDK tests reject
+mismatched pins or multiple locked SDK versions. The evals override Promptfoo's
+transitive Codex SDK to the direct SDK dependency so it follows the same update.
Each pnpm project applies the same seven-day age policy to newly resolved
dependencies, including transitive packages, with `openai` and `@openai/*` exempt.
diff --git a/evals/README.md b/evals/README.md
new file mode 100644
index 000000000..7839a00aa
--- /dev/null
+++ b/evals/README.md
@@ -0,0 +1,13 @@
+# Evaluations
+
+Model-based evaluations live here so they can evolve with the plugin without
+being embedded in its source tree or shipped npm runtime.
+
+- [Triage finding](triage-finding/README.md): Promptfoo input contracts, OSS
+ calibration, and SastBench. This suite owns its private package and lockfile.
+- [Deep reducer](deep-reducer/README.md): optional model evaluation of real IPC
+ size-limit recovery, pagination, and finding retention. It reuses the MCP
+ test helpers and dependencies.
+
+Model runs are opt-in. CI still runs the deterministic triage helper checks and
+the real-IPC reducer regression through the normal MCP test suite.
diff --git a/evals/deep-reducer/.gitignore b/evals/deep-reducer/.gitignore
new file mode 100644
index 000000000..a9a1bd38a
--- /dev/null
+++ b/evals/deep-reducer/.gitignore
@@ -0,0 +1 @@
+reports/
diff --git a/evals/deep-reducer/README.md b/evals/deep-reducer/README.md
new file mode 100644
index 000000000..15a6a68ee
--- /dev/null
+++ b/evals/deep-reducer/README.md
@@ -0,0 +1,27 @@
+# Deep reducer paging eval
+
+This optional model eval runs one reducer against synthetic persisted worker findings and a previous aggregate. It uses the pinned Codex CLI, its external code-mode host, the production reducer prompt, and the production artifact tool handlers. It does not launch a repository scan.
+
+The fixture has a 32 MiB individual semantic field containing quotes, backslashes, newlines, and Unicode. Its old duplicated MCP response is about 114 MiB. The first input read deliberately returns that old response; all subsequent calls use the production paged implementation. The IPC error comes from the real transport, not a mocked error message.
+
+The grader requires a smaller byte budget on the retry with the same cursor and reference, byte-bounded subsequent pages, one successful result submission, every source accounted for exactly once, complete original payloads matching their SHA-256 hashes, and preservation of the previous finding identity and synthesized history.
+
+Install the MCP app's locked dependencies using the repository's normal setup, then run from the repository root:
+
+```sh
+node evals/deep-reducer/run.mjs
+```
+
+The run uses the caller's normal Codex credentials and configuration and consumes model usage. An optional final argument selects a model; otherwise Codex uses its configured model. This eval is opt-in and does not run in CI. The model receives only the production reducer prompt and must discover and carry out recovery itself. The SDK does not expose every code-mode error event, so the grade relies on the observed oversized response, changed request, and saved result.
+
+The command prints the path to an ignored directory under `evals/deep-reducer/reports/` containing `report.json`, the compact tool-call trace, and the generated artifacts. The fixture is generated at runtime; no large payload files are checked in.
+
+## Deterministic regression test
+
+The same fixture, real transport, and grading helpers live under `plugins/codex-security/mcp-app/tests/support/reducer-paging/`. The normal MCP test suite exercises them with scripted model responses from a local HTTP provider, a private temporary Codex home, and no model credentials or external model requests. It additionally asserts the exact transport diagnostic and checks that the grader rejects missing input pages and collapsed independent findings.
+
+Run that test directly from the repository root:
+
+```sh
+node --test plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs
+```
diff --git a/evals/deep-reducer/run.mjs b/evals/deep-reducer/run.mjs
new file mode 100644
index 000000000..81b8c5373
--- /dev/null
+++ b/evals/deep-reducer/run.mjs
@@ -0,0 +1,15 @@
+import { mkdir, mkdtemp } from "node:fs/promises";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { runReducerPagingEval } from "../../plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging.mjs";
+
+const reports = fileURLToPath(new URL("./reports/", import.meta.url));
+await mkdir(reports, { recursive: true });
+const root = await mkdtemp(path.join(reports, "deep-reducer-paging-"));
+console.log(`Eval artifacts: ${root}`);
+const report = await runReducerPagingEval({
+ root,
+ mode: "model",
+ model: process.argv[2],
+});
+console.log(JSON.stringify(report, null, 2));
diff --git a/plugins/codex-security/skills/triage-finding/evals/.gitignore b/evals/triage-finding/.gitignore
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/.gitignore
rename to evals/triage-finding/.gitignore
diff --git a/plugins/codex-security/skills/triage-finding/evals/README.md b/evals/triage-finding/README.md
similarity index 80%
rename from plugins/codex-security/skills/triage-finding/evals/README.md
rename to evals/triage-finding/README.md
index 23e567763..e21c3ab23 100644
--- a/plugins/codex-security/skills/triage-finding/evals/README.md
+++ b/evals/triage-finding/README.md
@@ -8,36 +8,36 @@ The suite uses the Promptfoo Codex SDK provider because it only needs final assi
Use Node.js 22.22.0 or newer for the eval runner. Run these commands from the repository root to install dependencies under this eval directory.
-The eval directory has its own `pnpm-workspace.yaml` so pnpm treats it as a small standalone workspace instead of joining the root OpenAI monorepo workspace.
+The eval directory has its own `pnpm-workspace.yaml` so pnpm treats it as a small standalone workspace, independent of the SDK and MCP app packages.
Install the local eval runner:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run setup
+pnpm --dir evals/triage-finding run setup
```
Validate the config:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run validate
+pnpm --dir evals/triage-finding run validate
```
Run the full eval:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval
+pnpm --dir evals/triage-finding run eval
```
Run the first case as a quick smoke test:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:smoke
+pnpm --dir evals/triage-finding run eval:smoke
```
Run one case while iterating:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval --filter-metadata case_id=sarif-redirects
+pnpm --dir evals/triage-finding run eval --filter-metadata case_id=sarif-redirects
```
The eval target is `fixtures/repo`, a small synthetic Express app with both true positive and false positive/review cases. Assertions are deterministic:
@@ -60,43 +60,43 @@ ELI5: the dataset says "this exact old commit should be affected" and "this exac
Validate the dataset structure:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run test:dataset
+pnpm --dir evals/triage-finding run test:dataset
```
Run all deterministic calibration checks:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run test:calibration
+pnpm --dir evals/triage-finding run test:calibration
```
Regenerate the Promptfoo calibration tests:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run calibration:generate
+pnpm --dir evals/triage-finding run calibration:generate
```
Hydrate the local OSS checkouts:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run calibration:hydrate
+pnpm --dir evals/triage-finding run calibration:hydrate
```
Validate the calibration Promptfoo config:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run validate:calibration
+pnpm --dir evals/triage-finding run validate:calibration
```
Run one OSS variant as a smoke eval:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:calibration:smoke
+pnpm --dir evals/triage-finding run eval:calibration:smoke
```
Run the full OSS calibration eval:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:calibration
+pnpm --dir evals/triage-finding run eval:calibration
```
The calibration config is separate from the default synthetic eval:
diff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/calibration-evidence.js b/evals/triage-finding/assertions/calibration-evidence.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/assertions/calibration-evidence.js
rename to evals/triage-finding/assertions/calibration-evidence.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/github-rest-intake.js b/evals/triage-finding/assertions/github-rest-intake.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/assertions/github-rest-intake.js
rename to evals/triage-finding/assertions/github-rest-intake.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/missing-input.js b/evals/triage-finding/assertions/missing-input.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/assertions/missing-input.js
rename to evals/triage-finding/assertions/missing-input.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/ticket-intake.js b/evals/triage-finding/assertions/ticket-intake.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/assertions/ticket-intake.js
rename to evals/triage-finding/assertions/ticket-intake.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/assertions/triage-io.js b/evals/triage-finding/assertions/triage-io.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/assertions/triage-io.js
rename to evals/triage-finding/assertions/triage-io.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/datasets/README.md b/evals/triage-finding/datasets/README.md
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/datasets/README.md
rename to evals/triage-finding/datasets/README.md
diff --git a/plugins/codex-security/skills/triage-finding/evals/datasets/triage-calibration-seed.json b/evals/triage-finding/datasets/triage-calibration-seed.json
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/datasets/triage-calibration-seed.json
rename to evals/triage-finding/datasets/triage-calibration-seed.json
diff --git a/plugins/codex-security/skills/triage-finding/evals/fixtures/repo/package-lock.json b/evals/triage-finding/fixtures/repo/package-lock.json
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/fixtures/repo/package-lock.json
rename to evals/triage-finding/fixtures/repo/package-lock.json
diff --git a/plugins/codex-security/skills/triage-finding/evals/fixtures/repo/package.json b/evals/triage-finding/fixtures/repo/package.json
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/fixtures/repo/package.json
rename to evals/triage-finding/fixtures/repo/package.json
diff --git a/plugins/codex-security/skills/triage-finding/evals/fixtures/repo/src/server.js b/evals/triage-finding/fixtures/repo/src/server.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/fixtures/repo/src/server.js
rename to evals/triage-finding/fixtures/repo/src/server.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/fixtures/repo/tests/unsafe-query.test.js b/evals/triage-finding/fixtures/repo/tests/unsafe-query.test.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/fixtures/repo/tests/unsafe-query.test.js
rename to evals/triage-finding/fixtures/repo/tests/unsafe-query.test.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/package.json b/evals/triage-finding/package.json
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/package.json
rename to evals/triage-finding/package.json
diff --git a/plugins/codex-security/skills/triage-finding/evals/pnpm-lock.yaml b/evals/triage-finding/pnpm-lock.yaml
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/pnpm-lock.yaml
rename to evals/triage-finding/pnpm-lock.yaml
diff --git a/plugins/codex-security/skills/triage-finding/evals/pnpm-workspace.yaml b/evals/triage-finding/pnpm-workspace.yaml
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/pnpm-workspace.yaml
rename to evals/triage-finding/pnpm-workspace.yaml
diff --git a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration-smoke.yaml b/evals/triage-finding/promptfooconfig.calibration-smoke.yaml
similarity index 96%
rename from plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration-smoke.yaml
rename to evals/triage-finding/promptfooconfig.calibration-smoke.yaml
index 06a794069..7259be208 100644
--- a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration-smoke.yaml
+++ b/evals/triage-finding/promptfooconfig.calibration-smoke.yaml
@@ -8,7 +8,7 @@ providers:
- id: openai:codex-sdk:gpt-5.5
label: "Codex SDK triage-finding calibration smoke"
config:
- working_dir: ../../../../..
+ working_dir: ../..
sandbox_mode: read-only
approval_policy: never
network_access_enabled: false
diff --git a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration.yaml b/evals/triage-finding/promptfooconfig.calibration.yaml
similarity index 96%
rename from plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration.yaml
rename to evals/triage-finding/promptfooconfig.calibration.yaml
index 18783529b..3ebd971f3 100644
--- a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.calibration.yaml
+++ b/evals/triage-finding/promptfooconfig.calibration.yaml
@@ -8,7 +8,7 @@ providers:
- id: openai:codex-sdk:gpt-5.5
label: "Codex SDK triage-finding calibration"
config:
- working_dir: ../../../../..
+ working_dir: ../..
sandbox_mode: read-only
approval_policy: never
network_access_enabled: false
diff --git a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.yaml b/evals/triage-finding/promptfooconfig.yaml
similarity index 98%
rename from plugins/codex-security/skills/triage-finding/evals/promptfooconfig.yaml
rename to evals/triage-finding/promptfooconfig.yaml
index 7882244aa..088478f43 100644
--- a/plugins/codex-security/skills/triage-finding/evals/promptfooconfig.yaml
+++ b/evals/triage-finding/promptfooconfig.yaml
@@ -8,7 +8,7 @@ providers:
- id: openai:codex-sdk:gpt-5.5
label: "Codex SDK triage-finding"
config:
- working_dir: ../../../../..
+ working_dir: ../..
sandbox_mode: read-only
approval_policy: never
network_access_enabled: false
diff --git a/plugins/codex-security/skills/triage-finding/evals/prompts/triage-request.txt b/evals/triage-finding/prompts/triage-request.txt
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/prompts/triage-request.txt
rename to evals/triage-finding/prompts/triage-request.txt
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/DESIGN.md b/evals/triage-finding/sastbench/DESIGN.md
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/DESIGN.md
rename to evals/triage-finding/sastbench/DESIGN.md
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/README.md b/evals/triage-finding/sastbench/README.md
similarity index 93%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/README.md
rename to evals/triage-finding/sastbench/README.md
index 31860f0e8..31c964fbd 100644
--- a/plugins/codex-security/skills/triage-finding/evals/sastbench/README.md
+++ b/evals/triage-finding/sastbench/README.md
@@ -4,20 +4,20 @@ This Promptfoo lane measures how accurately the source version of `plugins/codex
SastBench contains 2,737 findings: 299 labeled true positive and 2,438 labeled false positive. The JavaScript test generator exposes the scanner claim and the exact affected checkout to Codex while keeping SastBench's label-bearing `finding_id` and ground-truth label out of the rendered prompt. The eval runner stages the skill in a throwaway working directory and uses a deny-by-default Codex permission profile so only that directory, hydrated target repos, and their label-free Git cache plus minimal Codex runtime paths are readable to the model.
-Run commands from the repository root. The parent `evals/` directory owns the pinned Promptfoo and Codex SDK dependencies and the ignored `artifacts/` tree.
+Run commands from the repository root. The parent `evals/triage-finding/` directory owns the pinned Promptfoo and Codex SDK dependencies and the ignored `artifacts/` tree.
## Prepare the benchmark
Install dependencies once:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run setup
+pnpm --dir evals/triage-finding run setup
```
Install the pinned SastBench checkout and hydrate all 275 repository revisions:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run sastbench:prepare
+pnpm --dir evals/triage-finding run sastbench:prepare
```
Preparation verifies the SastBench origin, pinned commit, dataset SHA-256, record counts, target origins, target commits, and clean target worktrees. It must finish before any paid model evaluation begins.
@@ -27,14 +27,14 @@ Preparation verifies the SastBench origin, pinned commit, dataset SHA-256, recor
Run deterministic tests and ask Promptfoo to load the dynamic test generator:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run test:sastbench
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run validate:sastbench
+pnpm --dir evals/triage-finding run test:sastbench
+pnpm --dir evals/triage-finding run validate:sastbench
```
## Run an evaluation
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:sastbench \
+pnpm --dir evals/triage-finding run eval:sastbench \
--output ./artifacts/sastbench-results/canary.promptfoo.jsonl
```
@@ -51,8 +51,8 @@ The sample reuses the full eval's pinned dataset, hydrated repositories, prompt,
Validate and run it from the repository root:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run validate:sastbench:sample
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run eval:sastbench:sample \
+pnpm --dir evals/triage-finding run validate:sastbench:sample
+pnpm --dir evals/triage-finding run eval:sastbench:sample \
--output ./artifacts/sastbench-results/representative-sample.promptfoo.jsonl
```
@@ -130,7 +130,7 @@ Every denominator uses `max(denominator, 1)`. An empty denominator therefore pro
Open the Promptfoo viewer to inspect the completed evaluation:
```bash
-pnpm --dir plugins/codex-security/skills/triage-finding/evals run pf:view
+pnpm --dir evals/triage-finding run pf:view
```
Promptfoo also records cost, latency, and token use without SastBench-specific code.
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/assertions/sastbench-metrics.js b/evals/triage-finding/sastbench/assertions/sastbench-metrics.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/assertions/sastbench-metrics.js
rename to evals/triage-finding/sastbench/assertions/sastbench-metrics.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/assertions/sastbench-verdict.js b/evals/triage-finding/sastbench/assertions/sastbench-verdict.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/assertions/sastbench-verdict.js
rename to evals/triage-finding/sastbench/assertions/sastbench-verdict.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/promptfooconfig.sastbench-sample.yaml b/evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yaml
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/promptfooconfig.sastbench-sample.yaml
rename to evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yaml
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/promptfooconfig.sastbench.yaml b/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/promptfooconfig.sastbench.yaml
rename to evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/prompts/sastbench-triage-request.txt b/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/prompts/sastbench-triage-request.txt
rename to evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/generate-sastbench-tests.js b/evals/triage-finding/sastbench/scripts/generate-sastbench-tests.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/generate-sastbench-tests.js
rename to evals/triage-finding/sastbench/scripts/generate-sastbench-tests.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/hydrate-sastbench-repos.js b/evals/triage-finding/sastbench/scripts/hydrate-sastbench-repos.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/hydrate-sastbench-repos.js
rename to evals/triage-finding/sastbench/scripts/hydrate-sastbench-repos.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/install-sastbench.js b/evals/triage-finding/sastbench/scripts/install-sastbench.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/install-sastbench.js
rename to evals/triage-finding/sastbench/scripts/install-sastbench.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/run-sastbench-promptfoo.js b/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.js
similarity index 94%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/run-sastbench-promptfoo.js
rename to evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.js
index 16cd518f2..2f7736751 100644
--- a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/run-sastbench-promptfoo.js
+++ b/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.js
@@ -7,8 +7,8 @@ const os = require("node:os");
const path = require("node:path");
const EVAL_ROOT = path.resolve(__dirname, "..", "..");
-const TRIAGE_SKILL_ROOT = path.resolve(EVAL_ROOT, "..");
-const PLUGIN_ROOT = path.resolve(TRIAGE_SKILL_ROOT, "..", "..");
+const PLUGIN_ROOT = path.resolve(EVAL_ROOT, "..", "..", "plugins", "codex-security");
+const TRIAGE_SKILL_ROOT = path.join(PLUGIN_ROOT, "skills", "triage-finding");
const DEFAULT_TARGET_ROOT = path.join(EVAL_ROOT, "artifacts", "sastbench-targets");
const DEFAULT_GIT_CACHE_ROOT = path.join(EVAL_ROOT, "artifacts", "sastbench-git-cache");
const PROMPTFOO_BIN = path.join(EVAL_ROOT, "node_modules", ".bin", "promptfoo");
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/sastbench-lib.js b/evals/triage-finding/sastbench/scripts/sastbench-lib.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/sastbench-lib.js
rename to evals/triage-finding/sastbench/scripts/sastbench-lib.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/sastbench-result.js b/evals/triage-finding/sastbench/scripts/sastbench-result.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/sastbench-result.js
rename to evals/triage-finding/sastbench/scripts/sastbench-result.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-hydrate-sastbench-repos.js b/evals/triage-finding/sastbench/scripts/test-hydrate-sastbench-repos.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-hydrate-sastbench-repos.js
rename to evals/triage-finding/sastbench/scripts/test-hydrate-sastbench-repos.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-install-sastbench.js b/evals/triage-finding/sastbench/scripts/test-install-sastbench.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-install-sastbench.js
rename to evals/triage-finding/sastbench/scripts/test-install-sastbench.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-promptfoo-native-harness.js b/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-promptfoo-native-harness.js
rename to evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-sastbench-lib.js b/evals/triage-finding/sastbench/scripts/test-sastbench-lib.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-sastbench-lib.js
rename to evals/triage-finding/sastbench/scripts/test-sastbench-lib.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-sastbench-result.js b/evals/triage-finding/sastbench/scripts/test-sastbench-result.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/sastbench/scripts/test-sastbench-result.js
rename to evals/triage-finding/sastbench/scripts/test-sastbench-result.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/schemas/triage-result-v0.schema.json b/evals/triage-finding/schemas/triage-result-v0.schema.json
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/schemas/triage-result-v0.schema.json
rename to evals/triage-finding/schemas/triage-result-v0.schema.json
diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/generate-calibration-tests.js b/evals/triage-finding/scripts/generate-calibration-tests.js
similarity index 98%
rename from plugins/codex-security/skills/triage-finding/evals/scripts/generate-calibration-tests.js
rename to evals/triage-finding/scripts/generate-calibration-tests.js
index 7bafe1cfc..1ac5f23e3 100644
--- a/plugins/codex-security/skills/triage-finding/evals/scripts/generate-calibration-tests.js
+++ b/evals/triage-finding/scripts/generate-calibration-tests.js
@@ -7,7 +7,7 @@ const path = require("node:path");
const DEFAULT_DATASET = path.join(__dirname, "..", "datasets", "triage-calibration-seed.json");
const DEFAULT_OUTPUT = path.join(__dirname, "..", "tests", "calibration-oss.yaml");
const DEFAULT_REPO_ROOT =
- "plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos";
+ "evals/triage-finding/artifacts/calibration-repos";
function parseArgs(argv) {
const args = {
diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/hydrate-calibration-repos.js b/evals/triage-finding/scripts/hydrate-calibration-repos.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/scripts/hydrate-calibration-repos.js
rename to evals/triage-finding/scripts/hydrate-calibration-repos.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-app-surface-instructions.js b/evals/triage-finding/scripts/test-app-surface-instructions.js
similarity index 87%
rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-app-surface-instructions.js
rename to evals/triage-finding/scripts/test-app-surface-instructions.js
index 257428c48..0d51c5fe8 100644
--- a/plugins/codex-security/skills/triage-finding/evals/scripts/test-app-surface-instructions.js
+++ b/evals/triage-finding/scripts/test-app-surface-instructions.js
@@ -2,13 +2,15 @@ const assert = require("node:assert/strict");
const fs = require("node:fs");
const path = require("node:path");
-const skillPath = path.join(__dirname, "..", "..", "SKILL.md");
+const pluginRoot = path.resolve(__dirname, "..", "..", "..", "plugins", "codex-security");
+const skillRoot = path.join(pluginRoot, "skills", "triage-finding");
+const skillPath = path.join(skillRoot, "SKILL.md");
const skill = fs.readFileSync(skillPath, "utf8");
-const ticketIntakePath = path.join(__dirname, "..", "..", "references", "ticket-intake.md");
+const ticketIntakePath = path.join(skillRoot, "references", "ticket-intake.md");
const ticketIntake = fs.readFileSync(ticketIntakePath, "utf8");
-const agentPath = path.join(__dirname, "..", "..", "agents", "openai.yaml");
+const agentPath = path.join(skillRoot, "agents", "openai.yaml");
const agent = fs.readFileSync(agentPath, "utf8");
-const pluginPath = path.join(__dirname, "..", "..", "..", "..", ".codex-plugin", "plugin.json");
+const pluginPath = path.join(pluginRoot, ".codex-plugin", "plugin.json");
const plugin = JSON.parse(fs.readFileSync(pluginPath, "utf8"));
assert.match(skill, /## Jira and Linear Intake/);
diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-calibration-dataset.js b/evals/triage-finding/scripts/test-calibration-dataset.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-calibration-dataset.js
rename to evals/triage-finding/scripts/test-calibration-dataset.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-calibration-evidence.js b/evals/triage-finding/scripts/test-calibration-evidence.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-calibration-evidence.js
rename to evals/triage-finding/scripts/test-calibration-evidence.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-generate-calibration-tests.js b/evals/triage-finding/scripts/test-generate-calibration-tests.js
similarity index 94%
rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-generate-calibration-tests.js
rename to evals/triage-finding/scripts/test-generate-calibration-tests.js
index 7644f30da..a70d34fa2 100644
--- a/plugins/codex-security/skills/triage-finding/evals/scripts/test-generate-calibration-tests.js
+++ b/evals/triage-finding/scripts/test-generate-calibration-tests.js
@@ -29,7 +29,7 @@ assert.match(generatedYaml, /case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerabl
assert.match(generatedYaml, /case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed/);
assert.match(generatedYaml, /expected_verdicts: confirmed/);
assert.match(generatedYaml, /expected_verdicts: not_actionable/);
-assert.match(generatedYaml, /target_repo: plugins\/codex-security\/skills\/triage-finding\/evals\/artifacts\/calibration-repos\//);
+assert.match(generatedYaml, /target_repo: evals\/triage-finding\/artifacts\/calibration-repos\//);
assert.doesNotMatch(
generatedYaml,
/expected_evidence_terms:\n\s+- /,
diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-hydrate-calibration-repos.js b/evals/triage-finding/scripts/test-hydrate-calibration-repos.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-hydrate-calibration-repos.js
rename to evals/triage-finding/scripts/test-hydrate-calibration-repos.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/scripts/test-triage-io.js b/evals/triage-finding/scripts/test-triage-io.js
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/scripts/test-triage-io.js
rename to evals/triage-finding/scripts/test-triage-io.js
diff --git a/plugins/codex-security/skills/triage-finding/evals/tests/calibration-oss.yaml b/evals/triage-finding/tests/calibration-oss.yaml
similarity index 93%
rename from plugins/codex-security/skills/triage-finding/evals/tests/calibration-oss.yaml
rename to evals/triage-finding/tests/calibration-oss.yaml
index f460edfde..89e3ab0c1 100644
--- a/plugins/codex-security/skills/triage-finding/evals/tests/calibration-oss.yaml
+++ b/evals/triage-finding/tests/calibration-oss.yaml
@@ -8,7 +8,7 @@
expected_binary_label: positive
vars:
case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerable
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/vulnerable
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/vulnerable
source_type_under_test: cve
expected_ids: GHSA-73vx-49mv-v8w5-000-vulnerable
expected_source_types: cve
@@ -43,7 +43,7 @@
expected_binary_label: negative
vars:
case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/fixed
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/fixed
source_type_under_test: cve
expected_ids: GHSA-73vx-49mv-v8w5-000-fixed
expected_source_types: cve
@@ -78,7 +78,7 @@
expected_binary_label: positive
vars:
case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-vulnerable
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/vulnerable
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/vulnerable
source_type_under_test: cve
expected_ids: GHSA-qpgx-jfcq-r59f-000-vulnerable
expected_source_types: cve
@@ -113,7 +113,7 @@
expected_binary_label: negative
vars:
case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-fixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/fixed
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/fixed
source_type_under_test: cve
expected_ids: GHSA-qpgx-jfcq-r59f-000-fixed
expected_source_types: cve
@@ -148,7 +148,7 @@
expected_binary_label: positive
vars:
case_id: oss-scitokens-ghsa-rh5m-2482-966c-vulnerable
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/vulnerable
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/vulnerable
source_type_under_test: cve
expected_ids: GHSA-rh5m-2482-966c-000-vulnerable
expected_source_types: cve
@@ -183,7 +183,7 @@
expected_binary_label: negative
vars:
case_id: oss-scitokens-ghsa-rh5m-2482-966c-fixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/fixed
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/fixed
source_type_under_test: cve
expected_ids: GHSA-rh5m-2482-966c-000-fixed
expected_source_types: cve
@@ -218,7 +218,7 @@
expected_binary_label: positive
vars:
case_id: oss-emissary-ghsa-3p24-9x7v-7789-vulnerable
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/vulnerable
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/vulnerable
source_type_under_test: cve
expected_ids: GHSA-3p24-9x7v-7789-000-vulnerable
expected_source_types: cve
@@ -253,7 +253,7 @@
expected_binary_label: negative
vars:
case_id: oss-emissary-ghsa-3p24-9x7v-7789-fixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/fixed
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/fixed
source_type_under_test: cve
expected_ids: GHSA-3p24-9x7v-7789-000-fixed
expected_source_types: cve
@@ -288,7 +288,7 @@
expected_binary_label: positive
vars:
case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-vulnerable
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/vulnerable
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/vulnerable
source_type_under_test: cve
expected_ids: GHSA-6RGM-GR97-X3J5-vulnerable
expected_source_types: cve
@@ -323,7 +323,7 @@
expected_binary_label: negative
vars:
case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-fixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/fixed
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/fixed
source_type_under_test: cve
expected_ids: GHSA-6RGM-GR97-X3J5-fixed
expected_source_types: cve
@@ -358,7 +358,7 @@
expected_binary_label: positive
vars:
case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-vulnerable
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/vulnerable
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/vulnerable
source_type_under_test: cve
expected_ids: GHSA-27vg-33gh-4hwg-000-vulnerable
expected_source_types: cve
@@ -393,7 +393,7 @@
expected_binary_label: negative
vars:
case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-fixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/fixed
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/fixed
source_type_under_test: cve
expected_ids: GHSA-27vg-33gh-4hwg-000-fixed
expected_source_types: cve
@@ -428,7 +428,7 @@
expected_binary_label: positive
vars:
case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-vulnerable
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/vulnerable
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/vulnerable
source_type_under_test: cve
expected_ids: GHSA-v8jm-5vwx-cfxm-gap-fill-000-vulnerable
expected_source_types: cve
@@ -463,7 +463,7 @@
expected_binary_label: negative
vars:
case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-fixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/fixed
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/fixed
source_type_under_test: cve
expected_ids: GHSA-v8jm-5vwx-cfxm-gap-fill-000-fixed
expected_source_types: cve
@@ -498,7 +498,7 @@
expected_binary_label: positive
vars:
case_id: oss-monetr-ghsa-29v9-frvh-c426-vulnerable
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/vulnerable
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/vulnerable
source_type_under_test: cve
expected_ids: GHSA-29v9-frvh-c426-000-vulnerable
expected_source_types: cve
@@ -533,7 +533,7 @@
expected_binary_label: negative
vars:
case_id: oss-monetr-ghsa-29v9-frvh-c426-fixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/fixed
+ target_repo: evals/triage-finding/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/fixed
source_type_under_test: cve
expected_ids: GHSA-29v9-frvh-c426-000-fixed
expected_source_types: cve
diff --git a/plugins/codex-security/skills/triage-finding/evals/tests/github-rest-intake.yaml b/evals/triage-finding/tests/github-rest-intake.yaml
similarity index 100%
rename from plugins/codex-security/skills/triage-finding/evals/tests/github-rest-intake.yaml
rename to evals/triage-finding/tests/github-rest-intake.yaml
diff --git a/plugins/codex-security/skills/triage-finding/evals/tests/input-types.yaml b/evals/triage-finding/tests/input-types.yaml
similarity index 92%
rename from plugins/codex-security/skills/triage-finding/evals/tests/input-types.yaml
rename to evals/triage-finding/tests/input-types.yaml
index 5f0ad7146..1c950751c 100644
--- a/plugins/codex-security/skills/triage-finding/evals/tests/input-types.yaml
+++ b/evals/triage-finding/tests/input-types.yaml
@@ -4,7 +4,7 @@
source_type: sarif
vars:
case_id: sarif-redirects
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: sarif
expected_ids: SARIF-OPEN-REDIRECT,SARIF-SAFE-REDIRECT
expected_source_types: sarif,sarif
@@ -27,7 +27,7 @@
source_type: cve
vars:
case_id: cve-dependencies
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: cve
expected_ids: CVE-2026-0001,CVE-2026-0002
expected_source_types: cve,cve
@@ -50,7 +50,7 @@
source_type: advisory
vars:
case_id: advisory-downloads
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: advisory
expected_ids: ADV-DOWNLOAD-TRAVERSAL,ADV-SAFE-DOWNLOAD
expected_source_types: advisory,advisory
@@ -70,7 +70,7 @@
source_type: scanner_ticket
vars:
case_id: scanner-query
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
expected_ids: SCAN-SQL-001,SCAN-SQL-TEST-ONLY
expected_source_types: scanner_ticket,scanner_ticket
@@ -92,7 +92,7 @@
source_type: bug_bounty
vars:
case_id: bug-bounty-access
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: bug_bounty
expected_ids: BB-ORDER-IDOR,BB-PARTNER-EXPORT
expected_source_types: bug_bounty,bug_bounty
@@ -112,7 +112,7 @@
source_type: codex_security_finding
vars:
case_id: codex-security-findings
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: codex_security_finding
expected_ids: csf_111111111111111111111111,csf_222222222222222222222222
expected_source_types: codex_security_finding,codex_security_finding
@@ -160,7 +160,7 @@
source_type: freeform
vars:
case_id: freeform-mixed
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: freeform
expected_ids: FREEFORM-REDIRECT,FREEFORM-UNSPECIFIED-AUTH
expected_source_types: freeform,freeform
@@ -178,7 +178,7 @@
source_type: unknown
vars:
case_id: unknown-fallback
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: unknown
expected_ids: UNKNOWN-001,UNKNOWN-002
expected_source_types: unknown,unknown
diff --git a/plugins/codex-security/skills/triage-finding/evals/tests/invocation-behavior.yaml b/evals/triage-finding/tests/invocation-behavior.yaml
similarity index 87%
rename from plugins/codex-security/skills/triage-finding/evals/tests/invocation-behavior.yaml
rename to evals/triage-finding/tests/invocation-behavior.yaml
index 564df02d7..6526d1978 100644
--- a/plugins/codex-security/skills/triage-finding/evals/tests/invocation-behavior.yaml
+++ b/evals/triage-finding/tests/invocation-behavior.yaml
@@ -4,7 +4,7 @@
source_type: none
vars:
case_id: missing-finding-input
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: none
finding_input: ""
invocation_preamble: "Use $codex-security:triage-finding with no finding supplied."
diff --git a/plugins/codex-security/skills/triage-finding/evals/tests/ticket-intake.yaml b/evals/triage-finding/tests/ticket-intake.yaml
similarity index 91%
rename from plugins/codex-security/skills/triage-finding/evals/tests/ticket-intake.yaml
rename to evals/triage-finding/tests/ticket-intake.yaml
index acfd45b3a..2a8fd4b2f 100644
--- a/plugins/codex-security/skills/triage-finding/evals/tests/ticket-intake.yaml
+++ b/evals/triage-finding/tests/ticket-intake.yaml
@@ -4,7 +4,7 @@
source_type: scanner_ticket
vars:
case_id: ticket-connector-unavailable
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear issue SEC-293. Connector observation: the Linear connector is unavailable because it is not connected."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
@@ -25,7 +25,7 @@
source_type: scanner_ticket
vars:
case_id: ticket-connector-permission
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear issue SEC-293. Connector observation: access was denied because the authenticated user lacks permission."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
@@ -46,7 +46,7 @@
source_type: scanner_ticket
vars:
case_id: ticket-not-found
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear issue SEC-999999. Connector observation: the issue was not found or is inaccessible."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-999999."
@@ -67,7 +67,7 @@
source_type: scanner_ticket
vars:
case_id: ticket-transient-retry-exhausted
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear issue SEC-293. Connector observation: the first read returned a transient timeout; the identical retry also timed out."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
@@ -89,7 +89,7 @@
source_type: scanner_ticket
vars:
case_id: linear-direct-subissue-confirmation
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear parent SEC-293 was fetched. Child listing returned SEC-294 'Auth bypass' and SEC-295 'Unsafe redirect'. Their full bodies have not been fetched."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
@@ -110,7 +110,7 @@
source_type: scanner_ticket
vars:
case_id: linear-nested-subissue-confirmation
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "The user approved direct child SEC-294. Its full body was fetched. The next child listing found SEC-296 'Nested token leak'; its full body has not been fetched."
invocation_preamble: "Continue $codex-security:triage-finding for the approved Linear issue tree."
@@ -131,7 +131,7 @@
source_type: scanner_ticket
vars:
case_id: linear-ambiguous-parent
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear parent SEC-293 contains collection notes plus a possible standalone authorization claim. The user approved importing its child findings."
invocation_preamble: "Continue $codex-security:triage-finding for Linear parent SEC-293."
@@ -152,7 +152,7 @@
source_type: scanner_ticket
vars:
case_id: linear-subissue-limit
- target_repo: plugins/codex-security/skills/triage-finding/evals/fixtures/repo
+ target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "The approved Linear parent and selected sub-issue levels contain 251 triage items. No issue bodies beyond the limit have been imported."
invocation_preamble: "Continue $codex-security:triage-finding for this Linear issue tree."
diff --git a/plugins/codex-security/mcp-app/evals/README.md b/plugins/codex-security/mcp-app/evals/README.md
deleted file mode 100644
index b8ac619b4..000000000
--- a/plugins/codex-security/mcp-app/evals/README.md
+++ /dev/null
@@ -1,25 +0,0 @@
-# Deep reducer paging eval
-
-This eval runs one reducer against synthetic persisted worker findings and a previous aggregate. It uses the pinned Codex CLI, its external code-mode host, and the production artifact tool handlers. It does not launch a repository scan.
-
-The fixture has a 32 MiB individual semantic field containing quotes, backslashes, newlines, and Unicode. Its old duplicated MCP response is about 114 MiB. The first input read deliberately returns that old response; all subsequent calls use the production paged implementation. The IPC error comes from the real transport, not a mocked error message.
-
-The grader requires a smaller byte budget on the retry with the same cursor and reference, byte-bounded subsequent pages, one successful result submission, every source accounted for exactly once, complete original payloads matching their SHA-256 hashes, and preservation of the previous finding identity and synthesized history.
-
-Install the MCP app's locked dependencies using the repository's normal setup, then run from the repository root:
-
-```sh
-node plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs deterministic
-```
-
-The deterministic mode supplies scripted model responses through a local HTTP provider. It uses a private temporary Codex home and no model credentials or external model requests. The script reads every input page, fetches original and prior canonical details by reference, and records the reduction through the real tools. This mode also runs in the normal MCP test suite as `test_deep_reducer_paging_eval.mjs`.
-
-To evaluate a model's recovery decisions using the production reducer prompt:
-
-```sh
-node plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs model
-```
-
-The model run uses the caller's normal Codex credentials and configuration and consumes model usage. An optional final argument selects a model; otherwise Codex uses its configured model. The fault injection and result grader are the same. This mode is opt-in and does not run in CI. Unlike the deterministic run, it does not supply the recovery code to the model. The SDK does not expose every code-mode error event, so the model-mode grade relies on the observed oversized response, changed request, and saved result; the deterministic mode additionally asserts the exact transport diagnostic.
-
-Each command prints the path to an ignored directory under `mcp-app/reports/` containing `report.json`, the compact tool-call trace, and the generated artifacts. The fixture is generated at runtime; no large payload files are checked in.
diff --git a/plugins/codex-security/mcp-app/evals/controlled-code-mode.mjs b/plugins/codex-security/mcp-app/tests/support/reducer-paging/controlled-code-mode.mjs
similarity index 100%
rename from plugins/codex-security/mcp-app/evals/controlled-code-mode.mjs
rename to plugins/codex-security/mcp-app/tests/support/reducer-paging/controlled-code-mode.mjs
diff --git a/plugins/codex-security/mcp-app/evals/deep-reducer-paging-fixture.mjs b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-fixture.mjs
similarity index 100%
rename from plugins/codex-security/mcp-app/evals/deep-reducer-paging-fixture.mjs
rename to plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-fixture.mjs
diff --git a/plugins/codex-security/mcp-app/evals/deep-reducer-paging-server.mjs b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-server.mjs
similarity index 92%
rename from plugins/codex-security/mcp-app/evals/deep-reducer-paging-server.mjs
rename to plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-server.mjs
index 6703ab1a4..97c42aebb 100644
--- a/plugins/codex-security/mcp-app/evals/deep-reducer-paging-server.mjs
+++ b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-server.mjs
@@ -1,8 +1,8 @@
import { appendFile, readFile } from "node:fs/promises";
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
-import { getCodexSecurityDeepReducerInputs } from "../src/artifact-deep-reducer.ts";
-import { registerCompactWorkerArtifactTools } from "../src/server/compact-artifact-tools.ts";
+import { getCodexSecurityDeepReducerInputs } from "../../../src/artifact-deep-reducer.ts";
+import { registerCompactWorkerArtifactTools } from "../../../src/server/compact-artifact-tools.ts";
async function main() {
const { context, tracePath } = JSON.parse(
diff --git a/plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging.mjs
similarity index 91%
rename from plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs
rename to plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging.mjs
index 505ace7c1..7d22a9485 100644
--- a/plugins/codex-security/mcp-app/evals/deep-reducer-paging.mjs
+++ b/plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging.mjs
@@ -1,5 +1,5 @@
import assert from "node:assert/strict";
-import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
+import { readFile, writeFile } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { Codex } from "@openai/codex-sdk";
@@ -10,7 +10,7 @@ import {
gradeReducerPagingResult,
} from "./deep-reducer-paging-fixture.mjs";
-const evalDirectory = path.dirname(fileURLToPath(import.meta.url));
+const supportDirectory = path.dirname(fileURLToPath(import.meta.url));
/** Run one reducer through real code-mode IPC and the production artifact tools. */
export async function runReducerPagingEval({
@@ -31,7 +31,9 @@ export async function runReducerPagingEval({
);
await Promise.all([
build({
- entryPoints: [path.join(evalDirectory, "deep-reducer-paging-server.mjs")],
+ entryPoints: [
+ path.join(supportDirectory, "deep-reducer-paging-server.mjs"),
+ ],
outfile: serverPath,
bundle: true,
platform: "node",
@@ -39,7 +41,9 @@ export async function runReducerPagingEval({
loader: { ".md": "text" },
}),
build({
- entryPoints: [path.join(evalDirectory, "../src/deep-scan/templates.ts")],
+ entryPoints: [
+ path.join(supportDirectory, "../../../src/deep-scan/templates.ts"),
+ ],
outfile: promptModulePath,
bundle: true,
platform: "node",
@@ -253,16 +257,3 @@ const result = await tools.mcp__cs_artifacts__record_codex_security_deep_reducti
if (result.isError) throw new Error(result.content[0].text);
text({reductionRecorded: true});
`;
-
-if (process.argv[1] === fileURLToPath(import.meta.url)) {
- const reports = path.join(evalDirectory, "../reports");
- await mkdir(reports, { recursive: true });
- const root = await mkdtemp(path.join(reports, "deep-reducer-paging-"));
- console.log(`Eval artifacts: ${root}`);
- const report = await runReducerPagingEval({
- root,
- mode: process.argv[2] ?? "deterministic",
- model: process.argv[3],
- });
- console.log(JSON.stringify(report, null, 2));
-}
diff --git a/plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs b/plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs
index e97926009..bf9fb1307 100644
--- a/plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs
+++ b/plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.mjs
@@ -6,11 +6,11 @@ import test from "node:test";
import {
gradeReducerPagingTrace,
runReducerPagingEval,
-} from "../evals/deep-reducer-paging.mjs";
+} from "./support/reducer-paging/deep-reducer-paging.mjs";
import {
createReducerPagingFixture,
gradeReducerPagingResult,
-} from "../evals/deep-reducer-paging-fixture.mjs";
+} from "./support/reducer-paging/deep-reducer-paging-fixture.mjs";
test("a reducer recovers from the real IPC frame limit and records all sources", async (t) => {
const root = await mkdtemp(path.join(tmpdir(), "deep-reducer-ipc-eval-"));
diff --git a/sdk/typescript/tests-ts/runtime.test.ts b/sdk/typescript/tests-ts/runtime.test.ts
index fc0d5e957..f47aef5ff 100644
--- a/sdk/typescript/tests-ts/runtime.test.ts
+++ b/sdk/typescript/tests-ts/runtime.test.ts
@@ -4791,9 +4791,12 @@ describe("runtime directories and plugin Python boundary", () => {
expect(result["details"]).toHaveLength(5 * 1024 * 1024);
});
- test.each([false, true])(
- "selects workbench Git for the requested target (bound: %s)",
- async (bound) => {
+ test.each([
+ ["unbound", false],
+ ["bound", true],
+ ] as const)(
+ "selects workbench Git for the requested target (%s)",
+ async (_label, bound) => {
const root = await temporaryDirectory();
const repository = join(root, "repository");
const untrustedBin = join(repository, "tools");
diff --git a/sdk/typescript/tests-ts/scan-logs.test.ts b/sdk/typescript/tests-ts/scan-logs.test.ts
index 972637477..f383cfafc 100644
--- a/sdk/typescript/tests-ts/scan-logs.test.ts
+++ b/sdk/typescript/tests-ts/scan-logs.test.ts
@@ -220,9 +220,12 @@ describe("saved scan logs", () => {
expect(JSON.stringify(result)).not.toContain("unrelated");
});
- test.each([undefined, "missing-owner"])(
- "feedback collects known worker descendants without the owner log with continuation %s",
- async (continuationThreadId) => {
+ test.each([
+ ["omitted", undefined],
+ ["recorded", "missing-owner"],
+ ] as const)(
+ "feedback collects known worker descendants without the owner log with %s continuation",
+ async (_label, continuationThreadId) => {
const home = await temporaryHome();
await writeSession(home, "owner-child", [], "missing-owner");
await writeSession(home, "worker", [
diff --git a/sdk/typescript/tests-ts/skeleton.test.ts b/sdk/typescript/tests-ts/skeleton.test.ts
index 5b6bfaec1..b5439c3cf 100644
--- a/sdk/typescript/tests-ts/skeleton.test.ts
+++ b/sdk/typescript/tests-ts/skeleton.test.ts
@@ -62,7 +62,7 @@ describe("TypeScript package skeleton", () => {
const directories = [
"sdk/typescript",
"plugins/codex-security/mcp-app",
- "plugins/codex-security/skills/triage-finding/evals",
+ "evals/triage-finding",
];
const manifests = await Promise.all(
directories.map(async (directory) =>