Skip to content

OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security

refactor(evals): remove Promptfoo patch and generated calibration YAML (#1678)

Michael D'Angelo88a6f6b0c31d
diff --git a/evals/triage-finding/.gitignore b/evals/triage-finding/.gitignore
index 5c33fe9d6..e68bc63df 100644
--- a/evals/triage-finding/.gitignore
+++ b/evals/triage-finding/.gitignore
@@ -1,3 +1,4 @@
node_modules/
.promptfoo/
artifacts/
+tests/calibration-oss.yaml
diff --git a/evals/triage-finding/README.md b/evals/triage-finding/README.md
index e6402f6a2..6548b6dd2 100644
--- a/evals/triage-finding/README.md
+++ b/evals/triage-finding/README.md
@@ -25,7 +25,9 @@ pnpm --dir plugins/codex-security/mcp-app install --frozen-lockfile
node plugins/codex-security/mcp-app/scripts/build_native.mjs
```
-The common runner builds the staged helper for validation, evaluation, and replay, then removes its temporary runtime after Promptfoo exits. The runtime extension runs after Promptfoo loads the native Codex provider, preserving its authentication precedence. A pinned Promptfoo patch runs saved `:beforeAll` startup hooks during viewer replay. Replay does not restore grading metadata, so grading hooks remain disabled.
+The common runner builds the staged helper for validation, evaluation, and replay, then removes its temporary runtime after Promptfoo exits. Separate YAML files under `providers/` retain the native Codex provider IDs and each suite's settings. The suite's `$ref` entries expand these settings into saved evaluation snapshots, so later edits to the provider files do not change an existing run's retry, resume, or replay settings. The runner uses Node's `--import` to preload the runtime adapter into the Promptfoo child process, so it can bind the staged runtime during evaluation, retry, resume, and viewer replay without a Promptfoo patch. Native provider loading preserves authentication precedence.
+
+The adapter depends on bundled Codex classes and internal methods in the pinned Promptfoo provider. The provider tests cover those methods, executable selection, authentication, and per-case settings. Replay does not restore grading metadata, so grading hooks remain disabled.
Saved runs whose provider ID is `file://.../triage-provider.mts` need to be rerun with the current configuration before retry, resume, or replay. Their saved results remain intact. Native Codex provider IDs are unchanged.
@@ -72,7 +74,7 @@ cases use freeform answers.
## Calibration Dataset
-`datasets/triage-calibration-seed.json` is the first OSS-only calibration dataset for scaling beyond the synthetic fixture app. It contains public OSS vulnerable/fixed commit pairs. Each dataset variant becomes one Promptfoo test case in `tests/calibration-oss.yaml`, and each test points Codex at a pinned local checkout under `artifacts/calibration-repos/`. Case IDs, finding IDs, and checkout directories use stable opaque names. Variant labels, fix references, and checkout commit IDs remain in harness metadata rather than the model prompt. Hydration keeps Git metadata in opaque sibling directories outside the model-readable checkout. Re-run `calibration:hydrate` to update older checkout layouts.
+`datasets/triage-calibration-seed.json` is the first OSS-only calibration dataset for scaling beyond the synthetic fixture app. It contains public OSS vulnerable/fixed commit pairs. Promptfoo calls `scripts/generate-calibration-tests.mts:generateTests` to load each dataset variant as a test case, and each test points Codex at a pinned local checkout under `artifacts/calibration-repos/`. Case IDs, finding IDs, and checkout directories use stable opaque names. Variant labels, fix references, and checkout commit IDs remain in harness metadata rather than the model prompt. Hydration keeps Git metadata in opaque sibling directories outside the model-readable checkout. Re-run `calibration:hydrate` to update older checkout layouts.
ELI5: the dataset says "this exact old commit should be affected" and "this exact fixed commit should not be affected." The generator turns those rows into Promptfoo test prompts. The hydrator downloads the exact repo commits so Codex can inspect real code instead of synthetic snippets.
@@ -88,12 +90,14 @@ Run all deterministic calibration checks:
pnpm --dir evals/triage-finding run test:calibration
```
-Regenerate the Promptfoo calibration tests:
+Validation and evaluation load calibration tests directly from the dataset. To export a YAML copy for inspection:
```bash
-pnpm --dir evals/triage-finding run calibration:generate
+pnpm --dir evals/triage-finding run calibration:generate --output artifacts/calibration-oss.yaml
```
+The optional exporter retains `--dataset`, `--output`, `--repo-root`, `--case`, and `--variant`. Its default output remains `tests/calibration-oss.yaml`; generated YAML is not needed to run the suite.
+
Hydrate the local OSS checkouts:
```bash
@@ -122,7 +126,7 @@ The calibration config is separate from the default synthetic eval:
- `promptfooconfig.yaml` covers the small fixture app and GitHub intake routing.
- `promptfooconfig.calibration.yaml` covers the OSS vulnerable/fixed commit pairs and adds `assertions/calibration-evidence.mts` to check that required paths or fix commits are cited in the response.
-- `promptfooconfig.calibration-smoke.yaml` covers a generated one-row smoke file under `artifacts/calibration-smoke.yaml`.
+- `promptfooconfig.calibration-smoke.yaml` passes case and variant filters to the native generator to load one smoke case.
The calibration eval uses the Codex SDK provider, which shells out to `codex exec`. Run it from a normal shell or an unsandboxed command runner so the Codex CLI can write its normal `$CODEX_HOME` state.
diff --git a/evals/triage-finding/datasets/README.md b/evals/triage-finding/datasets/README.md
index bfb354ea8..42b56f1d6 100644
--- a/evals/triage-finding/datasets/README.md
+++ b/evals/triage-finding/datasets/README.md
@@ -17,7 +17,7 @@ The important distinction is that GitHub advisory state is source material, not
- `triage-calibration-seed.json` is the first OSS-only seed dataset. It contains public vulnerable/fixed commit pairs.
- `../scripts/test-calibration-dataset.mts` performs deterministic structural validation so future additions do not silently break the dataset shape.
-- `../scripts/generate-calibration-tests.mts` converts the dataset into Promptfoo tests at `../tests/calibration-oss.yaml`.
+- `../scripts/generate-calibration-tests.mts` supplies test objects directly to Promptfoo and can optionally export them as YAML.
- `../scripts/hydrate-calibration-repos.mts` downloads the pinned OSS checkouts under `../artifacts/calibration-repos/` for local eval runs.
## Case Families
diff --git a/evals/triage-finding/package.json b/evals/triage-finding/package.json
index 8862ef327..e5fc4a7c3 100644
--- a/evals/triage-finding/package.json
+++ b/evals/triage-finding/package.json
@@ -3,7 +3,7 @@
"private": true,
"description": "Promptfoo evals for the Codex Security triage-finding skill",
"scripts": {
- "setup": "pnpm install --frozen-lockfile && pnpm rebuild better-sqlite3 && pnpm --dir ../../plugins/codex-security/mcp-app install --frozen-lockfile && pnpm --dir ../../sdk/typescript run build:evals && node ../../plugins/codex-security/mcp-app/scripts/build_native.mjs",
+ "setup": "pnpm install --frozen-lockfile && pnpm --dir ../../plugins/codex-security/mcp-app install --frozen-lockfile && pnpm --dir ../../sdk/typescript run build:evals && node ../../plugins/codex-security/mcp-app/scripts/build_native.mjs",
"calibration:generate": "node --experimental-strip-types scripts/generate-calibration-tests.mts",
"calibration:hydrate": "node --experimental-strip-types scripts/hydrate-calibration-repos.mts",
"sastbench:install": "node --experimental-strip-types sastbench/scripts/install-sastbench.mts",
@@ -18,13 +18,13 @@
"test:instructions": "node --experimental-strip-types scripts/test-app-surface-instructions.mts",
"test:sastbench": "node --experimental-strip-types sastbench/scripts/test-sastbench-lib.mts && node --experimental-strip-types sastbench/scripts/test-install-sastbench.mts && node --experimental-strip-types sastbench/scripts/test-hydrate-sastbench-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-result.mts && node --experimental-strip-types sastbench/scripts/test-promptfoo-native-harness.mts",
"validate": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts validate config -c ./promptfooconfig.yaml",
- "validate:calibration": "pnpm run calibration:generate && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts validate config -c ./promptfooconfig.calibration.yaml",
+ "validate:calibration": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts validate config -c ./promptfooconfig.calibration.yaml",
"validate:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts validate config -c ./sastbench/promptfooconfig.sastbench.yaml",
"validate:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts validate config -c ./sastbench/promptfooconfig.sastbench-sample.yaml",
"eval": "mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts eval -c ./promptfooconfig.yaml -o ./artifacts/triage-finding-eval.json --no-cache --no-share",
"eval:smoke": "mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts eval -c ./promptfooconfig.yaml -o ./artifacts/triage-finding-eval-smoke.json --no-cache --no-share --filter-range 0:1",
- "eval:calibration": "pnpm run calibration:generate && pnpm run calibration:hydrate && mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts eval -c ./promptfooconfig.calibration.yaml -o ./artifacts/triage-finding-calibration-eval.json --no-cache --no-share",
- "eval:calibration:smoke": "node --experimental-strip-types scripts/generate-calibration-tests.mts --case oss-dompurify-ghsa-v8jm-5vwx-cfxm --variant vulnerable --output artifacts/calibration-smoke.yaml && node --experimental-strip-types scripts/hydrate-calibration-repos.mts --case oss-dompurify-ghsa-v8jm-5vwx-cfxm --variant vulnerable && mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts eval -c ./promptfooconfig.calibration-smoke.yaml -o ./artifacts/triage-finding-calibration-smoke.json --no-cache --no-share",
+ "eval:calibration": "pnpm run calibration:hydrate && mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts eval -c ./promptfooconfig.calibration.yaml -o ./artifacts/triage-finding-calibration-eval.json --no-cache --no-share",
+ "eval:calibration:smoke": "node --experimental-strip-types scripts/hydrate-calibration-repos.mts --case oss-dompurify-ghsa-v8jm-5vwx-cfxm --variant vulnerable && mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts eval -c ./promptfooconfig.calibration-smoke.yaml -o ./artifacts/triage-finding-calibration-smoke.json --no-cache --no-share",
"eval:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts eval -c ./sastbench/promptfooconfig.sastbench.yaml --no-cache --no-share",
"eval:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts eval -c ./sastbench/promptfooconfig.sastbench-sample.yaml --no-cache --no-share --max-concurrency 32",
"pf:view": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types scripts/run-promptfoo.mts view"
diff --git a/evals/triage-finding/patches/[email protected] b/evals/triage-finding/patches/[email protected]
deleted file mode 100644
index 5c3293d29..000000000
--- a/evals/triage-finding/patches/[email protected]
+++ /dev/null
@@ -1,22 +0,0 @@
-diff --git a/dist/src/main.js b/dist/src/main.js
---- a/dist/src/main.js
-+++ b/dist/src/main.js
-@@ -7537,6 +7537,7 @@
- label: "Replay"
- }],
- providers: [providerConfig],
-+ extensions: eval_.config.extensions?.filter((extension) => extension.endsWith(":beforeAll")),
- tests: [{ vars: variables || {} }]
- }, {
- maxConcurrency: 1,
-diff --git a/dist/src/server/index.js b/dist/src/server/index.js
---- a/dist/src/server/index.js
-+++ b/dist/src/server/index.js
-@@ -64103,6 +64103,7 @@
- label: "Replay"
- }],
- providers: [providerConfig],
-+ extensions: eval_.config.extensions?.filter((extension) => extension.endsWith(":beforeAll")),
- tests: [{ vars: variables || {} }]
- }, {
- maxConcurrency: 1,
diff --git a/evals/triage-finding/pnpm-lock.yaml b/evals/triage-finding/pnpm-lock.yaml
index 7685c84e6..ac871519a 100644
--- a/evals/triage-finding/pnpm-lock.yaml
+++ b/evals/triage-finding/pnpm-lock.yaml
@@ -7,15 +7,11 @@ settings:
overrides:
'@openai/codex': 0.163.0
'@openai/codex-sdk': 0.163.0
- '@openai/[email protected]>smol-toml': 1.9.0
'@opencode-ai/sdk': 1.18.35
[email protected]>axios: 1.20.0
[email protected]>basic-ftp: 6.2.1
[email protected]>ip-address: 10.7.2
-patchedDependencies:
- [email protected]: d12be8093bf09cfc17abeaaf6f0d96b86ce2c36a42df62968fbd219deced36ee
-
importers:
.:
@@ -28,7 +24,7 @@ importers:
version: 1.18.35
promptfoo:
specifier: 0.124.1
devDependencies:
yaml:
specifier: 2.9.1
@@ -1518,10 +1514,6 @@ packages:
resolution: {integrity: sha512-bK67isD+lKq46AU8vNtjvMaT2ZqAOAmNCbxUHlFBRD4k15NWxyEjmaKtZPlgce58So4BNTjITGQOVTjL9y0ECA==}
engines: {node: '>=10.0.0'}
- resolution: {integrity: sha512-CyzaZRQKyHkB2ZInfTTl2nvT33EbDpjkLEbE8/Zck3Ll6O0qqvuGdrJ45HgtH+HykRg88ITY3AdreBGN70aBSQ==}
- engines: {node: 20.x || 22.x || 23.x || 24.x || 25.x || 26.x}
-
resolution: {integrity: sha512-QxD8cf2eVqJOOz63z6JIN9BzvVs/dlySa5HGSBH5xtR8dPteIRQnBxxKqkNTiT6jbDTF6jAfrd4oMcND9RGbQg==}
engines: {node: '>=0.6'}
@@ -1537,12 +1529,6 @@ packages:
resolution: {integrity: sha512-sXnYK/Ij80TO3lcqZVV2YgfKN5QjUWIRk/XSm2J/4bd/lPko3lvk0O4ZppH6m+6hB2/GTu+ptNwVFe1xh+QLQw==}
engines: {node: '>=4'}
- resolution: {integrity: sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==}
-
- resolution: {integrity: sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==}
-
resolution: {integrity: sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw==}
engines: {node: '>=18'}
@@ -1564,9 +1550,6 @@ packages:
resolution: {integrity: sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA==}
- resolution: {integrity: sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==}
-
resolution: {integrity: sha512-tjwM5exMg6BGRI+kNmTntNsvdZS1X8BFYS6tnJ2hdH0kVxM6/eVZ2xy+FqStSWvYmtfFMDLIxurorHwDKfDz5Q==}
engines: {node: '>=18'}
@@ -1608,9 +1591,6 @@ packages:
resolution: {integrity: sha512-TQMmc3w+5AxjpL8iIiwebF73dRDF4fBIieAqGn9RGCWaEVwQ6Fb2cGe31Yns0RRIzii5goJ1Y7xbMwo1TxMplw==}
engines: {node: '>= 20.19.0'}
- resolution: {integrity: sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==}
-
resolution: {integrity: sha512-aCj4O5wKyszjMmDT4tZj93kxyydN/K5zPWSCe6/0AV/AA1pqe5ZBIw0a2ZfPQV7lL5/yb5HsUreJ6UFAF1tEQw==}
engines: {node: '>=18'}
@@ -1755,10 +1735,6 @@ packages:
resolution: {integrity: sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg==}
- resolution: {integrity: sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==}
- engines: {node: '>=10'}
-
resolution: {integrity: sha512-WzMx3mW98SN+zn3hgemf4OzdmyNhhhKz5Ay0pUfQiMQ3e1g+xmTJWp/pKdwKVXhdSkAEGIIzqeuWrL3mV/AXbA==}
peerDependencies:
@@ -1767,10 +1743,6 @@ packages:
babel-plugin-macros:
optional: true
- resolution: {integrity: sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==}
- engines: {node: '>=4.0.0'}
-
resolution: {integrity: sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A==}
engines: {node: '>=0.10.0'}
@@ -1952,9 +1924,6 @@ packages:
resolution: {integrity: sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==}
engines: {node: '>= 0.8'}
- resolution: {integrity: sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==}
-
resolution: {integrity: sha512-iY6QdftLQ9pyiPoX082bpf/u1UewnOaJrtJIF9T0++QB34lZrj0uP+Q/bj8AlUsAxqhnkTV2BS8SBZSxOmoV5Q==}
@@ -2049,10 +2018,6 @@ packages:
resolution: {integrity: sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA==}
engines: {node: '>=18.0.0'}
- resolution: {integrity: sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==}
- engines: {node: '>=6'}
-
resolution: {integrity: sha512-hOwV7WOxXfjRpAM1DSJWZDXx3GhplwD8IfwuwvogD8i1Qnkgosw/H45s4ZnFAUHDAhPjlY9hLBvJhKmGMyY26g==}
engines: {node: '>= 16'}
@@ -2115,9 +2080,6 @@ packages:
resolution: {integrity: sha512-DLkUvGwep3poOV2wpzbHCOnSKGk1LzyXTv+aHFgN2VFl96wnp8YA9YjO2qPzg5PuL8q/SW9Pdi6WTkYOIh995w==}
engines: {node: '>=20'}
- resolution: {integrity: sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==}
-
resolution: {integrity: sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==}
engines: {node: '>= 18.0.0'}
@@ -2168,9 +2130,6 @@ packages:
resolution: {integrity: sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==}
engines: {node: '>= 0.8'}
- resolution: {integrity: sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==}
-
resolution: {integrity: sha512-oRXApq54ETRj4eMiFzGnHWGy+zo5raudjuxN0b8H7s/RU2oW0Wvsx9O0ACRN/kRq9E8Vu/ReskGB5o3ji+FzHQ==}
engines: {node: '>=12'}
@@ -2210,9 +2169,6 @@ packages:
resolution: {integrity: sha512-/5N/P4Lrh0p/mDwlDRi7Y1+P2o/OyzZI3l6Iz1Ov6XXwwm1y3RlZLuo3gVgML99djrEDtV980bBxSuOeHLk8ww==}
engines: {node: '>= 20'}
- resolution: {integrity: sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==}
-
resolution: {integrity: sha512-Wjlyrolmm8uDpm/ogGyXZXb1Z+Ca2B8NbJwqBVg0axK9GbBeoS7yGV6vjXnYdGm6X53iehEuxxbyiKp8QmN4Vw==}
engines: {node: 18 || 20 || >=22}
@@ -2327,9 +2283,6 @@ packages:
resolution: {integrity: sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==}
- resolution: {integrity: sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==}
-
resolution: {integrity: sha512-7H/2gFSIitxc0hG3nOI1glS8QLo/EHBFFLk8vEUjXY/xu0AdL8jZ9U1IzO2PUm0d2D/ofQcAifb0g6OBkt8U7w==}
engines: {node: '>= 12'}
@@ -2590,10 +2543,6 @@ packages:
resolution: {integrity: sha512-VP79XUPxV2CigYP3jWwAUFSku2aKqBH7uTAapFWCBqutsbmDo96KY5o8uh6U+/YSIn5OxJnXp73beVkpqMIGhA==}
engines: {node: '>=18'}
- resolution: {integrity: sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==}
- engines: {node: '>=10'}
-
resolution: {integrity: sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==}
engines: {node: 18 || 20 || >=22}
@@ -2601,9 +2550,6 @@ packages:
resolution: {integrity: sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==}
- resolution: {integrity: sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==}
-
resolution: {integrity: sha512-tEBHqDnIoM/1rXME1zgka9g6Q2lcoCkxHLuc7ODJ5BxbP5d4c2Z5cGgtXAku59200Cx7diuHTOYfSBD8n6mm8A==}
engines: {node: '>=16 || 14 >=14.17'}
@@ -2612,9 +2558,6 @@ packages:
resolution: {integrity: sha512-ALGF1Jt9ouehcaXaHhn6t1yGWRqGaHkPFndtFVHfZXOvkIZ/yoGaSi0AHVTafb3ZBGg4dr/bDwnaEKqCXzchMA==}
engines: {node: '>=0.10.0'}
- resolution: {integrity: sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==}
-
resolution: {integrity: sha512-ZoS07RoFqpKYQwAk59qmrx8+jJHNHU30UjlU96QktiGn1ltvDr+vCznLX5DiUBLEpMAHatHNWV1nM/74ul66kA==}
engines: {node: '>=20.19.0'}
@@ -2675,9 +2618,6 @@ packages:
resolution: {integrity: sha512-dkEJPVvun4FryqBmZ5KhDo0K9iDXAwn08tMLDinNdRBNPcYEDiWYysLcc6k3mjTMlbP9KyylvRpd4wFtwrT9rw==}
engines: {node: ^20.17.0 || >=22.9.0}
- resolution: {integrity: sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==}
-
resolution: {integrity: sha512-Ucb+lsUcGxUqu3rn8cwHjT6gJQosO63nIX/aBQXB3+IDkNbFV7PuviysO+Rzz3aKn7PZhPj3bNF4PS9gDVjYCQ==}
engines: {node: '>=0.4.10'}
@@ -2698,10 +2638,6 @@ packages:
resolution: {integrity: sha512-eonl3sLUha+S1GzTPxychyhnUzKyeQkZ7jLjKrBagJgPla13F+uQ71HgpFefyHgqrjEbCPkDArxYsjY8/+gLKA==}
engines: {node: '>= 0.4.0'}
- resolution: {integrity: sha512-KdHvFWZjEKDf0cakgFjebl371GPsISX2oZHcuyKqM7DtogIsHrqKeLTo8wBHxaXRAQlY2PsPlZmfo+9ZCxEREQ==}
- engines: {node: '>=10'}
-
resolution: {integrity: sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==}
engines: {node: '>=10.5.0'}
@@ -2986,12 +2922,6 @@ packages:
resolution: {integrity: sha512-mdb8TKt+YCRbGQdYar3AKNUPCyEiqcprScF4unYpGALF6HlBaEuO6wPuIqXXpCWkw4VclJYCKbb6lq6pH6bJeA==}
engines: {node: ^20.20.0 || >=22.22.0}
- resolution: {integrity: sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==}
- engines: {node: '>=10'}
- deprecated: No longer maintained. Please contact the author of the relevant native addon; alternatives are available.
- hasBin: true
-
resolution: {integrity: sha512-7nJ6v5lnJsXwGprnGXga4wx6d1POjvi5Qmf1ivTRxTjH4Z/9Czja/UCMLVmB9N93GeWOU93XaFaEt6jbuoagNw==}
@@ -3064,9 +2994,6 @@ packages:
resolution: {integrity: sha512-JZd3gMVBAVQkSs6HdNZo9Sdo0LNcQeMNP3CozBJb3JYC/QUYZTnKxP+f8oWRX4rHP5EurWxqAHTSwUCjlNKa1w==}
- resolution: {integrity: sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==}
-
resolution: {integrity: sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==}
engines: {node: '>=6'}
@@ -3141,10 +3068,6 @@ packages:
resolution: {integrity: sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==}
engines: {node: '>= 0.10'}
- resolution: {integrity: sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==}
- hasBin: true
-
resolution: {integrity: sha512-zFcBdzunLCGBmLRT4Q3mLPJnhKPAXXfGAZ83feODlEqX2aRUbgF1h/n2oY0UF7I8tVwnJNRas6fTRD5veDUs+g==}
engines: {node: '>=18'}
@@ -3286,12 +3209,6 @@ packages:
resolution: {integrity: sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==}
engines: {node: '>=14'}
- resolution: {integrity: sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==}
-
- resolution: {integrity: sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==}
-
resolution: {integrity: sha512-l0sIsv9VrPwovKELFDBOj/lhmIcbMvxBk2B+hRhsMM4/VL3gKtuZvYnBNixVo+N/0HnqQx2cBdLvxnAbygHpqA==}
@@ -3370,10 +3287,6 @@ packages:
resolution: {integrity: sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w==}
engines: {node: '>=12'}
- resolution: {integrity: sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==}
- engines: {node: '>=0.10.0'}
-
resolution: {integrity: sha512-rDG3Ah4TV0k1hWvLSzkZtMmLN9+eS+h3knq4MP6A42Y3Yh5qGNnOUs1jJkoSr8FG5dsL28c7KgkIBzSEykqtuw==}
@@ -3385,13 +3298,6 @@ packages:
resolution: {integrity: sha512-yUT0ukFkFEt4nb+NY+n2ag51aS/u9UHXoZw+A4jgD77/jzZsBoSDHuqysrVCBC4CYR4TYvUJq54ONpXgDBH8tA==}
engines: {node: '>=0.2.6'}
- resolution: {integrity: sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==}
-
- resolution: {integrity: sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==}
- engines: {node: '>=6'}
-
resolution: {integrity: sha512-anOjtXr8OT5w4vc/2mP4AYTCE0GWc/21icGmaHtBHnI7pN7o01a/oqG9m06/rGzoAsDm/WNzggBpqptuCmRlZQ==}
engines: {node: '>=18.20'}
@@ -3440,9 +3346,6 @@ packages:
engines: {node: '>=18.0.0'}
hasBin: true
- resolution: {integrity: sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==}
-
resolution: {integrity: sha512-Ne+eE4r0/iWnpAxD852z3A+N0Bt5RN//NjJwRd2VFHEmrywxf5vsZlh4R6lixl6B+wz/8d+maTSAkN1FIkI3LQ==}
engines: {node: '>=10'}
@@ -5319,12 +5222,6 @@ snapshots:
- dependencies:
- bindings: 1.5.0
- prebuild-install: 7.1.3
- optional: true
-
optional: true
@@ -5336,18 +5233,6 @@ snapshots:
dependencies:
editions: 6.22.0
- dependencies:
- file-uri-to-path: 1.0.0
- optional: true
-
- dependencies:
- buffer: 5.7.1
- inherits: 2.0.4
- readable-stream: 3.6.2
- optional: true
-
dependencies:
bytes: 3.1.2
@@ -5381,12 +5266,6 @@ snapshots:
optional: true
- dependencies:
- base64-js: 1.5.1
- ieee754: 1.2.1
- optional: true
-
dependencies:
run-applescript: 7.1.0
@@ -5425,9 +5304,6 @@ snapshots:
dependencies:
readdirp: 5.1.1
- optional: true
-
dependencies:
restore-cursor: 5.1.0
@@ -5551,16 +5427,8 @@ snapshots:
- dependencies:
- mimic-response: 3.1.0
- optional: true
-
- optional: true
-
optional: true
@@ -5618,11 +5486,10 @@ snapshots:
optional: true
optionalDependencies:
'@libsql/client': 0.18.0
'@opentelemetry/api': 1.9.1
- better-sqlite3: 12.10.0
pg: 8.23.0
@@ -5648,11 +5515,6 @@ snapshots:
- dependencies:
- once: 1.4.0
- optional: true
-
dependencies:
'@socket.io/component-emitter': 3.1.2
@@ -5775,9 +5637,6 @@ snapshots:
eventsource-parser: 3.1.1
optional: true
- optional: true
-
dependencies:
debug: 4.4.3
@@ -5878,9 +5737,6 @@ snapshots:
- supports-color
optional: true
- optional: true
-
dependencies:
debug: 4.4.3
@@ -5932,9 +5788,6 @@ snapshots:
- optional: true
-
dependencies:
graceful-fs: 4.2.11
@@ -5994,9 +5847,6 @@ snapshots:
transitivePeerDependencies:
- supports-color
- optional: true
-
dependencies:
minimatch: 10.2.6
@@ -6185,9 +6035,6 @@ snapshots:
- optional: true
-
@@ -6462,9 +6309,6 @@ snapshots:
- optional: true
-
dependencies:
brace-expansion: 5.0.12
@@ -6474,9 +6318,6 @@ snapshots:
brace-expansion: 1.1.21
optional: true
- optional: true
-
@@ -6485,9 +6326,6 @@ snapshots:
is-extendable: 0.1.1
optional: true
- optional: true
-
dependencies:
'@types/whatwg-url': 13.0.0
@@ -6542,9 +6380,6 @@ snapshots:
- optional: true
-
dependencies:
afinn-165: 2.0.2
@@ -6584,11 +6419,6 @@ snapshots:
- dependencies:
- semver: 7.8.5
- optional: true
-
@@ -6875,25 +6705,9 @@ snapshots:
dependencies:
'@posthog/core': 1.23.1
- dependencies:
- detect-libc: 2.1.2
- expand-template: 2.0.3
- github-from-package: 0.0.0
- minimist: 1.2.8
- mkdirp-classic: 0.5.3
- napi-build-utils: 2.0.0
- node-abi: 3.92.0
- pump: 3.0.4
- rc: 1.2.8
- simple-get: 4.0.1
- tar-fs: 2.1.4
- tunnel-agent: 0.6.0
- optional: true
-
dependencies:
'@anthropic-ai/sdk': 0.128.0([email protected])
'@apidevtools/json-schema-ref-parser': 16.0.3(@types/[email protected])
@@ -6929,7 +6743,7 @@ snapshots:
csv-parse: 7.0.3
csv-stringify: 6.9.0
dedent: 1.7.2
- drizzle-orm: 0.45.4(@libsql/[email protected])(@opentelemetry/[email protected])([email protected])([email protected])
+ drizzle-orm: 0.45.4(@libsql/[email protected])(@opentelemetry/[email protected])([email protected])
express: 5.2.1
exsolve: 1.1.1
fast-deep-equal: 3.1.3
@@ -7103,12 +6917,6 @@ snapshots:
punycode: 2.3.1
optional: true
- dependencies:
- end-of-stream: 1.4.5
- once: 1.4.0
- optional: true
-
optional: true
@@ -7179,14 +6987,6 @@ snapshots:
iconv-lite: 0.7.3
unpipe: 1.0.0
- dependencies:
- deep-extend: 0.6.0
- ini: 1.3.8
- minimist: 1.2.8
- strip-json-comments: 2.0.1
- optional: true
-
dependencies:
fflate: 0.8.3
@@ -7386,16 +7186,6 @@ snapshots:
- optional: true
-
- dependencies:
- decompress-response: 6.0.0
- once: 1.4.0
- simple-concat: 1.0.1
- optional: true
-
dependencies:
'@kwsites/file-exists': 1.1.1
@@ -7509,9 +7299,6 @@ snapshots:
dependencies:
ansi-regex: 6.3.0
- optional: true
-
dependencies:
anynum: 1.0.1
@@ -7524,23 +7311,6 @@ snapshots:
optional: true
- dependencies:
- chownr: 1.1.4
- mkdirp-classic: 0.5.3
- pump: 3.0.4
- tar-stream: 2.2.0
- optional: true
-
- dependencies:
- bl: 4.1.0
- end-of-stream: 1.4.5
- fs-constants: 1.0.0
- inherits: 2.0.4
- readable-stream: 3.6.2
- optional: true
-
@@ -7587,11 +7357,6 @@ snapshots:
optionalDependencies:
fsevents: 2.3.3
- dependencies:
- safe-buffer: 5.2.1
- optional: true
-
optional: true
diff --git a/evals/triage-finding/pnpm-workspace.yaml b/evals/triage-finding/pnpm-workspace.yaml
index ab8a11e7f..a94b0f041 100644
--- a/evals/triage-finding/pnpm-workspace.yaml
+++ b/evals/triage-finding/pnpm-workspace.yaml
@@ -9,7 +9,6 @@ trustLockfile: true
overrides:
"@openai/codex": "$@openai/codex-sdk"
"@openai/codex-sdk": "$@openai/codex-sdk"
- "@openai/[email protected]>smol-toml": 1.9.0
"@opencode-ai/sdk": "$@opencode-ai/sdk"
"[email protected]>axios": 1.20.0
"[email protected]>basic-ftp": 6.2.1
@@ -18,11 +17,7 @@ overrides:
allowBuilds:
"@playwright/browser-chromium": false
"@swc/core": false
- better-sqlite3: true
esbuild: false
onnxruntime-node: false
protobufjs: false
sharp: false
-
-patchedDependencies:
diff --git a/evals/triage-finding/promptfooconfig.calibration-smoke.yaml b/evals/triage-finding/promptfooconfig.calibration-smoke.yaml
index a9588c191..585e5e267 100644
--- a/evals/triage-finding/promptfooconfig.calibration-smoke.yaml
+++ b/evals/triage-finding/promptfooconfig.calibration-smoke.yaml
@@ -1,4 +1,8 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
$ref: ./promptfooconfig.calibration.yaml
description: "Codex Security triage-finding OSS calibration smoke eval"
-tests: file://./artifacts/calibration-smoke.yaml
+tests:
+ path: file://./scripts/generate-calibration-tests.mts:generateTests
+ config:
+ caseId: oss-dompurify-ghsa-v8jm-5vwx-cfxm
+ variantId: vulnerable
diff --git a/evals/triage-finding/promptfooconfig.calibration.yaml b/evals/triage-finding/promptfooconfig.calibration.yaml
index 230111a88..ed343e509 100644
--- a/evals/triage-finding/promptfooconfig.calibration.yaml
+++ b/evals/triage-finding/promptfooconfig.calibration.yaml
@@ -4,34 +4,8 @@ description: "Codex Security triage-finding OSS calibration eval"
prompts:
- file://./prompts/triage-request.txt
-extensions:
- - file://{{env.TRIAGE_PROVIDER_PATH}}:beforeAll
-
providers:
- - id: openai:codex-sdk:gpt-5.5
- label: "Codex SDK triage-finding calibration"
- config:
- skip_git_repo_check: true
- approval_policy: never
- cli_env:
- HTTP_PROXY: "{{env.HTTP_PROXY}}"
- HTTPS_PROXY: "{{env.HTTPS_PROXY}}"
- ALL_PROXY: "{{env.ALL_PROXY}}"
- NO_PROXY: "{{env.NO_PROXY}}"
- additional_directories:
- - "{% if calibration_repo_root %}{{calibration_repo_root}}{% else %}{{env.TRIAGE_CALIBRATION_ROOT}}{% endif %}/{{calibration_repo}}"
- cli_config:
- default_permissions: triage_runtime_only
- permissions:
- triage_runtime_only:
- filesystem:
- ":minimal": read
- ":workspace_roots": read
- network:
- enabled: false
- features:
- memories: false
- plugins: false
+ - $ref: ./providers/calibration.yaml
evaluateOptions:
maxConcurrency: 1
@@ -49,4 +23,4 @@ defaultTest:
- type: javascript
value: file://./assertions/calibration-evidence.mts
-tests: file://./tests/calibration-oss.yaml
+tests: file://./scripts/generate-calibration-tests.mts:generateTests
diff --git a/evals/triage-finding/promptfooconfig.sastbench.yaml b/evals/triage-finding/promptfooconfig.sastbench.yaml
new file mode 100644
index 000000000..1bb5507fa
--- /dev/null
+++ b/evals/triage-finding/promptfooconfig.sastbench.yaml
@@ -0,0 +1,86 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+# Shared by the entry points in sastbench/, which retain their file:// base path.
+description: "Codex Security triage-finding SastBench v0.1 eval"
+tags:
+ suite: sastbench-v0.1
+
+prompts:
+ - file://./prompts/sastbench-triage-request.txt
+
+providers:
+ - $ref: ./providers/sastbench.yaml
+
+extensions:
+ - file://./assertions/sastbench-metrics.mts:afterEach
+
+defaultTest:
+ assert:
+ - type: contains-json
+ value: file://../schemas/triage-result-v0.schema.json
+ metric: schema_valid
+ - type: javascript
+ value: file://./assertions/sastbench-verdict.mts
+ metric: strict_case_correct
+
+# The afterEach extension emits additive counters for every row. Representative
+# sample rows scale those counters by their population weight.
+# Promptfoo recalculates these aggregate metrics as rows complete.
+derivedMetrics:
+ - name: strict_precision
+ value: "strict_tp / max(strict_tp + strict_fp, 1)"
+ - name: strict_recall
+ value: "strict_tp / max(strict_tp + strict_fn, 1)"
+ - name: strict_f1
+ value: "2 * strict_tp / max(2 * strict_tp + strict_fp + strict_fn, 1)"
+ - name: strict_f2
+ value: "5 * strict_tp / max(5 * strict_tp + strict_fp + 4 * strict_fn, 1)"
+ - name: strict_accuracy
+ value: "(strict_tp + strict_tn) / max(positive_cases + negative_cases, 1)"
+ - name: strict_mcc
+ value: >-
+ (strict_tp * strict_tn - strict_fp * strict_fn) /
+ max(sqrt(
+ (strict_tp + strict_fp) *
+ (strict_tp + strict_fn) *
+ (strict_tn + strict_fp) *
+ (strict_tn + strict_fn)
+ ), 1)
+ - name: decided_precision
+ value: "decided_tp / max(decided_tp + decided_fp, 1)"
+ - name: decided_recall
+ value: "decided_tp / max(decided_tp + decided_fn, 1)"
+ - name: decided_f1
+ value: "2 * decided_tp / max(2 * decided_tp + decided_fp + decided_fn, 1)"
+ - name: decided_f2
+ value: "5 * decided_tp / max(5 * decided_tp + decided_fp + 4 * decided_fn, 1)"
+ - name: decided_accuracy
+ value: "(decided_tp + decided_tn) / max(decided_cases, 1)"
+ - name: decided_mcc
+ value: >-
+ (decided_tp * decided_tn - decided_fp * decided_fn) /
+ max(sqrt(
+ (decided_tp + decided_fp) *
+ (decided_tp + decided_fn) *
+ (decided_tn + decided_fp) *
+ (decided_tn + decided_fn)
+ ), 1)
+ - name: decided_coverage
+ value: "decided_cases / max(positive_cases + negative_cases, 1)"
+ - name: true_positive_retention
+ value: "retained_positive / max(positive_cases, 1)"
+ - name: unsafe_closure_rate
+ value: "unsafe_closure / max(positive_cases, 1)"
+ - name: false_alert_auto_closure_rate
+ value: "false_alert_auto_closure / max(negative_cases, 1)"
+ - name: false_alert_escalation_rate
+ value: "false_alert_escalation / max(negative_cases, 1)"
+ - name: confirmed_precision
+ value: "confirmed_true / max(confirmed_total, 1)"
+ - name: abstention_rate
+ value: "abstention / max(positive_cases + negative_cases, 1)"
+ - name: remaining_analyst_workload
+ value: "remaining_analyst_work / max(positive_cases + negative_cases, 1)"
+ - name: execution_or_parse_error_rate
+ value: "execution_or_parse_error / max(positive_cases + negative_cases, 1)"
+
+tests: file://./scripts/sastbench-lib.mts:generateTests
diff --git a/evals/triage-finding/promptfooconfig.yaml b/evals/triage-finding/promptfooconfig.yaml
index 70d4ac7f1..4e9890a3f 100644
--- a/evals/triage-finding/promptfooconfig.yaml
+++ b/evals/triage-finding/promptfooconfig.yaml
@@ -4,32 +4,8 @@ description: "Codex Security triage-finding input-type I/O eval"
prompts:
- file://./prompts/triage-request.txt
-extensions:
- - file://{{env.TRIAGE_PROVIDER_PATH}}:beforeAll
-
providers:
- - id: openai:codex-sdk:gpt-5.5
- label: "Codex SDK triage-finding"
- config:
- skip_git_repo_check: true
- approval_policy: never
- cli_env:
- HTTP_PROXY: "{{env.HTTP_PROXY}}"
- HTTPS_PROXY: "{{env.HTTPS_PROXY}}"
- ALL_PROXY: "{{env.ALL_PROXY}}"
- NO_PROXY: "{{env.NO_PROXY}}"
- cli_config:
- default_permissions: triage_runtime_only
- permissions:
- triage_runtime_only:
- filesystem:
- ":minimal": read
- ":workspace_roots": read
- network:
- enabled: false
- features:
- memories: false
- plugins: false
+ - $ref: ./providers/triage.yaml
evaluateOptions:
maxConcurrency: 1
diff --git a/evals/triage-finding/providers/calibration.yaml b/evals/triage-finding/providers/calibration.yaml
new file mode 100644
index 000000000..7bed52cfd
--- /dev/null
+++ b/evals/triage-finding/providers/calibration.yaml
@@ -0,0 +1,24 @@
+id: openai:codex-sdk:gpt-5.5
+label: "Codex SDK triage-finding calibration"
+config:
+ skip_git_repo_check: true
+ approval_policy: never
+ cli_env:
+ HTTP_PROXY: "{{env.HTTP_PROXY}}"
+ HTTPS_PROXY: "{{env.HTTPS_PROXY}}"
+ ALL_PROXY: "{{env.ALL_PROXY}}"
+ NO_PROXY: "{{env.NO_PROXY}}"
+ additional_directories:
+ - "{% if calibration_repo_root %}{{calibration_repo_root}}{% else %}{{env.TRIAGE_CALIBRATION_ROOT}}{% endif %}/{{calibration_repo}}"
+ cli_config:
+ default_permissions: triage_runtime_only
+ permissions:
+ triage_runtime_only:
+ filesystem:
+ ":minimal": read
+ ":workspace_roots": read
+ network:
+ enabled: false
+ features:
+ memories: false
+ plugins: false
diff --git a/evals/triage-finding/providers/sastbench.yaml b/evals/triage-finding/providers/sastbench.yaml
new file mode 100644
index 000000000..b548cf834
--- /dev/null
+++ b/evals/triage-finding/providers/sastbench.yaml
@@ -0,0 +1,32 @@
+id: openai:codex-sdk:gpt-5.5
+label: "Codex SDK triage-finding SastBench (gpt-5.5)"
+config:
+ # The provider binds working_dir to the current staged skill runtime.
+ # The selected target worktree and its label-free Git cache are the only
+ # additional readable workspace roots.
+ additional_directories:
+ - "{{env.SASTBENCH_GIT_CACHE_ROOT}}"
+ - "{{target_repo}}"
+ skip_git_repo_check: true
+ approval_policy: never
+ # Pin a value supported by the eval's Codex SDK instead of inheriting a
+ # potentially newer user-level setting from ~/.codex/config.toml.
+ model_reasoning_effort: xhigh
+ # Do not set the legacy sandbox option here: it grants full-disk
+ # read access and overrides this named, deny-by-default profile.
+ cli_config:
+ default_permissions: sastbench_runtime_only
+ permissions:
+ sastbench_runtime_only:
+ filesystem:
+ # :minimal keeps Codex's own runtime usable. :workspace_roots
+ # resolves to working_dir plus target Git roots; all else is denied.
+ ":minimal": read
+ ":workspace_roots": read
+ network:
+ enabled: false
+ # The source skill is loaded explicitly from the staged runtime.
+ # Global plugin and memory state would add machine-specific context.
+ features:
+ memories: false
+ plugins: false
diff --git a/evals/triage-finding/providers/triage.yaml b/evals/triage-finding/providers/triage.yaml
new file mode 100644
index 000000000..4ae66d3f9
--- /dev/null
+++ b/evals/triage-finding/providers/triage.yaml
@@ -0,0 +1,22 @@
+id: openai:codex-sdk:gpt-5.5
+label: "Codex SDK triage-finding"
+config:
+ skip_git_repo_check: true
+ approval_policy: never
+ cli_env:
+ HTTP_PROXY: "{{env.HTTP_PROXY}}"
+ HTTPS_PROXY: "{{env.HTTPS_PROXY}}"
+ ALL_PROXY: "{{env.ALL_PROXY}}"
+ NO_PROXY: "{{env.NO_PROXY}}"
+ cli_config:
+ default_permissions: triage_runtime_only
+ permissions:
+ triage_runtime_only:
+ filesystem:
+ ":minimal": read
+ ":workspace_roots": read
+ network:
+ enabled: false
+ features:
+ memories: false
+ plugins: false
diff --git a/evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yaml b/evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yaml
index aa4c5e25d..fe4eb1d10 100644
--- a/evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yaml
+++ b/evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yaml
@@ -1,7 +1,6 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
-# Promptfoo 0.121.15 resolves external $ref paths from the eval package's
-# working directory rather than from this file's directory.
-$ref: ./sastbench/promptfooconfig.sastbench.yaml
+# Promptfoo resolves this $ref from the eval package's working directory.
+$ref: ./promptfooconfig.sastbench.yaml
description: "Codex Security triage-finding representative SastBench v0.1 sample"
tags:
diff --git a/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml b/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml
index 31dbda0f3..804d8a62b 100644
--- a/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml
+++ b/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml
@@ -1,117 +1,3 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
-description: "Codex Security triage-finding SastBench v0.1 eval"
-tags:
- suite: sastbench-v0.1
-
-prompts:
- - file://./prompts/sastbench-triage-request.txt
-
-providers:
- - id: openai:codex-sdk:gpt-5.5
- label: "Codex SDK triage-finding SastBench (gpt-5.5)"
- config:
- # The provider binds working_dir to the current staged skill runtime.
- # The selected target worktree and its label-free Git cache are the only
- # additional readable workspace roots.
- additional_directories:
- - "{{env.SASTBENCH_GIT_CACHE_ROOT}}"
- - "{{target_repo}}"
- skip_git_repo_check: true
- approval_policy: never
- # Pin a value supported by the eval's Codex SDK instead of inheriting a
- # potentially newer user-level setting from ~/.codex/config.toml.
- model_reasoning_effort: xhigh
- # Do not set the legacy sandbox option here: it grants full-disk
- # read access and overrides this named, deny-by-default profile.
- cli_config:
- default_permissions: sastbench_runtime_only
- permissions:
- sastbench_runtime_only:
- filesystem:
- # :minimal keeps Codex's own runtime usable. :workspace_roots
- # resolves to working_dir plus target Git roots; all else is denied.
- ":minimal": read
- ":workspace_roots": read
- network:
- enabled: false
- # The source skill is loaded explicitly from the staged runtime.
- # Global plugin and memory state would add machine-specific context.
- features:
- memories: false
- plugins: false
-
-extensions:
- - file://{{env.TRIAGE_PROVIDER_PATH}}:beforeAll
- - file://./assertions/sastbench-metrics.mts:afterEach
-
-defaultTest:
- assert:
- - type: contains-json
- value: file://../schemas/triage-result-v0.schema.json
- metric: schema_valid
- - type: javascript
- value: file://./assertions/sastbench-verdict.mts
- metric: strict_case_correct
-
-# The afterEach extension emits additive counters for every row. Representative
-# sample rows scale those counters by their population weight.
-# Promptfoo recalculates these aggregate metrics as rows complete.
-derivedMetrics:
- - name: strict_precision
- value: "strict_tp / max(strict_tp + strict_fp, 1)"
- - name: strict_recall
- value: "strict_tp / max(strict_tp + strict_fn, 1)"
- - name: strict_f1
- value: "2 * strict_tp / max(2 * strict_tp + strict_fp + strict_fn, 1)"
- - name: strict_f2
- value: "5 * strict_tp / max(5 * strict_tp + strict_fp + 4 * strict_fn, 1)"
- - name: strict_accuracy
- value: "(strict_tp + strict_tn) / max(positive_cases + negative_cases, 1)"
- - name: strict_mcc
- value: >-
- (strict_tp * strict_tn - strict_fp * strict_fn) /
- max(sqrt(
- (strict_tp + strict_fp) *
- (strict_tp + strict_fn) *
- (strict_tn + strict_fp) *
- (strict_tn + strict_fn)
- ), 1)
- - name: decided_precision
- value: "decided_tp / max(decided_tp + decided_fp, 1)"
- - name: decided_recall
- value: "decided_tp / max(decided_tp + decided_fn, 1)"
- - name: decided_f1
- value: "2 * decided_tp / max(2 * decided_tp + decided_fp + decided_fn, 1)"
- - name: decided_f2
- value: "5 * decided_tp / max(5 * decided_tp + decided_fp + 4 * decided_fn, 1)"
- - name: decided_accuracy
- value: "(decided_tp + decided_tn) / max(decided_cases, 1)"
- - name: decided_mcc
- value: >-
- (decided_tp * decided_tn - decided_fp * decided_fn) /
- max(sqrt(
- (decided_tp + decided_fp) *
- (decided_tp + decided_fn) *
- (decided_tn + decided_fp) *
- (decided_tn + decided_fn)
- ), 1)
- - name: decided_coverage
- value: "decided_cases / max(positive_cases + negative_cases, 1)"
- - name: true_positive_retention
- value: "retained_positive / max(positive_cases, 1)"
- - name: unsafe_closure_rate
- value: "unsafe_closure / max(positive_cases, 1)"
- - name: false_alert_auto_closure_rate
- value: "false_alert_auto_closure / max(negative_cases, 1)"
- - name: false_alert_escalation_rate
- value: "false_alert_escalation / max(negative_cases, 1)"
- - name: confirmed_precision
- value: "confirmed_true / max(confirmed_total, 1)"
- - name: abstention_rate
- value: "abstention / max(positive_cases + negative_cases, 1)"
- - name: remaining_analyst_workload
- value: "remaining_analyst_work / max(positive_cases + negative_cases, 1)"
- - name: execution_or_parse_error_rate
- value: "execution_or_parse_error / max(positive_cases + negative_cases, 1)"
-
-tests: file://./scripts/sastbench-lib.mts:generateTests
+# Promptfoo resolves this $ref from the eval package's working directory.
+$ref: ./promptfooconfig.sastbench.yaml
diff --git a/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts b/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts
index 14bf0fe1d..40660de8e 100644
--- a/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts
+++ b/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts
@@ -5,53 +5,55 @@ import os from "node:os";
import { pathToFileURL } from "node:url";
import fs from "node:fs";
import path from "node:path";
+import { parse } from "yaml";
const sastbenchRoot = path.resolve(import.meta.dirname, "..");
const evalRoot = path.resolve(sastbenchRoot, "..");
-const config = fs.readFileSync(
- path.join(sastbenchRoot, "promptfooconfig.sastbench.yaml"),
- "utf8",
+const entryConfig = parse(
+ fs.readFileSync(
+ path.join(sastbenchRoot, "promptfooconfig.sastbench.yaml"),
+ "utf8",
+ ),
+);
+assert.equal(entryConfig.$ref, "./promptfooconfig.sastbench.yaml");
+const config = parse(
+ fs.readFileSync(path.resolve(evalRoot, entryConfig.$ref), "utf8"),
);
-const sampleConfig = fs.readFileSync(
- path.join(sastbenchRoot, "promptfooconfig.sastbench-sample.yaml"),
- "utf8",
+const sampleConfig = parse(
+ fs.readFileSync(
+ path.join(sastbenchRoot, "promptfooconfig.sastbench-sample.yaml"),
+ "utf8",
+ ),
);
const packageJson = JSON.parse(
fs.readFileSync(path.join(evalRoot, "package.json"), "utf8"),
);
-assert.doesNotMatch(config, /working_dir:/);
-assert.match(config, /skip_git_repo_check:\s+true/);
-assert.doesNotMatch(
- config,
- /working_dir:\s+\.\.\/\.\.\/\.\.\/\.\.\/\.\.\/\.\./,
-);
-assert.match(
- config,
- /additional_directories:\n\s+- "\{\{env\.SASTBENCH_GIT_CACHE_ROOT\}\}"\n\s+- "\{\{target_repo\}\}"/,
-);
-assert.doesNotMatch(config, /sandbox_mode:/);
-assert.doesNotMatch(config, /network_access_enabled:/);
-assert.match(config, /default_permissions:\s+sastbench_runtime_only/);
-assert.match(config, /sastbench_runtime_only:\n\s+filesystem:/);
-assert.match(config, /":minimal":\s+read/);
-assert.match(config, /":workspace_roots":\s+read/);
-assert.match(config, /network:\n\s+enabled:\s+false/);
-assert.match(config, /id:\s+openai:codex-sdk:gpt-5\.5/);
-assert.match(
- config,
- /label:\s+"Codex SDK triage-finding SastBench \(gpt-5\.5\)"/,
+const providerPath = path.resolve(evalRoot, config.providers[0].$ref);
+const provider = parse(fs.readFileSync(providerPath, "utf8"));
+assert.match(provider.id, /^openai:codex-sdk:/);
+assert.deepEqual(provider.config.additional_directories, [
+ "{{env.SASTBENCH_GIT_CACHE_ROOT}}",
+ "{{target_repo}}",
+]);
+assert.equal(provider.config.cli_env, undefined);
+assert.equal(config.evaluateOptions?.maxConcurrency, undefined);
+assert.equal(config.tests, "file://./scripts/sastbench-lib.mts:generateTests");
+assert.deepEqual(
+ config.defaultTest.assert.map(
+ (assertion: { metric: string }) => assertion.metric,
+ ),
+ ["schema_valid", "strict_case_correct"],
);
-assert.doesNotMatch(config, /gpt-5\.6/);
-assert.match(
- config,
- /tests:\s+file:\/\/\.\/scripts\/sastbench-lib\.mts:generateTests/,
+assert.ok(
+ config.extensions.includes(
+ "file://./assertions/sastbench-metrics.mts:afterEach",
+ ),
);
-assert.match(config, /metric:\s+schema_valid/);
-assert.match(config, /metric:\s+strict_case_correct/);
-assert.match(
- config,
- /extensions:\n\s+- file:\/\/\{\{env\.TRIAGE_PROVIDER_PATH\}\}:beforeAll\n\s+- file:\/\/\.\/assertions\/sastbench-metrics\.mts:afterEach/,
+const derivedMetrics = new Map<string, string>(
+ config.derivedMetrics.map(
+ ({ name, value }: { name: string; value: string }) => [name, value],
+ ),
);
for (const metric of [
"strict_precision",
@@ -69,33 +71,25 @@ for (const metric of [
"remaining_analyst_workload",
"execution_or_parse_error_rate",
]) {
- assert.match(config, new RegExp(`name:\\s+${metric}`));
+ assert.ok(derivedMetrics.has(metric), `Missing aggregate metric: ${metric}`);
}
-assert.doesNotMatch(config, /HTTP_PROXY|HTTPS_PROXY|ALL_PROXY|NO_PROXY/);
-assert.doesNotMatch(config, /maxConcurrency/);
-assert.doesNotMatch(config, /__count/);
-assert.match(
- config,
- /strict_accuracy\n\s+value:\s+"\(strict_tp \+ strict_tn\) \/ max\(positive_cases \+ negative_cases, 1\)"/,
-);
-assert.match(
- sampleConfig,
- /^\$ref:\s+\.\/sastbench\/promptfooconfig\.sastbench\.yaml/m,
+assert.equal(
+ derivedMetrics.get("strict_accuracy"),
+ "(strict_tp + strict_tn) / max(positive_cases + negative_cases, 1)",
);
-assert.match(
- sampleConfig,
- /tests:\s+file:\/\/\.\/scripts\/sastbench-lib\.mts:generateSampleTests/,
+assert.equal(sampleConfig.$ref, entryConfig.$ref);
+assert.equal(
+ sampleConfig.tests,
+ "file://./scripts/sastbench-lib.mts:generateSampleTests",
);
-assert.doesNotMatch(sampleConfig, /providers:/);
-assert.doesNotMatch(sampleConfig, /derivedMetrics:/);
+assert.equal(sampleConfig.providers, undefined);
+assert.equal(sampleConfig.derivedMetrics, undefined);
assert.match(packageJson.scripts["eval:sastbench"], /run-promptfoo\.mts eval/);
assert.match(packageJson.scripts["eval:sastbench"], /--no-cache/);
assert.match(packageJson.scripts["eval:sastbench"], /--no-share/);
assert.doesNotMatch(packageJson.scripts["eval:sastbench"], /--filter-range/);
assert.doesNotMatch(packageJson.scripts["eval:sastbench"], /--max-concurrency/);
-assert.equal("score:sastbench" in packageJson.scripts, false);
-assert.equal("report:sastbench" in packageJson.scripts, false);
assert.doesNotMatch(
packageJson.scripts["eval:sastbench"],
/PROMPTFOO_FAILED_TEST_EXIT_CODE/,
@@ -114,22 +108,6 @@ assert.match(
packageJson.scripts["eval:sastbench:sample"],
/--max-concurrency 32/,
);
-assert.equal("sastbench:generate" in packageJson.scripts, false);
-assert.equal(
- fs.existsSync(path.join(import.meta.dirname, "run-sastbench-promptfoo.mts")),
- false,
-);
-assert.equal(
- fs.existsSync(
- path.join(
- sastbenchRoot,
- "docs",
- "2026-06-22-sastbench-triage-finding-eval.md",
- ),
- ),
- false,
-);
-
console.log("sastbench native Promptfoo harness tests passed");
const dispatchRoot = fs.mkdtempSync(
diff --git a/evals/triage-finding/scripts/generate-calibration-tests.mts b/evals/triage-finding/scripts/generate-calibration-tests.mts
index 33bb8af51..f74f90acd 100644
--- a/evals/triage-finding/scripts/generate-calibration-tests.mts
+++ b/evals/triage-finding/scripts/generate-calibration-tests.mts
@@ -4,6 +4,7 @@ import type { CalibrationCase, CalibrationVariant } from "../types.ts";
import fs from "node:fs";
import { hash } from "node:crypto";
import path from "node:path";
+import { stringify } from "yaml";
export const DEFAULT_DATASET = path.join(
import.meta.dirname,
@@ -48,16 +49,6 @@ function parseArgs(argv: string[]) {
return args;
}
-function quote(value: unknown) {
- return JSON.stringify(String(value));
-}
-
-function indentedBlock(value: unknown) {
- return ` ${String(value)
- .replace(/\r\n?/g, "\n")
- .replace(/\n/g, "\n ")}`;
-}
-
export function variantCaseId(
testCase: Pick<CalibrationCase, "case_id">,
variant: CalibrationVariant,
@@ -105,45 +96,42 @@ export function findingInput(
return lines.join("\n");
}
-function testYaml(
+function buildPromptfooTest(
testCase: CalibrationCase,
variant: CalibrationVariant,
repoRoot: string,
) {
const generatedCaseId = variantCaseId(testCase, variant);
const terms = evidenceTerms(testCase);
- const lines = [
- `- description: ${quote(`calibration ${variant.variant_id}: ${testCase.case_id}`)}`,
- " metadata:",
- ` case_id: ${generatedCaseId}`,
- " suite: calibration-oss",
- ` calibration_case_id: ${testCase.case_id}`,
- ` calibration_variant: ${variant.variant_id}`,
- ` repo: ${quote(testCase.repo.name)}`,
- ` expected_binary_label: ${variant.expected_binary_label}`,
- " vars:",
- ` case_id: ${generatedCaseId}`,
- ` calibration_repo: ${generatedCaseId}`,
- ` calibration_repo_root: ${quote(repoRoot === DEFAULT_REPO_ROOT ? "" : path.resolve(repoRoot))}`,
- ` source_type_under_test: ${testCase.source_type}`,
- ` expected_ids: ${variantCaseId(testCase, variant)}`,
- ` expected_source_types: ${testCase.source_type}`,
- ` expected_verdicts: ${variant.expected_verdict}`,
- ` expected_binary_label: ${variant.expected_binary_label}`,
- ` expected_evidence_terms: ${quote(terms.join(", "))}`,
- " finding_input: |-",
- indentedBlock(findingInput(testCase, variant)),
- " eval_instructions: |-",
- indentedBlock(
- `This is an automated OSS calibration eval. Do not ask follow-up questions.
+ return {
+ description: `calibration ${variant.variant_id}: ${testCase.case_id}`,
+ metadata: {
+ case_id: generatedCaseId,
+ suite: "calibration-oss",
+ calibration_case_id: testCase.case_id,
+ calibration_variant: variant.variant_id,
+ repo: testCase.repo.name,
+ expected_binary_label: variant.expected_binary_label,
+ },
+ vars: {
+ case_id: generatedCaseId,
+ calibration_repo: generatedCaseId,
+ calibration_repo_root:
+ repoRoot === DEFAULT_REPO_ROOT ? "" : path.resolve(repoRoot),
+ source_type_under_test: testCase.source_type,
+ expected_ids: generatedCaseId,
+ expected_source_types: testCase.source_type,
+ expected_verdicts: variant.expected_verdict,
+ expected_binary_label: variant.expected_binary_label,
+ expected_evidence_terms: terms.join(", "),
+ finding_input: findingInput(testCase, variant),
+ eval_instructions: `This is an automated OSS calibration eval. Do not ask follow-up questions.
Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.`,
- ),
- ];
-
- return lines.join("\n");
+ },
+ };
}
export function selectedVariants(
@@ -170,19 +158,34 @@ export function selectedVariants(
return variants;
}
+export function generateTests(
+ options: {
+ datasetPath?: string;
+ repoRoot?: string;
+ caseId?: string | null;
+ variantId?: string | null;
+ } = {},
+) {
+ const dataset = JSON.parse(
+ fs.readFileSync(options.datasetPath ?? DEFAULT_DATASET, "utf8"),
+ );
+ return selectedVariants(dataset, options).map(({ testCase, variant }) =>
+ buildPromptfooTest(
+ testCase,
+ variant,
+ options.repoRoot ?? DEFAULT_REPO_ROOT,
+ ),
+ );
+}
+
if (
process.argv[1] &&
fs.existsSync(process.argv[1]) &&
import.meta.filename === fs.realpathSync(process.argv[1])
) {
const args = parseArgs(process.argv.slice(2));
- const dataset = JSON.parse(fs.readFileSync(args.dataset, "utf8"));
- const variants = selectedVariants(dataset, args);
- const tests = variants.map(({ testCase, variant }) =>
- testYaml(testCase, variant, args.repoRoot),
- );
- const output = `${tests.join("\n\n")}\n`;
+ const tests = generateTests({ ...args, datasetPath: args.dataset });
fs.mkdirSync(path.dirname(args.output), { recursive: true });
- fs.writeFileSync(args.output, output);
- console.log(`wrote ${variants.length} calibration tests to ${args.output}`);
+ fs.writeFileSync(args.output, stringify(tests));
+ console.log(`wrote ${tests.length} calibration tests to ${args.output}`);
}
diff --git a/evals/triage-finding/scripts/initialize-provider.mts b/evals/triage-finding/scripts/initialize-provider.mts
new file mode 100644
index 000000000..f8d5b7d40
--- /dev/null
+++ b/evals/triage-finding/scripts/initialize-provider.mts
@@ -0,0 +1,28 @@
+import { readdirSync } from "node:fs";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+import { initializeProvider } from "./triage-provider.mts";
+
+const promptfooDist = path.resolve(
+ import.meta.dirname,
+ "../node_modules/promptfoo/dist/src",
+);
+
+// Pinned Promptfoo 0.124.1 bundles separate native classes for the CLI, viewer,
+// and public API. Initialize every ESM copy before providers are constructed,
+// including viewer replay, which does not run saved extension hooks.
+for (const filename of readdirSync(promptfooDist)) {
+ if (!filename.startsWith("codex-sdk-") || !filename.endsWith(".js")) continue;
+ const module = await import(
+ pathToFileURL(path.join(promptfooDist, filename)).href
+ );
+ for (const Provider of Object.values(module)) {
+ if (
+ typeof Provider === "function" &&
+ typeof Provider.prototype?.getCodexInstanceForTurn === "function" &&
+ typeof Provider.prototype?.callApiInternal === "function"
+ ) {
+ initializeProvider(Provider.prototype);
+ }
+ }
+}
diff --git a/evals/triage-finding/scripts/run-promptfoo.mts b/evals/triage-finding/scripts/run-promptfoo.mts
index a6ec4a36b..ea9bbf8e4 100644
--- a/evals/triage-finding/scripts/run-promptfoo.mts
+++ b/evals/triage-finding/scripts/run-promptfoo.mts
@@ -3,6 +3,7 @@ import childProcess, { type ChildProcess } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
+import { pathToFileURL } from "node:url";
const EVAL_ROOT = path.resolve(import.meta.dirname, "..");
const PLUGIN_ROOT = path.resolve(
@@ -141,7 +142,16 @@ async function runPromptfoo(
path.join(PLUGIN_ROOT, "mcp-app"),
);
if (built !== 0) return built;
- return await runChild([PROMPTFOO_ENTRYPOINT, ...promptfooArgs], EVAL_ROOT);
+ return await runChild(
+ [
+ "--import",
+ pathToFileURL(path.join(import.meta.dirname, "initialize-provider.mts"))
+ .href,
+ PROMPTFOO_ENTRYPOINT,
+ ...promptfooArgs,
+ ],
+ EVAL_ROOT,
+ );
} finally {
for (const [signal, handler] of handlers) process.off(signal, handler);
fs.rmSync(runtimeRoot, { recursive: true, force: true });
diff --git a/evals/triage-finding/scripts/runtime-vars.mts b/evals/triage-finding/scripts/runtime-vars.mts
index 7f9020d43..932751bd0 100644
--- a/evals/triage-finding/scripts/runtime-vars.mts
+++ b/evals/triage-finding/scripts/runtime-vars.mts
@@ -6,7 +6,7 @@ const sourceRoot = path.resolve(evalRoot, "../..");
// Preserve the stable bindings used by saved native-provider configs, including
// viewer replay, which does not rerun transforms. Current configs bind their
-// temporary runtime through the beforeAll extension instead.
+// temporary runtime through the runner's preload instead.
export default (vars: Record<string, unknown>) => {
const nodePath = fs.realpathSync(process.execPath);
let targetRepo = vars.target_repo;
diff --git a/evals/triage-finding/scripts/test-generate-calibration-tests.mts b/evals/triage-finding/scripts/test-generate-calibration-tests.mts
index d2efbeb26..f47015d15 100644
--- a/evals/triage-finding/scripts/test-generate-calibration-tests.mts
+++ b/evals/triage-finding/scripts/test-generate-calibration-tests.mts
@@ -7,7 +7,11 @@ import os from "node:os";
import runtimeVars from "./runtime-vars.mts";
import path from "node:path";
import { parse } from "yaml";
-import { findingInput, variantCaseId } from "./generate-calibration-tests.mts";
+import {
+ findingInput,
+ generateTests,
+ variantCaseId,
+} from "./generate-calibration-tests.mts";
import type { CalibrationCase } from "../types.ts";
import { plannedJobs } from "./hydrate-calibration-repos.mts";
@@ -18,7 +22,6 @@ const generator = path.join(
"generate-calibration-tests.mts",
);
const dataset = path.join(evalDir, "datasets", "triage-calibration-seed.json");
-const trackedTests = path.join(evalDir, "tests", "calibration-oss.yaml");
const tmpDir = fs.realpathSync(
fs.mkdtempSync(path.join(os.tmpdir(), "triage-calibration-tests-")),
@@ -42,18 +45,8 @@ execFileSync(
},
);
-const generatedYaml = fs.readFileSync(generated, "utf8");
-const trackedYaml = fs.readFileSync(trackedTests, "utf8");
-
-assert.equal(
- generatedYaml,
- trackedYaml,
- "tracked calibration tests are stale; run calibration:generate",
-);
-const rows: {
- metadata: Record<string, string>;
- vars: Record<string, string>;
-}[] = parse(generatedYaml);
+const rows = generateTests();
+assert.deepEqual(parse(fs.readFileSync(generated, "utf8")), rows);
assert.equal(rows.length, 16);
assert.equal(new Set(rows.map((row) => row.vars.case_id)).size, rows.length);
for (const row of rows) {
@@ -61,6 +54,12 @@ for (const row of rows) {
assert.equal(row.vars.expected_ids, row.vars.case_id);
assert.equal(row.vars.calibration_repo, row.vars.case_id);
assert.ok(row.vars.finding_input.includes(`input_id: ${row.vars.case_id}`));
+ assert.equal(row.vars.calibration_repo_root, "");
+ assert.equal(
+ typeof row.vars.expected_evidence_terms,
+ "string",
+ "Promptfoo expands array-valued vars into extra test cases; evidence terms must be a scalar",
+ );
}
const cases: CalibrationCase[] = JSON.parse(
fs.readFileSync(dataset, "utf8"),
@@ -90,16 +89,28 @@ for (const testCase of cases) {
{ repoRoot },
);
assert.equal(job.targetDir, path.join(repoRoot, caseId));
- assert.ok(generatedYaml.includes(` calibration_repo: ${caseId}`));
+ assert.ok(rows.some((row) => row.vars.calibration_repo === caseId));
}
}
-assert.match(generatedYaml, /expected_verdicts: confirmed/);
-assert.match(generatedYaml, /expected_verdicts: not_actionable/);
-assert.match(generatedYaml, /calibration_repo_root: ""/);
-assert.doesNotMatch(
- generatedYaml,
- /expected_evidence_terms:\n\s+- /,
- "Promptfoo expands array-valued vars into extra test cases; evidence terms must be a scalar",
+assert.ok(rows.some((row) => row.vars.expected_verdicts === "confirmed"));
+assert.ok(rows.some((row) => row.vars.expected_verdicts === "not_actionable"));
+
+const smokeOptions = parse(
+ fs.readFileSync(
+ path.join(evalDir, "promptfooconfig.calibration-smoke.yaml"),
+ "utf8",
+ ),
+).tests.config;
+const smokeTests = generateTests(smokeOptions);
+assert.equal(smokeTests.length, 1);
+assert.equal(
+ smokeTests[0].metadata.calibration_case_id,
+ "oss-dompurify-ghsa-v8jm-5vwx-cfxm",
+);
+assert.equal(smokeTests[0].metadata.calibration_variant, "vulnerable");
+assert.throws(
+ () => generateTests({ caseId: "missing-case" }),
+ /No calibration variants matched/,
);
execFileSync(
@@ -124,35 +135,24 @@ execFileSync(
},
);
-const smokeYaml = fs.readFileSync(generatedSmoke, "utf8");
-assert.equal((smokeYaml.match(/^- description:/gm) || []).length, 1);
-assert.ok(
- smokeYaml.includes(
- `calibration_repo_root: ${JSON.stringify(path.join(tmpDir, "custom targets # space"))}`,
- ),
+const smokeRows = parse(fs.readFileSync(generatedSmoke, "utf8")) as typeof rows;
+assert.deepEqual(
+ smokeRows,
+ generateTests({
+ ...smokeOptions,
+ repoRoot: path.join(tmpDir, "custom targets # space"),
+ }),
);
const smokeCase = cases.find(
(testCase) => testCase.case_id === "oss-dompurify-ghsa-v8jm-5vwx-cfxm",
)!;
-assert.ok(
- smokeYaml.includes(
- `case_id: ${variantCaseId(
- smokeCase,
- smokeCase.variants.find(
- (variant) => variant.variant_id === "vulnerable",
- )!,
- )}`,
- ),
-);
-assert.ok(
- !smokeYaml.includes(
- `case_id: ${variantCaseId(
- smokeCase,
- smokeCase.variants.find((variant) => variant.variant_id === "fixed")!,
- )}`,
+assert.equal(
+ smokeRows[0].vars.case_id,
+ variantCaseId(
+ smokeCase,
+ smokeCase.variants.find((variant) => variant.variant_id === "vulnerable")!,
),
);
-const smokeRows = parse(smokeYaml) as typeof rows;
const hydration = execFileSync(
process.execPath,
[
@@ -178,5 +178,5 @@ assert.deepEqual(
fs.rmSync(tmpDir, { recursive: true, force: true });
console.log(
- "calibration test generation matches tracked YAML and supports filtered smoke output",
+ "native calibration test generation and optional YAML export preserve case identities and smoke filters",
);
diff --git a/evals/triage-finding/scripts/test-promptfoo-runner.mts b/evals/triage-finding/scripts/test-promptfoo-runner.mts
index 23121c2e2..aeb619421 100644
--- a/evals/triage-finding/scripts/test-promptfoo-runner.mts
+++ b/evals/triage-finding/scripts/test-promptfoo-runner.mts
@@ -3,6 +3,7 @@ import { spawn } from "node:child_process";
import fs from "node:fs";
import path from "node:path";
import test from "node:test";
+import { pathToFileURL } from "node:url";
import { stageSkillRuntime } from "./run-promptfoo.mts";
test("runtime contains checkout skill and fixtures without calibration labels", () => {
@@ -31,6 +32,62 @@ test("runtime contains checkout skill and fixtures without calibration labels",
}
});
+test("initializes native providers only in the Promptfoo child", async () => {
+ const driver = `
+ const cp = require('node:child_process');
+ const spawn = cp.spawn;
+ const calls = [];
+ cp.spawn = (command, args, options) => {
+ calls.push({ command, args, sameNodeOptions: options.env.NODE_OPTIONS === process.env.NODE_OPTIONS });
+ return spawn(command, ['--eval', ''], options);
+ };
+ import(${JSON.stringify(new URL("./run-promptfoo.mts", import.meta.url).href)}).then(({ runPromptfoo }) => runPromptfoo(['view', '--no'])).then(code => {
+ console.log(JSON.stringify({ code, calls }));
+ });
+ `;
+ const child = spawn(
+ process.execPath,
+ ["--experimental-strip-types", "--eval", driver],
+ { stdio: ["ignore", "pipe", "pipe"] },
+ );
+ let stdout = "";
+ let stderr = "";
+ child.stdout.on("data", (chunk) => {
+ stdout += chunk;
+ });
+ child.stderr.on("data", (chunk) => {
+ stderr += chunk;
+ });
+ const code = await new Promise<number | null>((resolve, reject) => {
+ child.once("error", reject);
+ child.once("close", resolve);
+ });
+ assert.equal(code, 0, stderr);
+ const result = JSON.parse(stdout) as {
+ code: number;
+ calls: { command: string; args: string[]; sameNodeOptions: boolean }[];
+ };
+ assert.equal(result.code, 0);
+ assert.equal(result.calls.length, 2);
+ const [helper, promptfoo] = result.calls;
+ assert.ok(helper!.args[0]!.endsWith("build_mcp_app.mjs"));
+ assert.equal(helper!.args.includes("--import"), false);
+ assert.equal(promptfoo!.args[0], "--import");
+ const initializer = promptfoo!.args[1]!;
+ assert.equal(
+ initializer.startsWith("file:")
+ ? initializer
+ : pathToFileURL(initializer).href,
+ new URL("./initialize-provider.mts", import.meta.url).href,
+ );
+ assert.ok(promptfoo!.args[2]!.endsWith("entrypoint.js"));
+ assert.deepEqual(promptfoo!.args.slice(3), ["view", "--no"]);
+ for (const call of result.calls) {
+ assert.equal(call.command, process.execPath);
+ assert.equal(call.sameNodeOptions, true);
+ }
+});
+
test(
"Windows console Ctrl+C lets the child save progress before runtime cleanup",
{ timeout: 15000 },
diff --git a/evals/triage-finding/scripts/test-promptfoo-runtime.mts b/evals/triage-finding/scripts/test-promptfoo-runtime.mts
index dd815d706..27ad04d52 100644
--- a/evals/triage-finding/scripts/test-promptfoo-runtime.mts
+++ b/evals/triage-finding/scripts/test-promptfoo-runtime.mts
@@ -4,6 +4,7 @@ import assert from "node:assert/strict";
import fs from "node:fs";
import path from "node:path";
import { pathToFileURL } from "node:url";
+import { parse } from "yaml";
import runtimeVars from "./runtime-vars.mts";
const evalRoot = path.resolve(import.meta.dirname, "..");
@@ -102,24 +103,57 @@ try {
for (const name of [
"promptfooconfig.yaml",
"promptfooconfig.calibration.yaml",
- "sastbench/promptfooconfig.sastbench.yaml",
+ "promptfooconfig.sastbench.yaml",
]) {
- const config = fs.readFileSync(path.join(evalRoot, name), "utf8");
- assert.match(config, /":minimal": read\n\s+":workspace_roots": read/);
- assert.match(config, /plugins: false/);
- assert.match(config, /memories: false/);
- assert.doesNotMatch(config, /working_dir:|CODEX_MCP_NODE_PATH:/);
+ const suite = parse(fs.readFileSync(path.join(evalRoot, name), "utf8"));
+ const providerPath = path.resolve(evalRoot, suite.providers[0].$ref);
+ const { id, config } = parse(fs.readFileSync(providerPath, "utf8"));
+ assert.match(id, /^openai:codex-sdk:/);
+ assert.deepEqual(
+ config.cli_config.permissions[config.cli_config.default_permissions],
+ {
+ filesystem: { ":minimal": "read", ":workspace_roots": "read" },
+ network: { enabled: false },
+ },
+ );
+ assert.deepEqual(config.cli_config.features, {
+ plugins: false,
+ memories: false,
+ });
+ assert.equal(config.skip_git_repo_check, true);
+ assert.equal(config.approval_policy, "never");
+ assert.equal(config.working_dir, undefined);
+ assert.equal(config.cli_env?.CODEX_MCP_NODE_PATH, undefined);
+ assert.equal(config.sandbox_mode, undefined);
+ assert.equal(config.network_access_enabled, undefined);
+ if (name !== "promptfooconfig.sastbench.yaml") {
+ for (const proxy of [
+ "HTTP_PROXY",
+ "HTTPS_PROXY",
+ "ALL_PROXY",
+ "NO_PROXY",
+ ]) {
+ assert.equal(config.cli_env[proxy], `{{env.${proxy}}}`);
+ }
+ }
if (name === "promptfooconfig.calibration.yaml") {
- assert.ok(config.includes("{{calibration_repo}}"));
- assert.match(config, /transformVars:/);
+ assert.ok(
+ config.additional_directories[0].includes("{{calibration_repo}}"),
+ );
+ assert.equal(
+ suite.defaultTest.options.transformVars,
+ "file://./scripts/runtime-vars.mts",
+ );
}
}
-assert.match(
- fs.readFileSync(
- path.join(evalRoot, "promptfooconfig.calibration-smoke.yaml"),
- "utf8",
- ),
- /\$ref: \.\/promptfooconfig\.calibration\.yaml/,
+assert.equal(
+ parse(
+ fs.readFileSync(
+ path.join(evalRoot, "promptfooconfig.calibration-smoke.yaml"),
+ "utf8",
+ ),
+ ).$ref,
+ "./promptfooconfig.calibration.yaml",
);
assert.ok(
fs
diff --git a/evals/triage-finding/scripts/test-triage-provider.mts b/evals/triage-finding/scripts/test-triage-provider.mts
index aed7e7e8e..5c5eda611 100644
--- a/evals/triage-finding/scripts/test-triage-provider.mts
+++ b/evals/triage-finding/scripts/test-triage-provider.mts
@@ -11,6 +11,11 @@ import { beforeAll } from "./triage-provider.mts";
const evalRoot = path.resolve(import.meta.dirname, "..");
const runner = path.join(import.meta.dirname, "run-promptfoo.mts");
+function providerConfig(name: string) {
+ return parse(
+ fs.readFileSync(path.join(evalRoot, "providers", `${name}.yaml`), "utf8"),
+ );
+}
const extensions = [
`file://${path.join(import.meta.dirname, "triage-provider.mts")}:beforeAll`,
];
@@ -57,7 +62,7 @@ function invoke(args: string[], environment: NodeJS.ProcessEnv) {
}
test(
- "runtime extension preserves native credential precedence and wraps providers once",
+ "preloaded runtime preserves native credentials and legacy hooks wrap once",
{ skip: process.platform === "win32" },
async (t) => {
const root = fs.realpathSync(
@@ -80,6 +85,7 @@ console.log(JSON.stringify({type:'turn.completed',usage:{input_tokens:1,output_t
`,
{ mode: 0o755 },
);
+ await import("./initialize-provider.mts");
const { loadApiProvider } = await import("promptfoo");
const cases = [
{
@@ -113,13 +119,16 @@ console.log(JSON.stringify({type:'turn.completed',usage:{input_tokens:1,output_t
},
];
for (const entry of cases) {
- const load = () =>
+ const load = (runtime = false) =>
loadApiProvider("openai:codex-sdk:gpt-5.5", {
basePath: root,
env: entry.env,
options: {
env: entry.providerEnv,
config: {
+ cli_config: runtime
+ ? { default_permissions: "triage_runtime_only" }
+ : undefined,
working_dir: root,
codex_path_override: fakeCodex,
skip_git_repo_check: true,
@@ -131,13 +140,19 @@ console.log(JSON.stringify({type:'turn.completed',usage:{input_tokens:1,output_t
},
});
const native = await load();
- const adapted = await load();
+ const adapted = await load(true);
try {
+ const preloaded = await adapted.callApi("synthetic");
const context = { suite: { providers: [adapted] } };
beforeAll(context);
beforeAll(context);
const original = await native.callApi("synthetic");
const result = await adapted.callApi("synthetic");
+ assert.equal(preloaded.error, undefined);
+ assert.deepEqual(
+ JSON.parse(String(preloaded.output)),
+ JSON.parse(String(result.output)),
+ );
assert.equal(original.error, undefined);
assert.equal(result.error, undefined);
const originalOutput = JSON.parse(String(original.output));
@@ -159,6 +174,80 @@ console.log(JSON.stringify({type:'turn.completed',usage:{input_tokens:1,output_t
},
);
+test(
+ "native CLI provider overrides without a working directory use the staged runtime",
+ { skip: process.platform === "win32", timeout: 120000 },
+ async (t) => {
+ const root = fs.realpathSync(
+ fs.mkdtempSync(path.join(os.tmpdir(), "triage-cli-provider-")),
+ );
+ t.after(() => fs.rmSync(root, { recursive: true, force: true }));
+ const capture = path.join(root, "capture.json");
+ const fakeCodex = path.join(root, "custom-codex");
+ fs.writeFileSync(
+ fakeCodex,
+ `#!${process.execPath}
+const fs = require('node:fs');
+const path = require('node:path');
+const cwd = process.argv[process.argv.indexOf('--cd') + 1];
+fs.readFileSync(path.join(cwd, 'plugins/codex-security/skills/triage-finding/SKILL.md'));
+fs.writeFileSync(${JSON.stringify(capture)}, JSON.stringify({cwd,node:process.env.CODEX_MCP_NODE_PATH,directories:process.argv.flatMap((arg,index)=>arg==='--add-dir'?[process.argv[index+1]]:[])}));
+console.log(JSON.stringify({type:'thread.started',thread_id:'synthetic-cli-provider'}));
+console.log(JSON.stringify({type:'item.completed',item:{id:'message',type:'agent_message',text:'ok'}}));
+console.log(JSON.stringify({type:'turn.completed',usage:{input_tokens:1,output_tokens:1}}));
+`,
+ { mode: 0o755 },
+ );
+ const provider = {
+ id: "openai:codex-sdk:gpt-5.5",
+ config: {
+ codex_path_override: fakeCodex,
+ skip_git_repo_check: true,
+ approval_policy: "never",
+ cli_env: { CODEX_MCP_NODE_PATH: process.execPath },
+ },
+ };
+ const configPath = path.join(root, "config.json");
+ fs.writeFileSync(
+ configPath,
+ JSON.stringify({
+ providers: [provider],
+ prompts: ["hello"],
+ tests: [{ assert: [{ type: "equals", value: "ok" }] }],
+ }),
+ );
+ const result = await invoke(
+ [
+ "eval",
+ "-c",
+ configPath,
+ "--providers",
+ provider.id,
+ "--no-cache",
+ "--no-share",
+ "--no-progress-bar",
+ ],
+ {
+ OPENAI_API_KEY: "synthetic-test-key",
+ PROMPTFOO_CONFIG_DIR: path.join(root, "state"),
+ PROMPTFOO_DISABLE_WAL_MODE: "true",
+ PROMPTFOO_DISABLE_TELEMETRY: "1",
+ PROMPTFOO_DISABLE_UPDATE: "1",
+ },
+ );
+ assert.equal(result.code, 0, result.output);
+ const row = JSON.parse(fs.readFileSync(capture, "utf8"));
+ assert.notEqual(row.cwd, evalRoot);
+ assert.equal(fs.existsSync(row.cwd), false);
+ assert.equal(row.node, process.execPath);
+ assert.deepEqual(row.directories, [
+ path.dirname(fs.realpathSync(process.execPath)),
+ path.dirname(process.execPath),
+ path.dirname(fakeCodex),
+ ]);
+ },
+);
+
// Exercise the pinned provider and Codex SDK subprocess without a model request.
// The normal setup command supplies the host helper build used by the runner.
test(
@@ -213,25 +302,17 @@ if (fs.existsSync(${JSON.stringify(fail)})) {
PROMPTFOO_DISABLE_UPDATE: "1",
};
const configPath = path.join(root, "config.json");
- const provider = parse(
- fs.readFileSync(path.join(evalRoot, "promptfooconfig.yaml"), "utf8"),
- ).providers[0];
+ const provider = providerConfig("triage");
provider.config.codex_path_override = fakeCodex;
fs.writeFileSync(
configPath,
JSON.stringify({
- extensions,
providers: [provider],
prompts: ["hello"],
tests: [{ assert: [{ type: "equals", value: "ok" }] }],
}),
);
- const calibration = parse(
- fs.readFileSync(
- path.join(evalRoot, "promptfooconfig.calibration.yaml"),
- "utf8",
- ),
- ).providers[0];
+ const calibration = providerConfig("calibration");
calibration.id = provider.id;
calibration.config.codex_path_override = fakeCodex;
const customNode = path.join(root, "custom-node", "bin", "node");
@@ -278,7 +359,6 @@ case "$0" in */node-alias) exec '${process.execPath.replaceAll("'", "'\\''")}' "
fs.writeFileSync(
calibrationConfig,
JSON.stringify({
- extensions,
defaultTest: {
options: {
transformVars: `file://${path.join(import.meta.dirname, "runtime-vars.mts")}`,
@@ -366,7 +446,6 @@ case "$0" in */node-alias) exec '${process.execPath.replaceAll("'", "'\\''")}' "
fs.writeFileSync(
templatedConfig,
JSON.stringify({
- extensions,
providers: [
{
...provider,
@@ -1138,9 +1217,13 @@ if (fs.existsSync(${JSON.stringify(failure)})) {
},
);
-for (const suite of ["SAST", "calibration"] as const) {
+for (const [suite, legacy] of [
+ ["SAST", false],
+ ["calibration", false],
+ ["SAST", true],
+] as const) {
test(
- `${suite} provider survives persisted retry, resume, and viewer replay`,
+ `${suite}${legacy ? " legacy hook" : ""} provider survives persisted retry, resume, and viewer replay`,
{ skip: process.platform === "win32", timeout: 120000 },
async (t) => {
const root = fs.realpathSync(
@@ -1163,7 +1246,10 @@ for (const suite of ["SAST", "calibration"] as const) {
`#!${process.execPath}
const fs = require('node:fs');
const cwd = process.argv[process.argv.indexOf('--cd') + 1];
-fs.appendFileSync(${JSON.stringify(capture)}, JSON.stringify({ cwd, directories: process.argv.flatMap((arg, index) => arg === '--add-dir' ? [process.argv[index + 1]] : []) }) + '\\n');
+const model = process.argv[process.argv.indexOf('--model') + 1];
+const overrides = process.argv.flatMap((arg, index) => arg === '--config' ? [process.argv[index + 1]] : []);
+const effort = overrides.find(value => value.startsWith('model_reasoning_effort='));
+fs.appendFileSync(${JSON.stringify(capture)}, JSON.stringify({ cwd, model, effort, marker: process.env.SAVED_PROVIDER_MARKER, directories: process.argv.flatMap((arg, index) => arg === '--add-dir' ? [process.argv[index + 1]] : []) }) + '\\n');
console.log(JSON.stringify({ type: 'thread.started', thread_id: 'synthetic-sast-replay' }));
if (fs.existsSync(${JSON.stringify(failure)})) {
console.log(JSON.stringify({ type: 'turn.failed', error: { message: 'synthetic retryable failure' } }));
@@ -1187,15 +1273,25 @@ if (fs.existsSync(${JSON.stringify(failure)})) {
path.join(
evalRoot,
suite === "SAST"
- ? "sastbench/promptfooconfig.sastbench.yaml"
+ ? "promptfooconfig.sastbench.yaml"
: "promptfooconfig.calibration.yaml",
),
"utf8",
),
);
- const provider = template.providers[0];
+ const provider = providerConfig(
+ suite === "SAST" ? "sastbench" : "calibration",
+ );
provider.config.codex_path_override = fakeCodex;
- // Keep the real YAML provider reference and its original configuration directory.
+ provider.config.model = "gpt-5.5";
+ provider.config.model_reasoning_effort = "high";
+ provider.config.cli_env = {
+ ...provider.config.cli_env,
+ SAVED_PROVIDER_MARKER: "original-provider",
+ };
+ const providerFile = path.join(root, "provider.json");
+ fs.writeFileSync(providerFile, JSON.stringify(provider));
+ // Resolve suite hooks relative to their original configuration directory.
const configPath = path.join(
evalRoot,
suite === "SAST" ? "sastbench" : ".",
@@ -1205,8 +1301,10 @@ if (fs.existsSync(${JSON.stringify(failure)})) {
fs.writeFileSync(
configPath,
JSON.stringify({
- extensions: template.extensions,
- providers: [provider],
+ extensions: legacy
+ ? [...extensions, ...(template.extensions ?? [])]
+ : template.extensions,
+ providers: [legacy ? provider : { $ref: providerFile }],
prompts: ["hello"],
defaultTest:
suite === "calibration"
@@ -1248,20 +1346,52 @@ if (fs.existsSync(${JSON.stringify(failure)})) {
);
assert.notEqual(initial.code, 0, initial.output);
assert.match(initial.output, /synthetic retryable failure/);
+ const initialDatabase = new DatabaseSync(
+ path.join(environment.PROMPTFOO_CONFIG_DIR, "promptfoo.db"),
+ { enableForeignKeyConstraints: false },
+ );
+ const initialEvaluation = initialDatabase
+ .prepare("SELECT config FROM evals ORDER BY created_at DESC LIMIT 1")
+ .get();
+ assert.ok(
+ initialEvaluation && typeof initialEvaluation.config === "string",
+ );
+ const savedProvider = JSON.parse(initialEvaluation.config).providers[0];
+ assert.equal(savedProvider.id, provider.id);
+ assert.equal(savedProvider.$ref, undefined);
+ assert.equal(savedProvider.config.model, "gpt-5.5");
+ assert.equal(savedProvider.config.model_reasoning_effort, "high");
+ assert.equal(
+ savedProvider.config.cli_env.SAVED_PROVIDER_MARKER,
+ "original-provider",
+ );
if (suite === "SAST") {
- const initialDatabase = new DatabaseSync(
- path.join(environment.PROMPTFOO_CONFIG_DIR, "promptfoo.db"),
- { enableForeignKeyConstraints: false },
- );
const initialResult = initialDatabase
.prepare("SELECT metadata FROM eval_results LIMIT 1")
.get();
- initialDatabase.close();
assert.ok(
JSON.parse((initialResult?.metadata as string) || "{}").sastbench,
initial.output,
);
}
+ initialDatabase.close();
+ if (!legacy) {
+ fs.writeFileSync(
+ providerFile,
+ JSON.stringify({
+ ...provider,
+ config: {
+ ...provider.config,
+ model: "gpt-5.4",
+ model_reasoning_effort: "minimal",
+ cli_env: {
+ ...provider.config.cli_env,
+ SAVED_PROVIDER_MARKER: "changed-provider",
+ },
+ },
+ }),
+ );
+ }
fs.rmSync(failure);
const retry = await invoke(
[
@@ -1284,10 +1414,13 @@ if (fs.existsSync(${JSON.stringify(failure)})) {
)
.get();
assert.ok(stored && typeof stored.config === "string");
- assert.equal(JSON.parse(stored.config).providers[0].id, provider.id);
+ assert.deepEqual(JSON.parse(stored.config).providers[0], savedProvider);
assert.deepEqual(
JSON.parse(stored.config).extensions,
- template.extensions.map((extension: string) =>
+ (legacy
+ ? [...extensions, ...(template.extensions ?? [])]
+ : (template.extensions ?? [])
+ ).map((extension: string) =>
extension.replace(
"{{env.TRIAGE_PROVIDER_PATH}}",
path.join(import.meta.dirname, "triage-provider.mts"),
@@ -1394,6 +1527,9 @@ if (fs.existsSync(${JSON.stringify(failure)})) {
assert.equal(new Set(rows.map((row) => row.cwd)).size, 4);
for (const row of rows) {
assert.equal(fs.existsSync(row.cwd), false);
+ assert.equal(row.model, "gpt-5.5");
+ assert.equal(row.effort, 'model_reasoning_effort="high"');
+ assert.equal(row.marker, "original-provider");
assert.deepEqual(row.directories, [
...(suite === "SAST"
? [
diff --git a/evals/triage-finding/scripts/triage-provider.mts b/evals/triage-finding/scripts/triage-provider.mts
index dd9351db9..7aeb5993a 100644
--- a/evals/triage-finding/scripts/triage-provider.mts
+++ b/evals/triage-finding/scripts/triage-provider.mts
@@ -69,120 +69,153 @@ function resolveCommand(
throw new Error(`Executable not found on PATH: ${command}`);
}
-const wrappedProviders = new WeakSet<ApiProvider>();
+const wrappedProviders = new WeakSet<CodexProvider>();
+const explicitProviders = new WeakSet<ApiProvider>();
-// Keep native provider loading and authentication unchanged. Bind the staged
-// runtime after Promptfoo has created each provider for this evaluation.
-export function beforeAll({ suite }: { suite: { providers: ApiProvider[] } }) {
- for (const apiProvider of suite.providers) {
- const provider = apiProvider as CodexProvider;
- if (
- typeof provider.getCodexInstanceForTurn !== "function" ||
- wrappedProviders.has(provider)
- )
- continue;
- const getInstance = provider.getCodexInstanceForTurn.bind(provider);
- provider.getCodexInstanceForTurn = async (
+function usesTriageRuntime(provider: CodexProvider) {
+ return (
+ explicitProviders.has(provider) ||
+ provider.config?.working_dir === undefined ||
+ provider.config?.cli_config?.default_permissions ===
+ "triage_runtime_only" ||
+ provider.config?.cli_config?.default_permissions ===
+ "sastbench_runtime_only"
+ );
+}
+
+// Promptfoo 0.124.1 exposes these private methods after per-case rendering and
+// SDK selection. Keep the native loader, authentication, and lifecycle intact.
+export function initializeProvider(provider: CodexProvider) {
+ for (let target = provider; target; target = Object.getPrototypeOf(target)) {
+ if (wrappedProviders.has(target)) return;
+ }
+ const getInstance = provider.getCodexInstanceForTurn;
+ provider.getCodexInstanceForTurn = async function (
+ this: CodexProvider,
+ environment,
+ config,
+ apiKey,
+ cleanupGeneration,
+ ) {
+ const instance = await getInstance.call(
+ this,
environment,
config,
apiKey,
cleanupGeneration,
- ) => {
- const instance = await getInstance(
- environment,
- config,
- apiKey,
- cleanupGeneration,
- );
- // The pinned provider exposes its selected SDK executor before creating
- // thread options. Extend this call's config without changing selection
- // or the SDK's bundled-tool PATH handling.
- const { executablePath, pathDirs } = instance.activeInstance.exec;
- const executable = realpathSync.native(
- resolveCommand(executablePath, process.cwd(), environment),
- );
- const directories = [path.dirname(executable), ...pathDirs];
- if (path.basename(executable) === "codex.js") {
- try {
- const packageJson = path.join(
- path.dirname(executable),
- "..",
- "package.json",
- );
- if (
- JSON.parse(readFileSync(packageJson, "utf8")).name ===
- "@openai/codex"
- ) {
- const native = resolveBundledCodexExecutable(packageJson);
- directories.push(
- path.dirname(native),
- ...Object.values(bundledCodexSdkEnvironment(native, {})),
- );
- }
- } catch {
- // Other launcher layouts keep their configured directories and
- // report their own dependency errors when the SDK starts them.
- }
- }
- config.additional_directories = [
- ...(config.additional_directories ?? []),
- ...directories,
- ];
- return instance;
- };
- const callApiInternal = provider.callApiInternal.bind(provider);
- // The pinned Codex provider merges prompt overrides and renders case
- // variables before this call. Keep that upstream behavior for each case.
- provider.callApiInternal = (
- prompt,
- context,
- options,
- config,
- ...lifecycle
- ) => {
- const runtimeRoot = process.env.TRIAGE_RUNTIME_ROOT;
- if (!runtimeRoot) {
- throw new Error(
- "Run this evaluation through scripts/run-promptfoo.mts.",
- );
- }
- const requestedNode =
- (config.cli_env?.CODEX_MCP_NODE_PATH ??
- process.env.CODEX_MCP_NODE_PATH ??
- process.execPath) ||
- process.execPath;
- const environment = { ...process.env, ...config.cli_env };
- let nodeCommand: string;
+ );
+ if (!usesTriageRuntime(this)) return instance;
+ // The pinned provider exposes its selected SDK executor before creating
+ // thread options. Extend this call's config without changing selection
+ // or the SDK's bundled-tool PATH handling.
+ const { executablePath, pathDirs } = instance.activeInstance.exec;
+ const executable = realpathSync.native(
+ resolveCommand(executablePath, process.cwd(), environment),
+ );
+ const directories = [path.dirname(executable), ...pathDirs];
+ if (path.basename(executable) === "codex.js") {
try {
- nodeCommand = resolveCommand(requestedNode, runtimeRoot, environment);
- accessSync(
- nodeCommand,
- process.platform === "win32" ? constants.F_OK : constants.X_OK,
+ const packageJson = path.join(
+ path.dirname(executable),
+ "..",
+ "package.json",
);
+ if (
+ JSON.parse(readFileSync(packageJson, "utf8")).name === "@openai/codex"
+ ) {
+ const native = resolveBundledCodexExecutable(packageJson);
+ directories.push(
+ path.dirname(native),
+ ...Object.values(bundledCodexSdkEnvironment(native, {})),
+ );
+ }
} catch {
- nodeCommand = resolveCommand("node", runtimeRoot, environment);
+ // Other launcher layouts keep their configured directories and
+ // report their own dependency errors when the SDK starts them.
}
- const nodePath =
- process.platform === "win32"
- ? realpathSync(nodeCommand)
- : realpathSync.native(nodeCommand);
- return callApiInternal(
+ }
+ config.additional_directories = [
+ ...(config.additional_directories ?? []),
+ ...directories,
+ ];
+ return instance;
+ };
+ const callApiInternal = provider.callApiInternal;
+ // The pinned Codex provider merges prompt overrides and renders case
+ // variables before this call. Keep that upstream behavior for each case.
+ provider.callApiInternal = function (
+ this: CodexProvider,
+ prompt,
+ context,
+ options,
+ config,
+ ...lifecycle
+ ) {
+ if (!usesTriageRuntime(this)) {
+ return callApiInternal.call(
+ this,
prompt,
context,
options,
- {
- ...config,
- working_dir: runtimeRoot,
- cli_env: { ...config.cli_env, CODEX_MCP_NODE_PATH: nodeCommand },
- additional_directories: [
- ...(config.additional_directories ?? []),
- path.dirname(nodePath),
- path.dirname(nodeCommand),
- ],
- },
+ config,
...lifecycle,
);
- };
- wrappedProviders.add(provider);
+ }
+ const runtimeRoot = process.env.TRIAGE_RUNTIME_ROOT;
+ if (!runtimeRoot) {
+ throw new Error("Run this evaluation through scripts/run-promptfoo.mts.");
+ }
+ const requestedNode =
+ (config.cli_env?.CODEX_MCP_NODE_PATH ??
+ process.env.CODEX_MCP_NODE_PATH ??
+ process.execPath) ||
+ process.execPath;
+ const environment = { ...process.env, ...config.cli_env };
+ let nodeCommand: string;
+ try {
+ nodeCommand = resolveCommand(requestedNode, runtimeRoot, environment);
+ accessSync(
+ nodeCommand,
+ process.platform === "win32" ? constants.F_OK : constants.X_OK,
+ );
+ } catch {
+ nodeCommand = resolveCommand("node", runtimeRoot, environment);
+ }
+ const nodePath =
+ process.platform === "win32"
+ ? realpathSync(nodeCommand)
+ : realpathSync.native(nodeCommand);
+ return callApiInternal.call(
+ this,
+ prompt,
+ context,
+ options,
+ {
+ ...config,
+ working_dir: runtimeRoot,
+ cli_env: { ...config.cli_env, CODEX_MCP_NODE_PATH: nodeCommand },
+ additional_directories: [
+ ...(config.additional_directories ?? []),
+ path.dirname(nodePath),
+ path.dirname(nodeCommand),
+ ],
+ },
+ ...lifecycle,
+ );
+ };
+ wrappedProviders.add(provider);
+}
+
+// Older saved evaluations still load this hook during retry and resume.
+export function beforeAll({ suite }: { suite: { providers: ApiProvider[] } }) {
+ for (const apiProvider of suite.providers) {
+ const provider = apiProvider as CodexProvider;
+ if (
+ typeof provider.getCodexInstanceForTurn !== "function" ||
+ typeof provider.callApiInternal !== "function"
+ )
+ continue;
+ explicitProviders.add(provider);
+ initializeProvider(provider);
}
}
diff --git a/evals/triage-finding/tests/calibration-oss.yaml b/evals/triage-finding/tests/calibration-oss.yaml
deleted file mode 100644
index 29f004e82..000000000
--- a/evals/triage-finding/tests/calibration-oss.yaml
+++ /dev/null
@@ -1,543 +0,0 @@
-- description: "calibration vulnerable: oss-mantisbt-ghsa-73vx-49mv-v8w5"
- metadata:
- case_id: calibration-393a6e303af5b190
- suite: calibration-oss
- calibration_case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5
- calibration_variant: vulnerable
- repo: "mantisbt/mantisbt"
- expected_binary_label: positive
- vars:
- case_id: calibration-393a6e303af5b190
- calibration_repo: calibration-393a6e303af5b190
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-393a6e303af5b190
- expected_source_types: cve
- expected_verdicts: confirmed
- expected_binary_label: positive
- expected_evidence_terms: "core/classes/IssueTagTimelineEvent.class.php"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-393a6e303af5b190
- title: GHSA-73vx-49mv-v8w5 / CVE-2026-33548
- advisory ids: GHSA-73vx-49mv-v8w5, CVE-2026-33548
- weakness: xss
- severity: high
- affected language: PHP
- anchor locations: core/classes/IssueTagTimelineEvent.class.php:61
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration fixed: oss-mantisbt-ghsa-73vx-49mv-v8w5"
- metadata:
- case_id: calibration-0310e3e9455d0db5
- suite: calibration-oss
- calibration_case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5
- calibration_variant: fixed
- repo: "mantisbt/mantisbt"
- expected_binary_label: negative
- vars:
- case_id: calibration-0310e3e9455d0db5
- calibration_repo: calibration-0310e3e9455d0db5
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-0310e3e9455d0db5
- expected_source_types: cve
- expected_verdicts: not_actionable
- expected_binary_label: negative
- expected_evidence_terms: "core/classes/IssueTagTimelineEvent.class.php"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-0310e3e9455d0db5
- title: GHSA-73vx-49mv-v8w5 / CVE-2026-33548
- advisory ids: GHSA-73vx-49mv-v8w5, CVE-2026-33548
- weakness: xss
- severity: high
- affected language: PHP
- anchor locations: core/classes/IssueTagTimelineEvent.class.php:61
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration vulnerable: oss-imagemagick-ghsa-qpgx-jfcq-r59f"
- metadata:
- case_id: calibration-89e4ec9f158dc8b9
- suite: calibration-oss
- calibration_case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f
- calibration_variant: vulnerable
- repo: "ImageMagick/ImageMagick"
- expected_binary_label: positive
- vars:
- case_id: calibration-89e4ec9f158dc8b9
- calibration_repo: calibration-89e4ec9f158dc8b9
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-89e4ec9f158dc8b9
- expected_source_types: cve
- expected_verdicts: confirmed
- expected_binary_label: positive
- expected_evidence_terms: "MagickCore/visual-effects.c"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-89e4ec9f158dc8b9
- title: GHSA-qpgx-jfcq-r59f / CVE-2026-27798
- advisory ids: GHSA-qpgx-jfcq-r59f, CVE-2026-27798
- weakness: memory-corruption
- severity: medium
- affected language: C/C++
- anchor locations: MagickCore/visual-effects.c:3566
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration fixed: oss-imagemagick-ghsa-qpgx-jfcq-r59f"
- metadata:
- case_id: calibration-11948efe569f3fb9
- suite: calibration-oss
- calibration_case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f
- calibration_variant: fixed
- repo: "ImageMagick/ImageMagick"
- expected_binary_label: negative
- vars:
- case_id: calibration-11948efe569f3fb9
- calibration_repo: calibration-11948efe569f3fb9
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-11948efe569f3fb9
- expected_source_types: cve
- expected_verdicts: not_actionable
- expected_binary_label: negative
- expected_evidence_terms: "MagickCore/visual-effects.c"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-11948efe569f3fb9
- title: GHSA-qpgx-jfcq-r59f / CVE-2026-27798
- advisory ids: GHSA-qpgx-jfcq-r59f, CVE-2026-27798
- weakness: memory-corruption
- severity: medium
- affected language: C/C++
- anchor locations: MagickCore/visual-effects.c:3566
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration vulnerable: oss-scitokens-ghsa-rh5m-2482-966c"
- metadata:
- case_id: calibration-576c7d9cbbc7c018
- suite: calibration-oss
- calibration_case_id: oss-scitokens-ghsa-rh5m-2482-966c
- calibration_variant: vulnerable
- repo: "scitokens/scitokens"
- expected_binary_label: positive
- vars:
- case_id: calibration-576c7d9cbbc7c018
- calibration_repo: calibration-576c7d9cbbc7c018
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-576c7d9cbbc7c018
- expected_source_types: cve
- expected_verdicts: confirmed
- expected_binary_label: positive
- expected_evidence_terms: "src/scitokens/utils/keycache.py"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-576c7d9cbbc7c018
- title: GHSA-rh5m-2482-966c / CVE-2026-32714
- advisory ids: GHSA-rh5m-2482-966c, CVE-2026-32714
- weakness: sql-injection
- severity: critical
- affected language: Python
- anchor locations: src/scitokens/utils/keycache.py:80
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration fixed: oss-scitokens-ghsa-rh5m-2482-966c"
- metadata:
- case_id: calibration-3aaf9d4f40ef518c
- suite: calibration-oss
- calibration_case_id: oss-scitokens-ghsa-rh5m-2482-966c
- calibration_variant: fixed
- repo: "scitokens/scitokens"
- expected_binary_label: negative
- vars:
- case_id: calibration-3aaf9d4f40ef518c
- calibration_repo: calibration-3aaf9d4f40ef518c
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-3aaf9d4f40ef518c
- expected_source_types: cve
- expected_verdicts: not_actionable
- expected_binary_label: negative
- expected_evidence_terms: "src/scitokens/utils/keycache.py"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-3aaf9d4f40ef518c
- title: GHSA-rh5m-2482-966c / CVE-2026-32714
- advisory ids: GHSA-rh5m-2482-966c, CVE-2026-32714
- weakness: sql-injection
- severity: critical
- affected language: Python
- anchor locations: src/scitokens/utils/keycache.py:80
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration vulnerable: oss-emissary-ghsa-3p24-9x7v-7789"
- metadata:
- case_id: calibration-1561f6cf0aa4fa71
- suite: calibration-oss
- calibration_case_id: oss-emissary-ghsa-3p24-9x7v-7789
- calibration_variant: vulnerable
- repo: "NationalSecurityAgency/emissary"
- expected_binary_label: positive
- vars:
- case_id: calibration-1561f6cf0aa4fa71
- calibration_repo: calibration-1561f6cf0aa4fa71
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-1561f6cf0aa4fa71
- expected_source_types: cve
- expected_verdicts: confirmed
- expected_binary_label: positive
- expected_evidence_terms: "src/main/java/emissary/util/shell/Executrix.java"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-1561f6cf0aa4fa71
- title: GHSA-3p24-9x7v-7789 / CVE-2026-35582
- advisory ids: GHSA-3p24-9x7v-7789, CVE-2026-35582
- weakness: command-injection
- severity: high
- affected language: Java
- anchor locations: src/main/java/emissary/util/shell/Executrix.java:77
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration fixed: oss-emissary-ghsa-3p24-9x7v-7789"
- metadata:
- case_id: calibration-ca1bddda373398e8
- suite: calibration-oss
- calibration_case_id: oss-emissary-ghsa-3p24-9x7v-7789
- calibration_variant: fixed
- repo: "NationalSecurityAgency/emissary"
- expected_binary_label: negative
- vars:
- case_id: calibration-ca1bddda373398e8
- calibration_repo: calibration-ca1bddda373398e8
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-ca1bddda373398e8
- expected_source_types: cve
- expected_verdicts: not_actionable
- expected_binary_label: negative
- expected_evidence_terms: "src/main/java/emissary/util/shell/Executrix.java"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-ca1bddda373398e8
- title: GHSA-3p24-9x7v-7789 / CVE-2026-35582
- advisory ids: GHSA-3p24-9x7v-7789, CVE-2026-35582
- weakness: command-injection
- severity: high
- affected language: Java
- anchor locations: src/main/java/emissary/util/shell/Executrix.java:77
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration vulnerable: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5"
- metadata:
- case_id: calibration-2cd95700e4d0032d
- suite: calibration-oss
- calibration_case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5
- calibration_variant: vulnerable
- repo: "free5gc/pcf"
- expected_binary_label: positive
- vars:
- case_id: calibration-2cd95700e4d0032d
- calibration_repo: calibration-2cd95700e4d0032d
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-2cd95700e4d0032d
- expected_source_types: cve
- expected_verdicts: confirmed
- expected_binary_label: positive
- expected_evidence_terms: "internal/sbi/server.go"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-2cd95700e4d0032d
- title: GHSA-6RGM-GR97-X3J5 / CVE-2026-42083
- advisory ids: GHSA-6RGM-GR97-X3J5, CVE-2026-42083
- weakness: authz
- severity: high
- affected language: Go
- anchor locations: internal/sbi/server.go:72
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration fixed: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5"
- metadata:
- case_id: calibration-1b89c72834a4a631
- suite: calibration-oss
- calibration_case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5
- calibration_variant: fixed
- repo: "free5gc/pcf"
- expected_binary_label: negative
- vars:
- case_id: calibration-1b89c72834a4a631
- calibration_repo: calibration-1b89c72834a4a631
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-1b89c72834a4a631
- expected_source_types: cve
- expected_verdicts: not_actionable
- expected_binary_label: negative
- expected_evidence_terms: "internal/sbi/server.go"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-1b89c72834a4a631
- title: GHSA-6RGM-GR97-X3J5 / CVE-2026-42083
- advisory ids: GHSA-6RGM-GR97-X3J5, CVE-2026-42083
- weakness: authz
- severity: high
- affected language: Go
- anchor locations: internal/sbi/server.go:72
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration vulnerable: oss-actualbudget-ghsa-27vg-33gh-4hwg"
- metadata:
- case_id: calibration-e4d2cc784a81e08c
- suite: calibration-oss
- calibration_case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg
- calibration_variant: vulnerable
- repo: "actualbudget/actual"
- expected_binary_label: positive
- vars:
- case_id: calibration-e4d2cc784a81e08c
- calibration_repo: calibration-e4d2cc784a81e08c
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-e4d2cc784a81e08c
- expected_source_types: cve
- expected_verdicts: confirmed
- expected_binary_label: positive
- expected_evidence_terms: "packages/sync-server/src/app-sync.ts, packages/sync-server/src/app-sync.test.ts"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-e4d2cc784a81e08c
- title: GHSA-27vg-33gh-4hwg / CVE-2026-3089
- advisory ids: GHSA-27vg-33gh-4hwg, CVE-2026-3089
- weakness: path-traversal
- severity: medium
- affected language: TypeScript
- anchor locations: packages/sync-server/src/app-sync.ts:49, packages/sync-server/src/app-sync.test.ts:366
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration fixed: oss-actualbudget-ghsa-27vg-33gh-4hwg"
- metadata:
- case_id: calibration-b4528e41b5de5a74
- suite: calibration-oss
- calibration_case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg
- calibration_variant: fixed
- repo: "actualbudget/actual"
- expected_binary_label: negative
- vars:
- case_id: calibration-b4528e41b5de5a74
- calibration_repo: calibration-b4528e41b5de5a74
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-b4528e41b5de5a74
- expected_source_types: cve
- expected_verdicts: not_actionable
- expected_binary_label: negative
- expected_evidence_terms: "packages/sync-server/src/app-sync.ts, packages/sync-server/src/app-sync.test.ts"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-b4528e41b5de5a74
- title: GHSA-27vg-33gh-4hwg / CVE-2026-3089
- advisory ids: GHSA-27vg-33gh-4hwg, CVE-2026-3089
- weakness: path-traversal
- severity: medium
- affected language: TypeScript
- anchor locations: packages/sync-server/src/app-sync.ts:49, packages/sync-server/src/app-sync.test.ts:366
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration vulnerable: oss-dompurify-ghsa-v8jm-5vwx-cfxm"
- metadata:
- case_id: calibration-6228a29e06832095
- suite: calibration-oss
- calibration_case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm
- calibration_variant: vulnerable
- repo: "cure53/DOMPurify"
- expected_binary_label: positive
- vars:
- case_id: calibration-6228a29e06832095
- calibration_repo: calibration-6228a29e06832095
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-6228a29e06832095
- expected_source_types: cve
- expected_verdicts: confirmed
- expected_binary_label: positive
- expected_evidence_terms: "src/purify.ts, dist/purify.js"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-6228a29e06832095
- title: GHSA-v8jm-5vwx-cfxm / CVE-2025-15599
- advisory ids: GHSA-v8jm-5vwx-cfxm, CVE-2025-15599
- weakness: xss
- severity: moderate
- affected language: JavaScript
- anchor locations: src/purify.ts:1385, dist/purify.js:1130
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration fixed: oss-dompurify-ghsa-v8jm-5vwx-cfxm"
- metadata:
- case_id: calibration-6a431e1b299f88f3
- suite: calibration-oss
- calibration_case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm
- calibration_variant: fixed
- repo: "cure53/DOMPurify"
- expected_binary_label: negative
- vars:
- case_id: calibration-6a431e1b299f88f3
- calibration_repo: calibration-6a431e1b299f88f3
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-6a431e1b299f88f3
- expected_source_types: cve
- expected_verdicts: not_actionable
- expected_binary_label: negative
- expected_evidence_terms: "src/purify.ts, dist/purify.js"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-6a431e1b299f88f3
- title: GHSA-v8jm-5vwx-cfxm / CVE-2025-15599
- advisory ids: GHSA-v8jm-5vwx-cfxm, CVE-2025-15599
- weakness: xss
- severity: moderate
- affected language: JavaScript
- anchor locations: src/purify.ts:1385, dist/purify.js:1130
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration vulnerable: oss-monetr-ghsa-29v9-frvh-c426"
- metadata:
- case_id: calibration-4ceeb56f4c1b0b23
- suite: calibration-oss
- calibration_case_id: oss-monetr-ghsa-29v9-frvh-c426
- calibration_variant: vulnerable
- repo: "monetr/monetr"
- expected_binary_label: positive
- vars:
- case_id: calibration-4ceeb56f4c1b0b23
- calibration_repo: calibration-4ceeb56f4c1b0b23
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-4ceeb56f4c1b0b23
- expected_source_types: cve
- expected_verdicts: confirmed
- expected_binary_label: positive
- expected_evidence_terms: "compose/monetr.yaml, server/controller/lunch_flow.go, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-4ceeb56f4c1b0b23
- title: GHSA-29v9-frvh-c426 / CVE-2026-41644
- advisory ids: GHSA-29v9-frvh-c426, CVE-2026-41644
- weakness: ssrf
- severity: high
- affected language: YAML/Go/TypeScript
- anchor locations: compose/monetr.yaml:34, server/controller/lunch_flow.go:100, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go:213
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
-
-- description: "calibration fixed: oss-monetr-ghsa-29v9-frvh-c426"
- metadata:
- case_id: calibration-e364368948a91896
- suite: calibration-oss
- calibration_case_id: oss-monetr-ghsa-29v9-frvh-c426
- calibration_variant: fixed
- repo: "monetr/monetr"
- expected_binary_label: negative
- vars:
- case_id: calibration-e364368948a91896
- calibration_repo: calibration-e364368948a91896
- calibration_repo_root: ""
- source_type_under_test: cve
- expected_ids: calibration-e364368948a91896
- expected_source_types: cve
- expected_verdicts: not_actionable
- expected_binary_label: negative
- expected_evidence_terms: "compose/monetr.yaml, server/controller/lunch_flow.go, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go"
- finding_input: |-
- Source type: cve
- 1. input_id: calibration-e364368948a91896
- title: GHSA-29v9-frvh-c426 / CVE-2026-41644
- advisory ids: GHSA-29v9-frvh-c426, CVE-2026-41644
- weakness: ssrf
- severity: high
- affected language: YAML/Go/TypeScript
- anchor locations: compose/monetr.yaml:34, server/controller/lunch_flow.go:100, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go:213
- eval_instructions: |-
- This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
- Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
- Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
- The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.