OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security
TypeScript ai-securityapplication-securityclicode-scanningcodexcodex-securitycybersecuritydevsecopsnodejsnpmopenaisdksecuritytypescriptvulnerability-scanning
fix(evals): hide expected answers and test the checkout's skill (#1054)
mldangelo-oai6c8ab22cb25d
diff --git a/evals/triage-finding/README.md b/evals/triage-finding/README.mdindex 3702f401f..ca020a8ca 100644--- a/evals/triage-finding/README.md+++ b/evals/triage-finding/README.md@@ -4,7 +4,7 @@ This Promptfoo suite verifies that `$codex-security:triage-finding` accepts theJSON result shape. It also covers bare skill invocation with no supplied finding,which should prompt the user for a finding in a supported format instead of returning triage JSON. GitHub intake cases cover repository-source selection, REST endpoints, and explicit Connector requests without querying live GitHub during the eval.-The suite uses the Promptfoo Codex SDK provider because it only needs final assistant output and deterministic assertions. The eval directory owns a small pinned pnpm environment so new cases can be added and run without a separate scratch setup.+The suite uses the Promptfoo Codex SDK provider because it only needs final assistant output and deterministic assertions. Codex runs from the checkout's plugin directory and reads its triage skill directly, with ambient plugins and memory disabled. The default suite can also read the synthetic fixtures; calibration cases can read their own hydrated checkout. Dataset labels, assertions, and calibration Git history stay outside those readable roots. The eval directory owns a small pinned pnpm environment so new cases can be added and run without a separate scratch setup.Use Node.js 22.22.0 or newer for the eval runner. Run these commands from the repository root to install dependencies under this eval directory.@@ -18,6 +18,16 @@ pnpm --dir evals/triage-finding run setupAfter editing the tooling, rerun `pnpm --dir sdk/typescript run build:evals`. It prepares the SDK and type-checks the eval sources. Node and Promptfoo execute the TypeScript sources directly; package commands enable type stripping for Node 22.13.+Before running model evals, build the checkout's policy helper (requires the [native build prerequisites](../../plugins/codex-security/native/README.md)):++```bash+pnpm --dir plugins/codex-security/mcp-app install --frozen-lockfile+node plugins/codex-security/mcp-app/scripts/build_native.mjs+node plugins/codex-security/mcp-app/scripts/build_mcp_app.mjs --output plugins/codex-security/mcp --native host+```++Config validation and deterministic tests do not need this helper build.+Validate the config:```bash@@ -59,7 +69,7 @@ paths and query parameters as JSON. Other intake cases use freeform answers.## Calibration Dataset-`datasets/triage-calibration-seed.json` is the first OSS-only calibration dataset for scaling beyond the synthetic fixture app. It contains public OSS vulnerable/fixed commit pairs. Each dataset variant becomes one Promptfoo test case in `tests/calibration-oss.yaml`, and each test points Codex at a pinned local checkout under `artifacts/calibration-repos/`.+`datasets/triage-calibration-seed.json` is the first OSS-only calibration dataset for scaling beyond the synthetic fixture app. It contains public OSS vulnerable/fixed commit pairs. Each dataset variant becomes one Promptfoo test case in `tests/calibration-oss.yaml`, and each test points Codex at a pinned local checkout under `artifacts/calibration-repos/`. Case IDs, finding IDs, and checkout directories use stable opaque names. Variant labels, fix references, and checkout commit IDs remain in harness metadata rather than the model prompt. Hydration keeps Git metadata in opaque sibling directories outside the model-readable checkout. Re-run `calibration:hydrate` to update older checkout layouts.ELI5: the dataset says "this exact old commit should be affected" and "this exact fixed commit should not be affected." The generator turns those rows into Promptfoo test prompts. The hydrator downloads the exact repo commits so Codex can inspect real code instead of synthetic snippets.diff --git a/evals/triage-finding/package.json b/evals/triage-finding/package.jsonindex e9c1be09a..bc8be363f 100644--- a/evals/triage-finding/package.json+++ b/evals/triage-finding/package.json@@ -14,19 +14,19 @@"test:calibration-generate": "node --experimental-strip-types scripts/test-generate-calibration-tests.mts","test:calibration-hydrate": "node --experimental-strip-types scripts/test-hydrate-calibration-repos.mts","test:dataset": "node --experimental-strip-types scripts/test-calibration-dataset.mts",- "test:deterministic": "node --experimental-strip-types scripts/test-app-surface-instructions.mts && node --experimental-strip-types scripts/test-triage-io.mts && node --experimental-strip-types scripts/test-intake-assertions.mts && node --experimental-strip-types scripts/test-calibration-dataset.mts && node --experimental-strip-types scripts/test-generate-calibration-tests.mts && node --experimental-strip-types scripts/test-calibration-evidence.mts && node --experimental-strip-types scripts/test-hydrate-calibration-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-lib.mts && node --experimental-strip-types sastbench/scripts/test-install-sastbench.mts && node --experimental-strip-types sastbench/scripts/test-hydrate-sastbench-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-result.mts && node --experimental-strip-types sastbench/scripts/test-promptfoo-native-harness.mts",+ "test:deterministic": "node --experimental-strip-types scripts/test-app-surface-instructions.mts && node --experimental-strip-types scripts/test-triage-io.mts && node --experimental-strip-types scripts/test-intake-assertions.mts && node --experimental-strip-types scripts/test-calibration-dataset.mts && node --experimental-strip-types scripts/test-generate-calibration-tests.mts && node --experimental-strip-types scripts/test-promptfoo-runtime.mts && node --experimental-strip-types scripts/test-calibration-evidence.mts && node --experimental-strip-types scripts/test-hydrate-calibration-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-lib.mts && node --experimental-strip-types sastbench/scripts/test-install-sastbench.mts && node --experimental-strip-types sastbench/scripts/test-hydrate-sastbench-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-result.mts && node --experimental-strip-types sastbench/scripts/test-promptfoo-native-harness.mts","test:instructions": "node --experimental-strip-types scripts/test-app-surface-instructions.mts","test:sastbench": "node --experimental-strip-types sastbench/scripts/test-sastbench-lib.mts && node --experimental-strip-types sastbench/scripts/test-install-sastbench.mts && node --experimental-strip-types sastbench/scripts/test-hydrate-sastbench-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-result.mts && node --experimental-strip-types sastbench/scripts/test-promptfoo-native-harness.mts","validate": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo validate config -c ./promptfooconfig.yaml","validate:calibration": "pnpm run calibration:generate && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo validate config -c ./promptfooconfig.calibration.yaml",- "validate:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types sastbench/scripts/run-sastbench-promptfoo.mts validate config -c ./sastbench/promptfooconfig.sastbench.yaml",- "validate:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types sastbench/scripts/run-sastbench-promptfoo.mts validate config -c ./sastbench/promptfooconfig.sastbench-sample.yaml",+ "validate:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo validate config -c ./sastbench/promptfooconfig.sastbench.yaml",+ "validate:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo validate config -c ./sastbench/promptfooconfig.sastbench-sample.yaml","eval": "mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./promptfooconfig.yaml -o ./artifacts/triage-finding-eval.json --no-cache --no-share","eval:smoke": "mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./promptfooconfig.yaml -o ./artifacts/triage-finding-eval-smoke.json --no-cache --no-share --filter-range 0:1","eval:calibration": "pnpm run calibration:generate && pnpm run calibration:hydrate && mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./promptfooconfig.calibration.yaml -o ./artifacts/triage-finding-calibration-eval.json --no-cache --no-share","eval:calibration:smoke": "node --experimental-strip-types scripts/generate-calibration-tests.mts --case oss-dompurify-ghsa-v8jm-5vwx-cfxm --variant vulnerable --output artifacts/calibration-smoke.yaml && node --experimental-strip-types scripts/hydrate-calibration-repos.mts --case oss-dompurify-ghsa-v8jm-5vwx-cfxm --variant vulnerable && mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./promptfooconfig.calibration-smoke.yaml -o ./artifacts/triage-finding-calibration-smoke.json --no-cache --no-share",- "eval:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types sastbench/scripts/run-sastbench-promptfoo.mts eval -c ./sastbench/promptfooconfig.sastbench.yaml --no-cache --no-share",- "eval:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types sastbench/scripts/run-sastbench-promptfoo.mts eval -c ./sastbench/promptfooconfig.sastbench-sample.yaml --no-cache --no-share --max-concurrency 32",+ "eval:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./sastbench/promptfooconfig.sastbench.yaml --no-cache --no-share",+ "eval:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./sastbench/promptfooconfig.sastbench-sample.yaml --no-cache --no-share --max-concurrency 32","pf:view": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo view"},"dependencies": {diff --git a/evals/triage-finding/promptfooconfig.calibration-smoke.yaml b/evals/triage-finding/promptfooconfig.calibration-smoke.yamlindex d02c6afa2..c0d70238b 100644--- a/evals/triage-finding/promptfooconfig.calibration-smoke.yaml+++ b/evals/triage-finding/promptfooconfig.calibration-smoke.yaml@@ -8,11 +8,26 @@ providers:- id: openai:codex-sdk:gpt-5.5label: "Codex SDK triage-finding calibration smoke"config:- working_dir: ../..- sandbox_mode: read-only+ working_dir: "{{triage_runtime_root}}"+ skip_git_repo_check: trueapproval_policy: never- network_access_enabled: false+ additional_directories:+ - "{{target_repo}}"+ - "{{triage_node_root}}"+ cli_config:+ default_permissions: triage_runtime_only+ permissions:+ triage_runtime_only:+ filesystem:+ ":minimal": read+ ":workspace_roots": read+ network:+ enabled: false+ features:+ memories: false+ plugins: falsecli_env:+ CODEX_MCP_NODE_PATH: "{{triage_node_path}}"HTTP_PROXY: "{{env.HTTP_PROXY}}"HTTPS_PROXY: "{{env.HTTPS_PROXY}}"ALL_PROXY: "{{env.ALL_PROXY}}"@@ -22,6 +37,8 @@ evaluateOptions:maxConcurrency: 1defaultTest:+ options:+ transformVars: file://./scripts/runtime-vars.mtsvars:invocation_preamble: "Use $codex-security:triage-finding to triage the supplied existing security finding."assert:diff --git a/evals/triage-finding/promptfooconfig.calibration.yaml b/evals/triage-finding/promptfooconfig.calibration.yamlindex acf6597d8..643fa6da4 100644--- a/evals/triage-finding/promptfooconfig.calibration.yaml+++ b/evals/triage-finding/promptfooconfig.calibration.yaml@@ -8,11 +8,26 @@ providers:- id: openai:codex-sdk:gpt-5.5label: "Codex SDK triage-finding calibration"config:- working_dir: ../..- sandbox_mode: read-only+ working_dir: "{{triage_runtime_root}}"+ skip_git_repo_check: trueapproval_policy: never- network_access_enabled: false+ additional_directories:+ - "{{target_repo}}"+ - "{{triage_node_root}}"+ cli_config:+ default_permissions: triage_runtime_only+ permissions:+ triage_runtime_only:+ filesystem:+ ":minimal": read+ ":workspace_roots": read+ network:+ enabled: false+ features:+ memories: false+ plugins: falsecli_env:+ CODEX_MCP_NODE_PATH: "{{triage_node_path}}"HTTP_PROXY: "{{env.HTTP_PROXY}}"HTTPS_PROXY: "{{env.HTTPS_PROXY}}"ALL_PROXY: "{{env.ALL_PROXY}}"@@ -22,6 +37,8 @@ evaluateOptions:maxConcurrency: 1defaultTest:+ options:+ transformVars: file://./scripts/runtime-vars.mtsvars:invocation_preamble: "Use $codex-security:triage-finding to triage the supplied existing security finding."assert:diff --git a/evals/triage-finding/promptfooconfig.yaml b/evals/triage-finding/promptfooconfig.yamlindex 29990c605..35be4ad6f 100644--- a/evals/triage-finding/promptfooconfig.yaml+++ b/evals/triage-finding/promptfooconfig.yaml@@ -8,11 +8,26 @@ providers:- id: openai:codex-sdk:gpt-5.5label: "Codex SDK triage-finding"config:- working_dir: ../..- sandbox_mode: read-only+ working_dir: "{{triage_runtime_root}}"+ skip_git_repo_check: trueapproval_policy: never- network_access_enabled: false+ additional_directories:+ - "{{triage_fixture_root}}"+ - "{{triage_node_root}}"+ cli_config:+ default_permissions: triage_runtime_only+ permissions:+ triage_runtime_only:+ filesystem:+ ":minimal": read+ ":workspace_roots": read+ network:+ enabled: false+ features:+ memories: false+ plugins: falsecli_env:+ CODEX_MCP_NODE_PATH: "{{triage_node_path}}"HTTP_PROXY: "{{env.HTTP_PROXY}}"HTTPS_PROXY: "{{env.HTTPS_PROXY}}"ALL_PROXY: "{{env.ALL_PROXY}}"@@ -22,6 +37,8 @@ evaluateOptions:maxConcurrency: 1defaultTest:+ options:+ transformVars: file://./scripts/runtime-vars.mtsvars:invocation_preamble: "Use $codex-security:triage-finding to triage the supplied existing security findings."eval_instructions: |-diff --git a/evals/triage-finding/prompts/triage-request.txt b/evals/triage-finding/prompts/triage-request.txtindex 40e1cbfec..22a7fc2fe 100644--- a/evals/triage-finding/prompts/triage-request.txt+++ b/evals/triage-finding/prompts/triage-request.txt@@ -1,3 +1,5 @@+Read and follow skills/triage-finding/SKILL.md from this checkout for the following request.+{{invocation_preamble}}Repository path to inspect:diff --git a/evals/triage-finding/sastbench/DESIGN.md b/evals/triage-finding/sastbench/DESIGN.mdindex 87b40b4f8..75fbcbd62 100644--- a/evals/triage-finding/sastbench/DESIGN.md+++ b/evals/triage-finding/sastbench/DESIGN.md@@ -24,7 +24,7 @@ The SastBench adapter owns only benchmark-specific behavior:The model-visible prompt contains the affected checkout, an opaque case ID, and the record's `to_analyzer` data. Ground truth is stored in Promptfoo test metadata. The upstream `finding_id`, `ground_truth`, `metadata.source`, and dataset class statistics are not prompt variables.-The Codex provider runs from a throwaway directory containing only the runtime skill files. A deny-by-default permission profile grants reads only to that directory, the hydrated target repos and their label-free Git cache, and Codex's minimal runtime paths; the label-bearing dataset and Promptfoo harness stay denied.+The Codex provider runs from the checkout's plugin directory. Its permission profile also grants reads to the selected target, its label-free Git cache, the Node runtime, and Codex's minimal runtime paths; the label-bearing dataset and Promptfoo harness stay denied.## Verdict mappingdiff --git a/evals/triage-finding/sastbench/README.md b/evals/triage-finding/sastbench/README.mdindex 6519a9f38..fdcfe1bcc 100644--- a/evals/triage-finding/sastbench/README.md+++ b/evals/triage-finding/sastbench/README.md@@ -2,17 +2,13 @@This Promptfoo lane measures how accurately the source version of `plugins/codex-security/skills/triage-finding/SKILL.md` triages supplied scanner findings from public SastBench v0.1. It does not measure repository-wide vulnerability discovery.-SastBench contains 2,737 findings: 299 labeled true positive and 2,438 labeled false positive. The JavaScript test generator exposes the scanner claim and the exact affected checkout to Codex while keeping SastBench's label-bearing `finding_id` and ground-truth label out of the rendered prompt. The eval runner stages the skill in a throwaway working directory and uses a deny-by-default Codex permission profile so only that directory, hydrated target repos, and their label-free Git cache plus minimal Codex runtime paths are readable to the model.+SastBench contains 2,737 findings: 299 labeled true positive and 2,438 labeled false positive. The JavaScript test generator exposes the scanner claim and the exact affected checkout to Codex while keeping SastBench's label-bearing `finding_id` and ground-truth label out of the rendered prompt. Codex reads the skill directly from the checkout's plugin directory. Its permission profile also allows the selected target, the label-free Git cache, and the Node runtime; the label-bearing eval harness remains outside those roots.Run commands from the repository root. The parent `evals/triage-finding/` directory owns the pinned Promptfoo and Codex SDK dependencies and the ignored `artifacts/` tree.## Prepare the benchmark-Install dependencies once:--```bash-pnpm --dir evals/triage-finding run setup-```+Follow the [parent setup instructions](../README.md) to install dependencies and build the checkout's policy helper before running model evals.Install the pinned SastBench checkout and hydrate all 275 repository revisions:diff --git a/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml b/evals/triage-finding/sastbench/promptfooconfig.sastbench.yamlindex b26ea2b25..295915146 100644--- a/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml+++ b/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml@@ -10,14 +10,13 @@ providers:- id: openai:codex-sdk:gpt-5.5label: "Codex SDK triage-finding SastBench (gpt-5.5)"config:- # The runner stages only the skill runtime here. The label-bearing- # dataset and Promptfoo harness stay outside this tree.- working_dir: "{{env.SASTBENCH_RUNTIME_ROOT}}"- # Target worktrees and their label-free Git cache are the only- # additional readable workspace roots.+ # The plugin checkout excludes the label-bearing dataset and eval harness.+ working_dir: "{{triage_runtime_root}}"+ # Read the target, its label-free Git cache, and the helper runtime.additional_directories:- - "{{env.SASTBENCH_TARGET_ROOT}}"- - "{{env.SASTBENCH_GIT_CACHE_ROOT}}"+ - "{{target_repo}}"+ - "{{sastbench_git_cache_root}}"+ - "{{triage_node_root}}"skip_git_repo_check: trueapproval_policy: never# Pin a value supported by the eval's Codex SDK instead of inheriting a@@ -36,16 +35,20 @@ providers:":workspace_roots": readnetwork:enabled: false- # The source skill is loaded explicitly from the staged runtime.+ # The source skill is loaded explicitly from the checkout.# Global plugin and memory state would add machine-specific context.features:memories: falseplugins: false+ cli_env:+ CODEX_MCP_NODE_PATH: "{{triage_node_path}}"extensions:- file://./assertions/sastbench-metrics.mts:afterEachdefaultTest:+ options:+ transformVars: process.mainModule.require(process.cwd() + '/scripts/runtime-vars.mts').default(vars)assert:- type: contains-jsonvalue: file://../schemas/triage-result-v0.schema.jsondiff --git a/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt b/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txtindex 88cf6eb7d..c33ce67fe 100644--- a/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt+++ b/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt@@ -1,4 +1,4 @@-Read and follow plugins/codex-security/skills/triage-finding/SKILL.md to triage the supplied scanner finding.+Read and follow skills/triage-finding/SKILL.md to triage the supplied scanner finding.Repository checkout to inspect:{{target_repo}}diff --git a/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.mts b/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.mtsdeleted file mode 100644index 3ab8483e4..000000000--- a/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.mts+++ /dev/null@@ -1,81 +0,0 @@-#!/usr/bin/env node-import childProcess from "node:child_process";-import fs from "node:fs";-import os from "node:os";-import path from "node:path";--import { EVAL_ROOT, DEFAULT_TARGET_ROOT } from "./sastbench-lib.mts";-import { DEFAULT_CACHE_ROOT } from "./hydrate-sastbench-repos.mts";-const PLUGIN_ROOT = path.resolve(- EVAL_ROOT,- "..",- "..",- "plugins",- "codex-security",-);-const TRIAGE_SKILL_ROOT = path.join(PLUGIN_ROOT, "skills", "triage-finding");-const PROMPTFOO_BIN = path.join(EVAL_ROOT, "node_modules", ".bin", "promptfoo");--function copyDirectory(- sourceRoot: string,- targetRoot: string,- excludedNames: Set<string> = new Set(),-) {- fs.mkdirSync(targetRoot, { recursive: true });- for (const entry of fs.readdirSync(sourceRoot, { withFileTypes: true })) {- if (excludedNames.has(entry.name)) {- continue;- }- const sourcePath = path.join(sourceRoot, entry.name);- const targetPath = path.join(targetRoot, entry.name);- if (entry.isDirectory()) {- copyDirectory(sourcePath, targetPath, excludedNames);- continue;- }- if (!entry.isFile()) {- throw new Error(- "Refusing to stage non-file runtime entry: " + sourcePath,- );- }- fs.copyFileSync(sourcePath, targetPath);- }-}--/**- * Give Codex a throwaway working directory that contains only the skill files- * it needs. The label-bearing dataset and Promptfoo harness stay in EVAL_ROOT.- */-const promptfooArgs = process.argv.slice(2);-if (promptfooArgs.length === 0) {- throw new Error("Expected Promptfoo arguments");-}-const runtimeRoot = fs.mkdtempSync(- path.join(os.tmpdir(), "codex-security-triage-finding-sastbench-"),-);-const stagedPluginRoot = path.join(runtimeRoot, "plugins", "codex-security");-copyDirectory(- TRIAGE_SKILL_ROOT,- path.join(stagedPluginRoot, "skills", "triage-finding"),- new Set(["evals"]),-);-for (const sharedDirectory of ["references", "schemas"]) {- const sourcePath = path.join(PLUGIN_ROOT, sharedDirectory);- if (fs.existsSync(sourcePath)) {- copyDirectory(sourcePath, path.join(stagedPluginRoot, sharedDirectory));- }-}-const env = {- ...process.env,- SASTBENCH_RUNTIME_ROOT: runtimeRoot,- SASTBENCH_TARGET_ROOT: DEFAULT_TARGET_ROOT,- SASTBENCH_GIT_CACHE_ROOT: DEFAULT_CACHE_ROOT,-};-try {- childProcess.execFileSync(PROMPTFOO_BIN, promptfooArgs, {- cwd: EVAL_ROOT,- env,- stdio: "inherit",- });-} finally {- fs.rmSync(runtimeRoot, { recursive: true, force: true });-}diff --git a/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts b/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mtsindex 8c201d53e..d077d2ad6 100644--- a/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts+++ b/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts@@ -17,7 +17,7 @@ const packageJson = JSON.parse(fs.readFileSync(path.join(evalRoot, "package.json"), "utf8"),);-assert.match(config, /working_dir:\s+"\{\{env\.SASTBENCH_RUNTIME_ROOT\}\}"/);+assert.match(config, /working_dir:\s+"\{\{triage_runtime_root\}\}"/);assert.match(config, /skip_git_repo_check:\s+true/);assert.doesNotMatch(config,@@ -25,7 +25,7 @@ assert.doesNotMatch();assert.match(config,- /additional_directories:\n\s+- "\{\{env\.SASTBENCH_TARGET_ROOT\}\}"\n\s+- "\{\{env\.SASTBENCH_GIT_CACHE_ROOT\}\}"/,+ /additional_directories:\n\s+- "\{\{target_repo\}\}"\n\s+- "\{\{sastbench_git_cache_root\}\}"/,);assert.doesNotMatch(config, /sandbox_mode:/);assert.doesNotMatch(config, /network_access_enabled:/);@@ -86,10 +86,7 @@ assert.match(assert.doesNotMatch(sampleConfig, /providers:/);assert.doesNotMatch(sampleConfig, /derivedMetrics:/);-assert.match(- packageJson.scripts["eval:sastbench"],- /run-sastbench-promptfoo\.mts eval/,-);+assert.match(packageJson.scripts["eval:sastbench"], /promptfoo eval/);assert.match(packageJson.scripts["eval:sastbench"], /--no-cache/);assert.match(packageJson.scripts["eval:sastbench"], /--no-share/);assert.doesNotMatch(packageJson.scripts["eval:sastbench"], /--filter-range/);@@ -102,11 +99,11 @@ assert.doesNotMatch();assert.match(packageJson.scripts["validate:sastbench:sample"],- /run-sastbench-promptfoo\.mts validate config -c .\/sastbench\/promptfooconfig\.sastbench-sample\.yaml/,+ /promptfoo validate config -c .\/sastbench\/promptfooconfig\.sastbench-sample\.yaml/,);assert.match(packageJson.scripts["eval:sastbench:sample"],- /run-sastbench-promptfoo\.mts eval -c .\/sastbench\/promptfooconfig\.sastbench-sample\.yaml/,+ /promptfoo eval -c .\/sastbench\/promptfooconfig\.sastbench-sample\.yaml/,);assert.match(packageJson.scripts["eval:sastbench:sample"], /--no-cache/);assert.match(packageJson.scripts["eval:sastbench:sample"], /--no-share/);@@ -117,7 +114,7 @@ assert.match(assert.equal("sastbench:generate" in packageJson.scripts, false);assert.equal(fs.existsSync(path.join(import.meta.dirname, "run-sastbench-promptfoo.mts")),- true,+ false,);assert.equal(fs.existsSync(diff --git a/evals/triage-finding/scripts/generate-calibration-tests.mts b/evals/triage-finding/scripts/generate-calibration-tests.mtsindex c2724e41c..33ae3f570 100644--- a/evals/triage-finding/scripts/generate-calibration-tests.mts+++ b/evals/triage-finding/scripts/generate-calibration-tests.mts@@ -2,6 +2,7 @@import type { CalibrationCase, CalibrationVariant } from "../types.ts";import fs from "node:fs";+import { hash } from "node:crypto";import path from "node:path";export const DEFAULT_DATASET = path.join(@@ -57,29 +58,28 @@ function indentedBlock(value: unknown) {.replace(/\n/g, "\n ")}`;}-function inputId(testCase: CalibrationCase, variant: CalibrationVariant) {- const base =- testCase.finding.input_id ||- testCase.finding.input_id_base ||- testCase.case_id;- return `${base}-${variant.variant_id}`;+export function variantCaseId(+ testCase: Pick<CalibrationCase, "case_id">,+ variant: CalibrationVariant,+) {+ return `calibration-${hash("sha256", `${testCase.case_id}\0${variant.checkout_ref}`).slice(0, 16)}`;}-function evidenceTerms(testCase: CalibrationCase, variant: CalibrationVariant) {+function evidenceTerms(testCase: CalibrationCase) {const terms = (testCase.finding.anchor_locations || []).map((location) => location.path,);- if (variant.variant_id === "fixed" && testCase.finding.fix_patch_ref) {- terms.push(variant.checkout_ref);- }return [...new Set(terms)];}-function findingInput(testCase: CalibrationCase, variant: CalibrationVariant) {+export function findingInput(+ testCase: CalibrationCase,+ variant: CalibrationVariant,+) {const finding = testCase.finding;const lines = [`Source type: ${testCase.source_type}`,- `1. input_id: ${inputId(testCase, variant)}`,+ `1. input_id: ${variantCaseId(testCase, variant)}`,` title: ${finding.title}`,];@@ -101,12 +101,6 @@ function findingInput(testCase: CalibrationCase, variant: CalibrationVariant) {.join(", ");lines.push(` anchor locations: ${anchors}`);}- if (finding.fix_patch_ref) {- lines.push(` fix evidence: ${finding.fix_patch_ref}`);- }- lines.push(- ` repository state: checked out at ${variant.checkout_ref}. Triage whether the original finding affects this exact state.`,- );return lines.join("\n");}@@ -116,8 +110,8 @@ function testYaml(variant: CalibrationVariant,repoRoot: string,) {- const generatedCaseId = `${testCase.case_id}-${variant.variant_id}`;- const terms = evidenceTerms(testCase, variant);+ const generatedCaseId = variantCaseId(testCase, variant);+ const terms = evidenceTerms(testCase);const lines = [`- description: ${quote(`calibration ${variant.variant_id}: ${testCase.case_id}`)}`," metadata:",@@ -129,9 +123,10 @@ function testYaml(` expected_binary_label: ${variant.expected_binary_label}`," vars:",` case_id: ${generatedCaseId}`,- ` target_repo: ${path.posix.join(repoRoot, testCase.case_id, variant.variant_id)}`,+ ` calibration_repo: ${generatedCaseId}`,+ ` calibration_repo_root: ${quote(repoRoot === DEFAULT_REPO_ROOT ? "" : path.resolve(repoRoot))}`,` source_type_under_test: ${testCase.source_type}`,- ` expected_ids: ${inputId(testCase, variant)}`,+ ` expected_ids: ${variantCaseId(testCase, variant)}`,` expected_source_types: ${testCase.source_type}`,` expected_verdicts: ${variant.expected_verdict}`,` expected_binary_label: ${variant.expected_binary_label}`,@@ -141,7 +136,7 @@ function testYaml(" eval_instructions: |-",indentedBlock(`This is an automated OSS calibration eval. Do not ask follow-up questions.-Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.`,diff --git a/evals/triage-finding/scripts/hydrate-calibration-repos.mts b/evals/triage-finding/scripts/hydrate-calibration-repos.mtsindex da15c1ddf..f69bd63cd 100644--- a/evals/triage-finding/scripts/hydrate-calibration-repos.mts+++ b/evals/triage-finding/scripts/hydrate-calibration-repos.mts@@ -4,6 +4,7 @@ import { runGit } from "../sastbench/scripts/hydrate-sastbench-repos.mts";import {DEFAULT_DATASET,selectedVariants,+ variantCaseId,} from "./generate-calibration-tests.mts";import fs from "node:fs";@@ -53,7 +54,7 @@ function gitOutput(args: string[], cwd: string) {}}-function ensureGitCheckout(job: (typeof jobs)[number]) {+function ensureGitCheckout(job: ReturnType<typeof plannedJobs>[number]) {fs.mkdirSync(path.dirname(job.targetDir), { recursive: true });if (!fs.existsSync(job.targetDir)) {@@ -81,6 +82,13 @@ function ensureGitCheckout(job: (typeof jobs)[number]) {}}+ if (fs.lstatSync(path.join(job.targetDir, ".git")).isDirectory()) {+ runGit(+ ["init", "--separate-git-dir", `${job.targetDir}.git`],+ job.targetDir,+ );+ }+const currentHead = gitOutput(["rev-parse", "HEAD"], job.targetDir);if (currentHead === job.checkoutRef) {return "already current";@@ -91,24 +99,33 @@ function ensureGitCheckout(job: (typeof jobs)[number]) {return "hydrated";}-const args = parseArgs(process.argv.slice(2));-const dataset = JSON.parse(fs.readFileSync(args.dataset, "utf8"));-const jobs = selectedVariants(dataset, args).map(({ testCase, variant }) => ({- repoUrl: testCase.repo.url,- checkoutRef: variant.checkout_ref,- targetDir: path.join(args.repoRoot, testCase.case_id, variant.variant_id),- label: `${testCase.case_id}/${variant.variant_id}`,-}));-const variantWord = jobs.length === 1 ? "variant" : "variants";-console.log(- `${args.dryRun ? "would hydrate" : "hydrating"} ${jobs.length} calibration ${variantWord}`,-);-for (const job of jobs) {- if (args.dryRun) {- console.log(`${job.label} <- ${job.repoUrl} @ ${job.checkoutRef}`);- console.log(` ${job.targetDir}`);- } else {- const status = ensureGitCheckout(job);- console.log(`${status}: ${job.label} @ ${job.checkoutRef}`);+export function plannedJobs(+ dataset: Parameters<typeof selectedVariants>[0],+ args: Parameters<typeof selectedVariants>[1] & { repoRoot: string },+) {+ return selectedVariants(dataset, args).map(({ testCase, variant }) => ({+ repoUrl: testCase.repo.url,+ checkoutRef: variant.checkout_ref,+ targetDir: path.join(args.repoRoot, variantCaseId(testCase, variant)),+ label: `${testCase.case_id}/${variant.variant_id}`,+ }));+}++if (import.meta.filename === fs.realpathSync(process.argv[1])) {+ const args = parseArgs(process.argv.slice(2));+ const dataset = JSON.parse(fs.readFileSync(args.dataset, "utf8"));+ const jobs = plannedJobs(dataset, args);+ const variantWord = jobs.length === 1 ? "variant" : "variants";+ console.log(+ `${args.dryRun ? "would hydrate" : "hydrating"} ${jobs.length} calibration ${variantWord}`,+ );+ for (const job of jobs) {+ if (args.dryRun) {+ console.log(`${job.label} <- ${job.repoUrl} @ ${job.checkoutRef}`);+ console.log(` ${job.targetDir}`);+ } else {+ const status = ensureGitCheckout(job);+ console.log(`${status}: ${job.label} @ ${job.checkoutRef}`);+ }}}diff --git a/evals/triage-finding/scripts/runtime-vars.mts b/evals/triage-finding/scripts/runtime-vars.mtsnew file mode 100644index 000000000..1cda16dd0--- /dev/null+++ b/evals/triage-finding/scripts/runtime-vars.mts@@ -0,0 +1,36 @@+import fs from "node:fs";+import path from "node:path";++const evalRoot = path.resolve(import.meta.dirname, "..");+const sourceRoot = path.resolve(evalRoot, "../..");++// Resolve paths for each execution, including retries of persisted evaluations.+export default (vars: Record<string, unknown>) => {+ const nodePath = fs.realpathSync(process.execPath);+ let targetRepo = vars.target_repo;+ if (vars.calibration_repo) {+ targetRepo = path.join(+ (vars.calibration_repo_root ||+ path.join(evalRoot, "artifacts", "calibration-repos")) as string,+ vars.calibration_repo as string,+ );+ } else if (+ typeof targetRepo === "string" &&+ targetRepo.startsWith("evals/triage-finding/fixtures/")+ ) {+ targetRepo = path.join(sourceRoot, targetRepo);+ }+ return {+ ...vars,+ target_repo: targetRepo,+ triage_runtime_root: path.join(sourceRoot, "plugins", "codex-security"),+ triage_fixture_root: path.join(evalRoot, "fixtures"),+ triage_node_path: nodePath,+ triage_node_root: path.dirname(nodePath),+ sastbench_git_cache_root: path.join(+ evalRoot,+ "artifacts",+ "sastbench-git-cache",+ ),+ };+};diff --git a/evals/triage-finding/scripts/test-generate-calibration-tests.mts b/evals/triage-finding/scripts/test-generate-calibration-tests.mtsindex f512906b6..9f8cc7d7e 100644--- a/evals/triage-finding/scripts/test-generate-calibration-tests.mts+++ b/evals/triage-finding/scripts/test-generate-calibration-tests.mts@@ -5,6 +5,9 @@ import { execFileSync } from "node:child_process";import fs from "node:fs";import os from "node:os";import path from "node:path";+import { findingInput, variantCaseId } from "./generate-calibration-tests.mts";+import type { CalibrationCase } from "../types.ts";+import { plannedJobs } from "./hydrate-calibration-repos.mts";const evalDir = path.resolve(import.meta.dirname, "..");const generator = path.join(@@ -15,8 +18,8 @@ const generator = path.join(const dataset = path.join(evalDir, "datasets", "triage-calibration-seed.json");const trackedTests = path.join(evalDir, "tests", "calibration-oss.yaml");-const tmpDir = fs.mkdtempSync(- path.join(os.tmpdir(), "triage-calibration-tests-"),+const tmpDir = fs.realpathSync(+ fs.mkdtempSync(path.join(os.tmpdir(), "triage-calibration-tests-")),);const generated = path.join(tmpDir, "calibration-oss.yaml");const generatedSmoke = path.join(tmpDir, "calibration-smoke.yaml");@@ -45,17 +48,40 @@ assert.equal(trackedYaml,"tracked calibration tests are stale; run calibration:generate",);-assert.match(- generatedYaml,- /case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerable/,-);-assert.match(generatedYaml, /case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed/);+const cases: CalibrationCase[] = JSON.parse(+ fs.readFileSync(dataset, "utf8"),+).cases;+const identities = new Set();+for (const testCase of cases) {+ for (const variant of testCase.variants) {+ const caseId = variantCaseId(testCase, variant);+ assert.match(caseId, /^calibration-[a-f0-9]{16}$/);+ assert.ok(!identities.has(caseId));+ identities.add(caseId);+ assert.equal(+ variantCaseId(testCase, {+ ...variant,+ variant_id: "renamed",+ expected_verdict: "needs_review",+ }),+ caseId,+ );+ const input = findingInput(testCase, variant);+ assert.ok(!input.includes(variant.checkout_ref));+ assert.ok(!input.includes(testCase.finding.fix_patch_ref!));+ assert.doesNotMatch(input, /-(?:vulnerable|fixed)\b/);+ const repoRoot = path.join(tmpDir, "targets");+ const [job] = plannedJobs(+ { cases: [{ ...testCase, variants: [variant] }] },+ { repoRoot },+ );+ assert.equal(job.targetDir, path.join(repoRoot, caseId));+ assert.ok(generatedYaml.includes(` calibration_repo: ${caseId}`));+ }+}assert.match(generatedYaml, /expected_verdicts: confirmed/);assert.match(generatedYaml, /expected_verdicts: not_actionable/);-assert.match(- generatedYaml,- /target_repo: evals\/triage-finding\/artifacts\/calibration-repos\//,-);+assert.match(generatedYaml, /calibration_repo_root: ""/);assert.doesNotMatch(generatedYaml,/expected_evidence_terms:\n\s+- /,@@ -75,23 +101,44 @@ execFileSync("oss-dompurify-ghsa-v8jm-5vwx-cfxm","--variant","vulnerable",+ "--repo-root",+ "custom targets # space",],{- cwd: evalDir,+ cwd: tmpDir,stdio: "pipe",},);const smokeYaml = fs.readFileSync(generatedSmoke, "utf8");assert.equal((smokeYaml.match(/^- description:/gm) || []).length, 1);-assert.match(- smokeYaml,- /case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-vulnerable/,+assert.ok(+ smokeYaml.includes(+ `calibration_repo_root: ${JSON.stringify(path.join(tmpDir, "custom targets # space"))}`,+ ),);-assert.doesNotMatch(- smokeYaml,- /case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-fixed/,+const smokeCase = cases.find(+ (testCase) => testCase.case_id === "oss-dompurify-ghsa-v8jm-5vwx-cfxm",+)!;+assert.ok(+ smokeYaml.includes(+ `case_id: ${variantCaseId(+ smokeCase,+ smokeCase.variants.find(+ (variant) => variant.variant_id === "vulnerable",+ )!,+ )}`,+ ),+);+assert.ok(+ !smokeYaml.includes(+ `case_id: ${variantCaseId(+ smokeCase,+ smokeCase.variants.find((variant) => variant.variant_id === "fixed")!,+ )}`,+ ),);+fs.rmSync(tmpDir, { recursive: true, force: true });console.log("calibration test generation matches tracked YAML and supports filtered smoke output",diff --git a/evals/triage-finding/scripts/test-hydrate-calibration-repos.mts b/evals/triage-finding/scripts/test-hydrate-calibration-repos.mtsindex dcf6d6532..429b3b6b1 100644--- a/evals/triage-finding/scripts/test-hydrate-calibration-repos.mts+++ b/evals/triage-finding/scripts/test-hydrate-calibration-repos.mts@@ -1,9 +1,11 @@#!/usr/bin/env node+import assert from "node:assert/strict";import childProcess from "node:child_process";-import path from "node:path";import fs from "node:fs";import os from "node:os";+import path from "node:path";+import { plannedJobs } from "./hydrate-calibration-repos.mts";const evalDir = path.join(import.meta.dirname, "..");const scriptPath = path.join(@@ -89,20 +91,20 @@ try {const originalOrigin = git(parent, "remote", "get-url", "origin");const expectedHead = git(source, "rev-parse", "HEAD");const dataset = path.join(temporary, "dataset.json");- fs.writeFileSync(- dataset,- JSON.stringify({- cases: [- {- case_id: "case",- repo: { url: source },- variants: [{ variant_id: "variant", checkout_ref: expectedHead }],- },- ],- }),- );- const repoRoot = path.join(parent, "checkouts");- const target = path.join(repoRoot, "case", "variant");+ const fixture = {+ cases: [+ {+ case_id: "case",+ repo: { url: source },+ variants: [{ variant_id: "variant", checkout_ref: expectedHead }],+ },+ ],+ };+ fs.writeFileSync(dataset, JSON.stringify(fixture));+ const repoRoot = path.join(parent, "checkout fixtures");+ const target = plannedJobs(fixture as Parameters<typeof plannedJobs>[0], {+ repoRoot,+ })[0].targetDir;fs.mkdirSync(path.join(target, ".git"), { recursive: true });const rejected = childProcess.spawnSync(process.execPath,@@ -128,6 +130,11 @@ try {/hydrated: case\/variant/,);assert.equal(git(target, "rev-parse", "HEAD"), expectedHead);+ assert.ok(fs.lstatSync(path.join(target, ".git")).isFile());+ assert.equal(+ fs.realpathSync(git(target, "rev-parse", "--absolute-git-dir")),+ fs.realpathSync(`${target}.git`),+ );assert.equal(git(parent, "remote", "get-url", "origin"), originalOrigin);assert.equal(git(parent, "rev-parse", "HEAD"), originalHead);assert.match(@@ -145,6 +152,50 @@ try {runHydrator(["--dataset", dataset, "--repo-root", alias]),/already current/,);++ const legacyRoot = path.join(parent, "legacy checkouts");+ const legacyTarget = path.join(legacyRoot, path.basename(target));+ fs.mkdirSync(legacyRoot);+ git(parent, "clone", source, legacyTarget);+ assert.ok(fs.lstatSync(path.join(legacyTarget, ".git")).isDirectory());+ const legacyAlias = path.join(temporary, "legacy checkout alias");+ fs.symlinkSync(+ legacyRoot,+ legacyAlias,+ process.platform === "win32" ? "junction" : "dir",+ );+ assert.match(+ runHydrator(["--dataset", dataset, "--repo-root", legacyAlias]),+ /already current/,+ );+ assert.ok(fs.lstatSync(path.join(legacyTarget, ".git")).isFile());+ assert.equal(git(legacyTarget, "rev-parse", "HEAD"), expectedHead);+ assert.equal(git(legacyTarget, "remote", "get-url", "origin"), source);+ assert.equal(+ fs.readFileSync(path.join(legacyTarget, "fixture.txt"), "utf8"),+ "source",+ );++ const worktreeRoot = path.join(parent, "linked checkouts");+ const worktreeTarget = path.join(worktreeRoot, path.basename(target));+ git(+ legacyTarget,+ "worktree",+ "add",+ "--detach",+ worktreeTarget,+ expectedHead,+ );+ const gitfile = fs.readFileSync(path.join(worktreeTarget, ".git"), "utf8");+ assert.match(+ runHydrator(["--dataset", dataset, "--repo-root", worktreeRoot]),+ /already current/,+ );+ assert.equal(+ fs.readFileSync(path.join(worktreeTarget, ".git"), "utf8"),+ gitfile,+ );+ assert.equal(git(worktreeTarget, "rev-parse", "HEAD"), expectedHead);} finally {fs.rmSync(temporary, { recursive: true, force: true });}diff --git a/evals/triage-finding/scripts/test-promptfoo-runtime.mts b/evals/triage-finding/scripts/test-promptfoo-runtime.mtsnew file mode 100644index 000000000..9b82ca927--- /dev/null+++ b/evals/triage-finding/scripts/test-promptfoo-runtime.mts@@ -0,0 +1,80 @@+#!/usr/bin/env node++import assert from "node:assert/strict";+import fs from "node:fs";+import path from "node:path";+import runtimeVars from "./runtime-vars.mts";+import { DEFAULT_CACHE_ROOT } from "../sastbench/scripts/hydrate-sastbench-repos.mts";++const evalRoot = path.resolve(import.meta.dirname, "..");+const sourceRoot = path.resolve(evalRoot, "../..");+const pluginRoot = path.join(sourceRoot, "plugins", "codex-security");+const vars = runtimeVars({});+assert.equal(vars.triage_runtime_root, pluginRoot);+assert.equal(vars.triage_fixture_root, path.join(evalRoot, "fixtures"));+assert.equal(vars.triage_node_path, fs.realpathSync(process.execPath));+assert.equal(vars.triage_node_root, path.dirname(vars.triage_node_path));+assert.equal(vars.sastbench_git_cache_root, DEFAULT_CACHE_ROOT);+assert.ok(+ fs.existsSync(path.join(pluginRoot, "skills/triage-finding/SKILL.md")),+);++for (const fixture of ["repo", "policy-repo"]) {+ const target = runtimeVars({+ target_repo: `evals/triage-finding/fixtures/${fixture}`,+ }).target_repo;+ assert.equal(target, path.join(evalRoot, "fixtures", fixture));+ assert.ok(fs.existsSync(target as string));+}+for (const target of [+ path.join(evalRoot, "artifacts", "sastbench-targets", "opaque-target"),+ "https://github.com/example/repository",+ "No explicit repository supplied by the user.",+]) {+ assert.equal(runtimeVars({ target_repo: target }).target_repo, target);+}++const calibrationRepo = "calibration-0123456789abcdef";+for (const customRoot of ["", path.join(evalRoot, "custom targets # space")]) {+ const replayed = runtimeVars({+ calibration_repo: calibrationRepo,+ calibration_repo_root: customRoot,+ target_repo: "old checkout",+ triage_runtime_root: "old plugin",+ });+ assert.equal(replayed.triage_runtime_root, pluginRoot);+ assert.equal(+ replayed.target_repo,+ path.join(+ customRoot || path.join(evalRoot, "artifacts", "calibration-repos"),+ calibrationRepo,+ ),+ );+}++for (const name of [+ "promptfooconfig.yaml",+ "promptfooconfig.calibration.yaml",+ "promptfooconfig.calibration-smoke.yaml",+ "sastbench/promptfooconfig.sastbench.yaml",+]) {+ const config = fs.readFileSync(path.join(evalRoot, name), "utf8");+ assert.match(config, /working_dir: "\{\{triage_runtime_root\}\}"/);+ assert.match(config, /":minimal": read\n\s+":workspace_roots": read/);+ assert.match(config, /plugins: false/);+ assert.match(config, /memories: false/);+ assert.ok(config.includes('"{{triage_node_root}}"'));+ assert.ok(+ config.includes(+ name === "promptfooconfig.yaml"+ ? '"{{triage_fixture_root}}"'+ : '"{{target_repo}}"',+ ),+ );+}+assert.ok(+ fs+ .readFileSync(path.join(evalRoot, "prompts/triage-request.txt"), "utf8")+ .startsWith("Read and follow skills/triage-finding/SKILL.md"),+);+console.log("triage Promptfoo checkout path tests passed");diff --git a/evals/triage-finding/tests/calibration-oss.yaml b/evals/triage-finding/tests/calibration-oss.yamlindex 89e3ab0c1..29f004e82 100644--- a/evals/triage-finding/tests/calibration-oss.yaml+++ b/evals/triage-finding/tests/calibration-oss.yaml@@ -1,559 +1,543 @@- description: "calibration vulnerable: oss-mantisbt-ghsa-73vx-49mv-v8w5"metadata:- case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerable+ case_id: calibration-393a6e303af5b190suite: calibration-osscalibration_case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5calibration_variant: vulnerablerepo: "mantisbt/mantisbt"expected_binary_label: positivevars:- case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerable- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/vulnerable+ case_id: calibration-393a6e303af5b190+ calibration_repo: calibration-393a6e303af5b190+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-73vx-49mv-v8w5-000-vulnerable+ expected_ids: calibration-393a6e303af5b190expected_source_types: cveexpected_verdicts: confirmedexpected_binary_label: positiveexpected_evidence_terms: "core/classes/IssueTagTimelineEvent.class.php"finding_input: |-Source type: cve- 1. input_id: GHSA-73vx-49mv-v8w5-000-vulnerable+ 1. input_id: calibration-393a6e303af5b190title: GHSA-73vx-49mv-v8w5 / CVE-2026-33548advisory ids: GHSA-73vx-49mv-v8w5, CVE-2026-33548weakness: xssseverity: highaffected language: PHPanchor locations: core/classes/IssueTagTimelineEvent.class.php:61- fix evidence: https://github.com/mantisbt/mantisbt/commit/f32787c14d4518476fe7f05f992dbfe6eaccd815- repository state: checked out at 80990f43153167c73f11eb4b2bc7108d0c3d6b46. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration fixed: oss-mantisbt-ghsa-73vx-49mv-v8w5"metadata:- case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed+ case_id: calibration-0310e3e9455d0db5suite: calibration-osscalibration_case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5calibration_variant: fixedrepo: "mantisbt/mantisbt"expected_binary_label: negativevars:- case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/fixed+ case_id: calibration-0310e3e9455d0db5+ calibration_repo: calibration-0310e3e9455d0db5+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-73vx-49mv-v8w5-000-fixed+ expected_ids: calibration-0310e3e9455d0db5expected_source_types: cveexpected_verdicts: not_actionableexpected_binary_label: negative- expected_evidence_terms: "core/classes/IssueTagTimelineEvent.class.php, f32787c14d4518476fe7f05f992dbfe6eaccd815"+ expected_evidence_terms: "core/classes/IssueTagTimelineEvent.class.php"finding_input: |-Source type: cve- 1. input_id: GHSA-73vx-49mv-v8w5-000-fixed+ 1. input_id: calibration-0310e3e9455d0db5title: GHSA-73vx-49mv-v8w5 / CVE-2026-33548advisory ids: GHSA-73vx-49mv-v8w5, CVE-2026-33548weakness: xssseverity: highaffected language: PHPanchor locations: core/classes/IssueTagTimelineEvent.class.php:61- fix evidence: https://github.com/mantisbt/mantisbt/commit/f32787c14d4518476fe7f05f992dbfe6eaccd815- repository state: checked out at f32787c14d4518476fe7f05f992dbfe6eaccd815. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration vulnerable: oss-imagemagick-ghsa-qpgx-jfcq-r59f"metadata:- case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-vulnerable+ case_id: calibration-89e4ec9f158dc8b9suite: calibration-osscalibration_case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59fcalibration_variant: vulnerablerepo: "ImageMagick/ImageMagick"expected_binary_label: positivevars:- case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-vulnerable- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/vulnerable+ case_id: calibration-89e4ec9f158dc8b9+ calibration_repo: calibration-89e4ec9f158dc8b9+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-qpgx-jfcq-r59f-000-vulnerable+ expected_ids: calibration-89e4ec9f158dc8b9expected_source_types: cveexpected_verdicts: confirmedexpected_binary_label: positiveexpected_evidence_terms: "MagickCore/visual-effects.c"finding_input: |-Source type: cve- 1. input_id: GHSA-qpgx-jfcq-r59f-000-vulnerable+ 1. input_id: calibration-89e4ec9f158dc8b9title: GHSA-qpgx-jfcq-r59f / CVE-2026-27798advisory ids: GHSA-qpgx-jfcq-r59f, CVE-2026-27798weakness: memory-corruptionseverity: mediumaffected language: C/C++anchor locations: MagickCore/visual-effects.c:3566- fix evidence: https://github.com/ImageMagick/ImageMagick/commit/0377e60b3c0d766bd7271221c95d9ee54f6a3738- repository state: checked out at 83847c348aafccbbea89ec8ed44382598aea3f98. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration fixed: oss-imagemagick-ghsa-qpgx-jfcq-r59f"metadata:- case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-fixed+ case_id: calibration-11948efe569f3fb9suite: calibration-osscalibration_case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59fcalibration_variant: fixedrepo: "ImageMagick/ImageMagick"expected_binary_label: negativevars:- case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-fixed- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/fixed+ case_id: calibration-11948efe569f3fb9+ calibration_repo: calibration-11948efe569f3fb9+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-qpgx-jfcq-r59f-000-fixed+ expected_ids: calibration-11948efe569f3fb9expected_source_types: cveexpected_verdicts: not_actionableexpected_binary_label: negative- expected_evidence_terms: "MagickCore/visual-effects.c, 0377e60b3c0d766bd7271221c95d9ee54f6a3738"+ expected_evidence_terms: "MagickCore/visual-effects.c"finding_input: |-Source type: cve- 1. input_id: GHSA-qpgx-jfcq-r59f-000-fixed+ 1. input_id: calibration-11948efe569f3fb9title: GHSA-qpgx-jfcq-r59f / CVE-2026-27798advisory ids: GHSA-qpgx-jfcq-r59f, CVE-2026-27798weakness: memory-corruptionseverity: mediumaffected language: C/C++anchor locations: MagickCore/visual-effects.c:3566- fix evidence: https://github.com/ImageMagick/ImageMagick/commit/0377e60b3c0d766bd7271221c95d9ee54f6a3738- repository state: checked out at 0377e60b3c0d766bd7271221c95d9ee54f6a3738. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration vulnerable: oss-scitokens-ghsa-rh5m-2482-966c"metadata:- case_id: oss-scitokens-ghsa-rh5m-2482-966c-vulnerable+ case_id: calibration-576c7d9cbbc7c018suite: calibration-osscalibration_case_id: oss-scitokens-ghsa-rh5m-2482-966ccalibration_variant: vulnerablerepo: "scitokens/scitokens"expected_binary_label: positivevars:- case_id: oss-scitokens-ghsa-rh5m-2482-966c-vulnerable- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/vulnerable+ case_id: calibration-576c7d9cbbc7c018+ calibration_repo: calibration-576c7d9cbbc7c018+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-rh5m-2482-966c-000-vulnerable+ expected_ids: calibration-576c7d9cbbc7c018expected_source_types: cveexpected_verdicts: confirmedexpected_binary_label: positiveexpected_evidence_terms: "src/scitokens/utils/keycache.py"finding_input: |-Source type: cve- 1. input_id: GHSA-rh5m-2482-966c-000-vulnerable+ 1. input_id: calibration-576c7d9cbbc7c018title: GHSA-rh5m-2482-966c / CVE-2026-32714advisory ids: GHSA-rh5m-2482-966c, CVE-2026-32714weakness: sql-injectionseverity: criticalaffected language: Pythonanchor locations: src/scitokens/utils/keycache.py:80- fix evidence: https://github.com/scitokens/scitokens/commit/3dba108853f2f4a6c0f2325c03779bf083c41cf2- repository state: checked out at 66b075c35da18f8364276553513a53c596b4b8e3. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration fixed: oss-scitokens-ghsa-rh5m-2482-966c"metadata:- case_id: oss-scitokens-ghsa-rh5m-2482-966c-fixed+ case_id: calibration-3aaf9d4f40ef518csuite: calibration-osscalibration_case_id: oss-scitokens-ghsa-rh5m-2482-966ccalibration_variant: fixedrepo: "scitokens/scitokens"expected_binary_label: negativevars:- case_id: oss-scitokens-ghsa-rh5m-2482-966c-fixed- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/fixed+ case_id: calibration-3aaf9d4f40ef518c+ calibration_repo: calibration-3aaf9d4f40ef518c+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-rh5m-2482-966c-000-fixed+ expected_ids: calibration-3aaf9d4f40ef518cexpected_source_types: cveexpected_verdicts: not_actionableexpected_binary_label: negative- expected_evidence_terms: "src/scitokens/utils/keycache.py, 3dba108853f2f4a6c0f2325c03779bf083c41cf2"+ expected_evidence_terms: "src/scitokens/utils/keycache.py"finding_input: |-Source type: cve- 1. input_id: GHSA-rh5m-2482-966c-000-fixed+ 1. input_id: calibration-3aaf9d4f40ef518ctitle: GHSA-rh5m-2482-966c / CVE-2026-32714advisory ids: GHSA-rh5m-2482-966c, CVE-2026-32714weakness: sql-injectionseverity: criticalaffected language: Pythonanchor locations: src/scitokens/utils/keycache.py:80- fix evidence: https://github.com/scitokens/scitokens/commit/3dba108853f2f4a6c0f2325c03779bf083c41cf2- repository state: checked out at 3dba108853f2f4a6c0f2325c03779bf083c41cf2. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration vulnerable: oss-emissary-ghsa-3p24-9x7v-7789"metadata:- case_id: oss-emissary-ghsa-3p24-9x7v-7789-vulnerable+ case_id: calibration-1561f6cf0aa4fa71suite: calibration-osscalibration_case_id: oss-emissary-ghsa-3p24-9x7v-7789calibration_variant: vulnerablerepo: "NationalSecurityAgency/emissary"expected_binary_label: positivevars:- case_id: oss-emissary-ghsa-3p24-9x7v-7789-vulnerable- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/vulnerable+ case_id: calibration-1561f6cf0aa4fa71+ calibration_repo: calibration-1561f6cf0aa4fa71+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-3p24-9x7v-7789-000-vulnerable+ expected_ids: calibration-1561f6cf0aa4fa71expected_source_types: cveexpected_verdicts: confirmedexpected_binary_label: positiveexpected_evidence_terms: "src/main/java/emissary/util/shell/Executrix.java"finding_input: |-Source type: cve- 1. input_id: GHSA-3p24-9x7v-7789-000-vulnerable+ 1. input_id: calibration-1561f6cf0aa4fa71title: GHSA-3p24-9x7v-7789 / CVE-2026-35582advisory ids: GHSA-3p24-9x7v-7789, CVE-2026-35582weakness: command-injectionseverity: highaffected language: Javaanchor locations: src/main/java/emissary/util/shell/Executrix.java:77- fix evidence: https://github.com/NationalSecurityAgency/emissary/commit/1faf33f2494c0128f250d7d2e8f2da99bbd32ae8- repository state: checked out at 082b263a19532454b24dfbed1605311854ea778f. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration fixed: oss-emissary-ghsa-3p24-9x7v-7789"metadata:- case_id: oss-emissary-ghsa-3p24-9x7v-7789-fixed+ case_id: calibration-ca1bddda373398e8suite: calibration-osscalibration_case_id: oss-emissary-ghsa-3p24-9x7v-7789calibration_variant: fixedrepo: "NationalSecurityAgency/emissary"expected_binary_label: negativevars:- case_id: oss-emissary-ghsa-3p24-9x7v-7789-fixed- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/fixed+ case_id: calibration-ca1bddda373398e8+ calibration_repo: calibration-ca1bddda373398e8+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-3p24-9x7v-7789-000-fixed+ expected_ids: calibration-ca1bddda373398e8expected_source_types: cveexpected_verdicts: not_actionableexpected_binary_label: negative- expected_evidence_terms: "src/main/java/emissary/util/shell/Executrix.java, 1faf33f2494c0128f250d7d2e8f2da99bbd32ae8"+ expected_evidence_terms: "src/main/java/emissary/util/shell/Executrix.java"finding_input: |-Source type: cve- 1. input_id: GHSA-3p24-9x7v-7789-000-fixed+ 1. input_id: calibration-ca1bddda373398e8title: GHSA-3p24-9x7v-7789 / CVE-2026-35582advisory ids: GHSA-3p24-9x7v-7789, CVE-2026-35582weakness: command-injectionseverity: highaffected language: Javaanchor locations: src/main/java/emissary/util/shell/Executrix.java:77- fix evidence: https://github.com/NationalSecurityAgency/emissary/commit/1faf33f2494c0128f250d7d2e8f2da99bbd32ae8- repository state: checked out at 1faf33f2494c0128f250d7d2e8f2da99bbd32ae8. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration vulnerable: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5"metadata:- case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-vulnerable+ case_id: calibration-2cd95700e4d0032dsuite: calibration-osscalibration_case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5calibration_variant: vulnerablerepo: "free5gc/pcf"expected_binary_label: positivevars:- case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-vulnerable- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/vulnerable+ case_id: calibration-2cd95700e4d0032d+ calibration_repo: calibration-2cd95700e4d0032d+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-6RGM-GR97-X3J5-vulnerable+ expected_ids: calibration-2cd95700e4d0032dexpected_source_types: cveexpected_verdicts: confirmedexpected_binary_label: positiveexpected_evidence_terms: "internal/sbi/server.go"finding_input: |-Source type: cve- 1. input_id: GHSA-6RGM-GR97-X3J5-vulnerable+ 1. input_id: calibration-2cd95700e4d0032dtitle: GHSA-6RGM-GR97-X3J5 / CVE-2026-42083advisory ids: GHSA-6RGM-GR97-X3J5, CVE-2026-42083weakness: authzseverity: highaffected language: Goanchor locations: internal/sbi/server.go:72- fix evidence: https://github.com/free5gc/pcf/commit/8c4d457cdf58bb239ee30e88c56b370b22073964- repository state: checked out at 508d70b8527a6c8c923179dad450ea01e16b6aeb. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration fixed: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5"metadata:- case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-fixed+ case_id: calibration-1b89c72834a4a631suite: calibration-osscalibration_case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5calibration_variant: fixedrepo: "free5gc/pcf"expected_binary_label: negativevars:- case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-fixed- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/fixed+ case_id: calibration-1b89c72834a4a631+ calibration_repo: calibration-1b89c72834a4a631+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-6RGM-GR97-X3J5-fixed+ expected_ids: calibration-1b89c72834a4a631expected_source_types: cveexpected_verdicts: not_actionableexpected_binary_label: negative- expected_evidence_terms: "internal/sbi/server.go, 8c4d457cdf58bb239ee30e88c56b370b22073964"+ expected_evidence_terms: "internal/sbi/server.go"finding_input: |-Source type: cve- 1. input_id: GHSA-6RGM-GR97-X3J5-fixed+ 1. input_id: calibration-1b89c72834a4a631title: GHSA-6RGM-GR97-X3J5 / CVE-2026-42083advisory ids: GHSA-6RGM-GR97-X3J5, CVE-2026-42083weakness: authzseverity: highaffected language: Goanchor locations: internal/sbi/server.go:72- fix evidence: https://github.com/free5gc/pcf/commit/8c4d457cdf58bb239ee30e88c56b370b22073964- repository state: checked out at 8c4d457cdf58bb239ee30e88c56b370b22073964. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration vulnerable: oss-actualbudget-ghsa-27vg-33gh-4hwg"metadata:- case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-vulnerable+ case_id: calibration-e4d2cc784a81e08csuite: calibration-osscalibration_case_id: oss-actualbudget-ghsa-27vg-33gh-4hwgcalibration_variant: vulnerablerepo: "actualbudget/actual"expected_binary_label: positivevars:- case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-vulnerable- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/vulnerable+ case_id: calibration-e4d2cc784a81e08c+ calibration_repo: calibration-e4d2cc784a81e08c+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-27vg-33gh-4hwg-000-vulnerable+ expected_ids: calibration-e4d2cc784a81e08cexpected_source_types: cveexpected_verdicts: confirmedexpected_binary_label: positiveexpected_evidence_terms: "packages/sync-server/src/app-sync.ts, packages/sync-server/src/app-sync.test.ts"finding_input: |-Source type: cve- 1. input_id: GHSA-27vg-33gh-4hwg-000-vulnerable+ 1. input_id: calibration-e4d2cc784a81e08ctitle: GHSA-27vg-33gh-4hwg / CVE-2026-3089advisory ids: GHSA-27vg-33gh-4hwg, CVE-2026-3089weakness: path-traversalseverity: mediumaffected language: TypeScriptanchor locations: packages/sync-server/src/app-sync.ts:49, packages/sync-server/src/app-sync.test.ts:366- fix evidence: https://github.com/actualbudget/actual/commit/18072e1d8b5281db43ded8b21433ee177bae9dfa- repository state: checked out at a1e0b3f45d36232a0f4319a60cf264912edac48f. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration fixed: oss-actualbudget-ghsa-27vg-33gh-4hwg"metadata:- case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-fixed+ case_id: calibration-b4528e41b5de5a74suite: calibration-osscalibration_case_id: oss-actualbudget-ghsa-27vg-33gh-4hwgcalibration_variant: fixedrepo: "actualbudget/actual"expected_binary_label: negativevars:- case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-fixed- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/fixed+ case_id: calibration-b4528e41b5de5a74+ calibration_repo: calibration-b4528e41b5de5a74+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-27vg-33gh-4hwg-000-fixed+ expected_ids: calibration-b4528e41b5de5a74expected_source_types: cveexpected_verdicts: not_actionableexpected_binary_label: negative- expected_evidence_terms: "packages/sync-server/src/app-sync.ts, packages/sync-server/src/app-sync.test.ts, 18072e1d8b5281db43ded8b21433ee177bae9dfa"+ expected_evidence_terms: "packages/sync-server/src/app-sync.ts, packages/sync-server/src/app-sync.test.ts"finding_input: |-Source type: cve- 1. input_id: GHSA-27vg-33gh-4hwg-000-fixed+ 1. input_id: calibration-b4528e41b5de5a74title: GHSA-27vg-33gh-4hwg / CVE-2026-3089advisory ids: GHSA-27vg-33gh-4hwg, CVE-2026-3089weakness: path-traversalseverity: mediumaffected language: TypeScriptanchor locations: packages/sync-server/src/app-sync.ts:49, packages/sync-server/src/app-sync.test.ts:366- fix evidence: https://github.com/actualbudget/actual/commit/18072e1d8b5281db43ded8b21433ee177bae9dfa- repository state: checked out at 18072e1d8b5281db43ded8b21433ee177bae9dfa. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration vulnerable: oss-dompurify-ghsa-v8jm-5vwx-cfxm"metadata:- case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-vulnerable+ case_id: calibration-6228a29e06832095suite: calibration-osscalibration_case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxmcalibration_variant: vulnerablerepo: "cure53/DOMPurify"expected_binary_label: positivevars:- case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-vulnerable- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/vulnerable+ case_id: calibration-6228a29e06832095+ calibration_repo: calibration-6228a29e06832095+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-v8jm-5vwx-cfxm-gap-fill-000-vulnerable+ expected_ids: calibration-6228a29e06832095expected_source_types: cveexpected_verdicts: confirmedexpected_binary_label: positiveexpected_evidence_terms: "src/purify.ts, dist/purify.js"finding_input: |-Source type: cve- 1. input_id: GHSA-v8jm-5vwx-cfxm-gap-fill-000-vulnerable+ 1. input_id: calibration-6228a29e06832095title: GHSA-v8jm-5vwx-cfxm / CVE-2025-15599advisory ids: GHSA-v8jm-5vwx-cfxm, CVE-2025-15599weakness: xssseverity: moderateaffected language: JavaScriptanchor locations: src/purify.ts:1385, dist/purify.js:1130- fix evidence: https://github.com/cure53/DOMPurify/commit/c861f5a83fb8d90800f1680f855fee551161ac2b- repository state: checked out at c361baa18dbdcb3344a41110f4c48ad85bf48f80. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration fixed: oss-dompurify-ghsa-v8jm-5vwx-cfxm"metadata:- case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-fixed+ case_id: calibration-6a431e1b299f88f3suite: calibration-osscalibration_case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxmcalibration_variant: fixedrepo: "cure53/DOMPurify"expected_binary_label: negativevars:- case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-fixed- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/fixed+ case_id: calibration-6a431e1b299f88f3+ calibration_repo: calibration-6a431e1b299f88f3+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-v8jm-5vwx-cfxm-gap-fill-000-fixed+ expected_ids: calibration-6a431e1b299f88f3expected_source_types: cveexpected_verdicts: not_actionableexpected_binary_label: negative- expected_evidence_terms: "src/purify.ts, dist/purify.js, c861f5a83fb8d90800f1680f855fee551161ac2b"+ expected_evidence_terms: "src/purify.ts, dist/purify.js"finding_input: |-Source type: cve- 1. input_id: GHSA-v8jm-5vwx-cfxm-gap-fill-000-fixed+ 1. input_id: calibration-6a431e1b299f88f3title: GHSA-v8jm-5vwx-cfxm / CVE-2025-15599advisory ids: GHSA-v8jm-5vwx-cfxm, CVE-2025-15599weakness: xssseverity: moderateaffected language: JavaScriptanchor locations: src/purify.ts:1385, dist/purify.js:1130- fix evidence: https://github.com/cure53/DOMPurify/commit/c861f5a83fb8d90800f1680f855fee551161ac2b- repository state: checked out at c861f5a83fb8d90800f1680f855fee551161ac2b. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration vulnerable: oss-monetr-ghsa-29v9-frvh-c426"metadata:- case_id: oss-monetr-ghsa-29v9-frvh-c426-vulnerable+ case_id: calibration-4ceeb56f4c1b0b23suite: calibration-osscalibration_case_id: oss-monetr-ghsa-29v9-frvh-c426calibration_variant: vulnerablerepo: "monetr/monetr"expected_binary_label: positivevars:- case_id: oss-monetr-ghsa-29v9-frvh-c426-vulnerable- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/vulnerable+ case_id: calibration-4ceeb56f4c1b0b23+ calibration_repo: calibration-4ceeb56f4c1b0b23+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-29v9-frvh-c426-000-vulnerable+ expected_ids: calibration-4ceeb56f4c1b0b23expected_source_types: cveexpected_verdicts: confirmedexpected_binary_label: positiveexpected_evidence_terms: "compose/monetr.yaml, server/controller/lunch_flow.go, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go"finding_input: |-Source type: cve- 1. input_id: GHSA-29v9-frvh-c426-000-vulnerable+ 1. input_id: calibration-4ceeb56f4c1b0b23title: GHSA-29v9-frvh-c426 / CVE-2026-41644advisory ids: GHSA-29v9-frvh-c426, CVE-2026-41644weakness: ssrfseverity: highaffected language: YAML/Go/TypeScriptanchor locations: compose/monetr.yaml:34, server/controller/lunch_flow.go:100, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go:213- fix evidence: https://github.com/monetr/monetr/commit/c260caa3c573a4a396ec2d264c7641a5d958385b- repository state: checked out at 195a0c94cad26105dc1e9ee9d5ed902af41ed46c. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.- description: "calibration fixed: oss-monetr-ghsa-29v9-frvh-c426"metadata:- case_id: oss-monetr-ghsa-29v9-frvh-c426-fixed+ case_id: calibration-e364368948a91896suite: calibration-osscalibration_case_id: oss-monetr-ghsa-29v9-frvh-c426calibration_variant: fixedrepo: "monetr/monetr"expected_binary_label: negativevars:- case_id: oss-monetr-ghsa-29v9-frvh-c426-fixed- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/fixed+ case_id: calibration-e364368948a91896+ calibration_repo: calibration-e364368948a91896+ calibration_repo_root: ""source_type_under_test: cve- expected_ids: GHSA-29v9-frvh-c426-000-fixed+ expected_ids: calibration-e364368948a91896expected_source_types: cveexpected_verdicts: not_actionableexpected_binary_label: negative- expected_evidence_terms: "compose/monetr.yaml, server/controller/lunch_flow.go, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go, c260caa3c573a4a396ec2d264c7641a5d958385b"+ expected_evidence_terms: "compose/monetr.yaml, server/controller/lunch_flow.go, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go"finding_input: |-Source type: cve- 1. input_id: GHSA-29v9-frvh-c426-000-fixed+ 1. input_id: calibration-e364368948a91896title: GHSA-29v9-frvh-c426 / CVE-2026-41644advisory ids: GHSA-29v9-frvh-c426, CVE-2026-41644weakness: ssrfseverity: highaffected language: YAML/Go/TypeScriptanchor locations: compose/monetr.yaml:34, server/controller/lunch_flow.go:100, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go:213- fix evidence: https://github.com/monetr/monetr/commit/c260caa3c573a4a396ec2d264c7641a5d958385b- repository state: checked out at c260caa3c573a4a396ec2d264c7641a5d958385b. Triage whether the original finding affects this exact state.eval_instructions: |-This is an automated OSS calibration eval. Do not ask follow-up questions.- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.