Skip to content

OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security

fix(evals): hide expected answers and test the checkout's skill (#1054)

mldangelo-oai6c8ab22cb25d
diff --git a/evals/triage-finding/README.md b/evals/triage-finding/README.md
index 3702f401f..ca020a8ca 100644
--- a/evals/triage-finding/README.md
+++ b/evals/triage-finding/README.md
@@ -4,7 +4,7 @@ This Promptfoo suite verifies that `$codex-security:triage-finding` accepts the
JSON result shape. It also covers bare skill invocation with no supplied finding,
which should prompt the user for a finding in a supported format instead of returning triage JSON. GitHub intake cases cover repository-source selection, REST endpoints, and explicit Connector requests without querying live GitHub during the eval.
-The suite uses the Promptfoo Codex SDK provider because it only needs final assistant output and deterministic assertions. The eval directory owns a small pinned pnpm environment so new cases can be added and run without a separate scratch setup.
+The suite uses the Promptfoo Codex SDK provider because it only needs final assistant output and deterministic assertions. Codex runs from the checkout's plugin directory and reads its triage skill directly, with ambient plugins and memory disabled. The default suite can also read the synthetic fixtures; calibration cases can read their own hydrated checkout. Dataset labels, assertions, and calibration Git history stay outside those readable roots. The eval directory owns a small pinned pnpm environment so new cases can be added and run without a separate scratch setup.
Use Node.js 22.22.0 or newer for the eval runner. Run these commands from the repository root to install dependencies under this eval directory.
@@ -18,6 +18,16 @@ pnpm --dir evals/triage-finding run setup
After editing the tooling, rerun `pnpm --dir sdk/typescript run build:evals`. It prepares the SDK and type-checks the eval sources. Node and Promptfoo execute the TypeScript sources directly; package commands enable type stripping for Node 22.13.
+Before running model evals, build the checkout's policy helper (requires the [native build prerequisites](../../plugins/codex-security/native/README.md)):
+
+```bash
+pnpm --dir plugins/codex-security/mcp-app install --frozen-lockfile
+node plugins/codex-security/mcp-app/scripts/build_native.mjs
+node plugins/codex-security/mcp-app/scripts/build_mcp_app.mjs --output plugins/codex-security/mcp --native host
+```
+
+Config validation and deterministic tests do not need this helper build.
+
Validate the config:
```bash
@@ -59,7 +69,7 @@ paths and query parameters as JSON. Other intake cases use freeform answers.
## Calibration Dataset
-`datasets/triage-calibration-seed.json` is the first OSS-only calibration dataset for scaling beyond the synthetic fixture app. It contains public OSS vulnerable/fixed commit pairs. Each dataset variant becomes one Promptfoo test case in `tests/calibration-oss.yaml`, and each test points Codex at a pinned local checkout under `artifacts/calibration-repos/`.
+`datasets/triage-calibration-seed.json` is the first OSS-only calibration dataset for scaling beyond the synthetic fixture app. It contains public OSS vulnerable/fixed commit pairs. Each dataset variant becomes one Promptfoo test case in `tests/calibration-oss.yaml`, and each test points Codex at a pinned local checkout under `artifacts/calibration-repos/`. Case IDs, finding IDs, and checkout directories use stable opaque names. Variant labels, fix references, and checkout commit IDs remain in harness metadata rather than the model prompt. Hydration keeps Git metadata in opaque sibling directories outside the model-readable checkout. Re-run `calibration:hydrate` to update older checkout layouts.
ELI5: the dataset says "this exact old commit should be affected" and "this exact fixed commit should not be affected." The generator turns those rows into Promptfoo test prompts. The hydrator downloads the exact repo commits so Codex can inspect real code instead of synthetic snippets.
diff --git a/evals/triage-finding/package.json b/evals/triage-finding/package.json
index e9c1be09a..bc8be363f 100644
--- a/evals/triage-finding/package.json
+++ b/evals/triage-finding/package.json
@@ -14,19 +14,19 @@
"test:calibration-generate": "node --experimental-strip-types scripts/test-generate-calibration-tests.mts",
"test:calibration-hydrate": "node --experimental-strip-types scripts/test-hydrate-calibration-repos.mts",
"test:dataset": "node --experimental-strip-types scripts/test-calibration-dataset.mts",
- "test:deterministic": "node --experimental-strip-types scripts/test-app-surface-instructions.mts && node --experimental-strip-types scripts/test-triage-io.mts && node --experimental-strip-types scripts/test-intake-assertions.mts && node --experimental-strip-types scripts/test-calibration-dataset.mts && node --experimental-strip-types scripts/test-generate-calibration-tests.mts && node --experimental-strip-types scripts/test-calibration-evidence.mts && node --experimental-strip-types scripts/test-hydrate-calibration-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-lib.mts && node --experimental-strip-types sastbench/scripts/test-install-sastbench.mts && node --experimental-strip-types sastbench/scripts/test-hydrate-sastbench-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-result.mts && node --experimental-strip-types sastbench/scripts/test-promptfoo-native-harness.mts",
+ "test:deterministic": "node --experimental-strip-types scripts/test-app-surface-instructions.mts && node --experimental-strip-types scripts/test-triage-io.mts && node --experimental-strip-types scripts/test-intake-assertions.mts && node --experimental-strip-types scripts/test-calibration-dataset.mts && node --experimental-strip-types scripts/test-generate-calibration-tests.mts && node --experimental-strip-types scripts/test-promptfoo-runtime.mts && node --experimental-strip-types scripts/test-calibration-evidence.mts && node --experimental-strip-types scripts/test-hydrate-calibration-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-lib.mts && node --experimental-strip-types sastbench/scripts/test-install-sastbench.mts && node --experimental-strip-types sastbench/scripts/test-hydrate-sastbench-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-result.mts && node --experimental-strip-types sastbench/scripts/test-promptfoo-native-harness.mts",
"test:instructions": "node --experimental-strip-types scripts/test-app-surface-instructions.mts",
"test:sastbench": "node --experimental-strip-types sastbench/scripts/test-sastbench-lib.mts && node --experimental-strip-types sastbench/scripts/test-install-sastbench.mts && node --experimental-strip-types sastbench/scripts/test-hydrate-sastbench-repos.mts && node --experimental-strip-types sastbench/scripts/test-sastbench-result.mts && node --experimental-strip-types sastbench/scripts/test-promptfoo-native-harness.mts",
"validate": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo validate config -c ./promptfooconfig.yaml",
"validate:calibration": "pnpm run calibration:generate && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo validate config -c ./promptfooconfig.calibration.yaml",
- "validate:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types sastbench/scripts/run-sastbench-promptfoo.mts validate config -c ./sastbench/promptfooconfig.sastbench.yaml",
- "validate:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types sastbench/scripts/run-sastbench-promptfoo.mts validate config -c ./sastbench/promptfooconfig.sastbench-sample.yaml",
+ "validate:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo validate config -c ./sastbench/promptfooconfig.sastbench.yaml",
+ "validate:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo validate config -c ./sastbench/promptfooconfig.sastbench-sample.yaml",
"eval": "mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./promptfooconfig.yaml -o ./artifacts/triage-finding-eval.json --no-cache --no-share",
"eval:smoke": "mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./promptfooconfig.yaml -o ./artifacts/triage-finding-eval-smoke.json --no-cache --no-share --filter-range 0:1",
"eval:calibration": "pnpm run calibration:generate && pnpm run calibration:hydrate && mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./promptfooconfig.calibration.yaml -o ./artifacts/triage-finding-calibration-eval.json --no-cache --no-share",
"eval:calibration:smoke": "node --experimental-strip-types scripts/generate-calibration-tests.mts --case oss-dompurify-ghsa-v8jm-5vwx-cfxm --variant vulnerable --output artifacts/calibration-smoke.yaml && node --experimental-strip-types scripts/hydrate-calibration-repos.mts --case oss-dompurify-ghsa-v8jm-5vwx-cfxm --variant vulnerable && mkdir -p artifacts && PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./promptfooconfig.calibration-smoke.yaml -o ./artifacts/triage-finding-calibration-smoke.json --no-cache --no-share",
- "eval:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types sastbench/scripts/run-sastbench-promptfoo.mts eval -c ./sastbench/promptfooconfig.sastbench.yaml --no-cache --no-share",
- "eval:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true node --experimental-strip-types sastbench/scripts/run-sastbench-promptfoo.mts eval -c ./sastbench/promptfooconfig.sastbench-sample.yaml --no-cache --no-share --max-concurrency 32",
+ "eval:sastbench": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./sastbench/promptfooconfig.sastbench.yaml --no-cache --no-share",
+ "eval:sastbench:sample": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo eval -c ./sastbench/promptfooconfig.sastbench-sample.yaml --no-cache --no-share --max-concurrency 32",
"pf:view": "PROMPTFOO_CONFIG_DIR=.promptfoo PROMPTFOO_DISABLE_WAL_MODE=true promptfoo view"
},
"dependencies": {
diff --git a/evals/triage-finding/promptfooconfig.calibration-smoke.yaml b/evals/triage-finding/promptfooconfig.calibration-smoke.yaml
index d02c6afa2..c0d70238b 100644
--- a/evals/triage-finding/promptfooconfig.calibration-smoke.yaml
+++ b/evals/triage-finding/promptfooconfig.calibration-smoke.yaml
@@ -8,11 +8,26 @@ providers:
- id: openai:codex-sdk:gpt-5.5
label: "Codex SDK triage-finding calibration smoke"
config:
- working_dir: ../..
- sandbox_mode: read-only
+ working_dir: "{{triage_runtime_root}}"
+ skip_git_repo_check: true
approval_policy: never
- network_access_enabled: false
+ additional_directories:
+ - "{{target_repo}}"
+ - "{{triage_node_root}}"
+ cli_config:
+ default_permissions: triage_runtime_only
+ permissions:
+ triage_runtime_only:
+ filesystem:
+ ":minimal": read
+ ":workspace_roots": read
+ network:
+ enabled: false
+ features:
+ memories: false
+ plugins: false
cli_env:
+ CODEX_MCP_NODE_PATH: "{{triage_node_path}}"
HTTP_PROXY: "{{env.HTTP_PROXY}}"
HTTPS_PROXY: "{{env.HTTPS_PROXY}}"
ALL_PROXY: "{{env.ALL_PROXY}}"
@@ -22,6 +37,8 @@ evaluateOptions:
maxConcurrency: 1
defaultTest:
+ options:
+ transformVars: file://./scripts/runtime-vars.mts
vars:
invocation_preamble: "Use $codex-security:triage-finding to triage the supplied existing security finding."
assert:
diff --git a/evals/triage-finding/promptfooconfig.calibration.yaml b/evals/triage-finding/promptfooconfig.calibration.yaml
index acf6597d8..643fa6da4 100644
--- a/evals/triage-finding/promptfooconfig.calibration.yaml
+++ b/evals/triage-finding/promptfooconfig.calibration.yaml
@@ -8,11 +8,26 @@ providers:
- id: openai:codex-sdk:gpt-5.5
label: "Codex SDK triage-finding calibration"
config:
- working_dir: ../..
- sandbox_mode: read-only
+ working_dir: "{{triage_runtime_root}}"
+ skip_git_repo_check: true
approval_policy: never
- network_access_enabled: false
+ additional_directories:
+ - "{{target_repo}}"
+ - "{{triage_node_root}}"
+ cli_config:
+ default_permissions: triage_runtime_only
+ permissions:
+ triage_runtime_only:
+ filesystem:
+ ":minimal": read
+ ":workspace_roots": read
+ network:
+ enabled: false
+ features:
+ memories: false
+ plugins: false
cli_env:
+ CODEX_MCP_NODE_PATH: "{{triage_node_path}}"
HTTP_PROXY: "{{env.HTTP_PROXY}}"
HTTPS_PROXY: "{{env.HTTPS_PROXY}}"
ALL_PROXY: "{{env.ALL_PROXY}}"
@@ -22,6 +37,8 @@ evaluateOptions:
maxConcurrency: 1
defaultTest:
+ options:
+ transformVars: file://./scripts/runtime-vars.mts
vars:
invocation_preamble: "Use $codex-security:triage-finding to triage the supplied existing security finding."
assert:
diff --git a/evals/triage-finding/promptfooconfig.yaml b/evals/triage-finding/promptfooconfig.yaml
index 29990c605..35be4ad6f 100644
--- a/evals/triage-finding/promptfooconfig.yaml
+++ b/evals/triage-finding/promptfooconfig.yaml
@@ -8,11 +8,26 @@ providers:
- id: openai:codex-sdk:gpt-5.5
label: "Codex SDK triage-finding"
config:
- working_dir: ../..
- sandbox_mode: read-only
+ working_dir: "{{triage_runtime_root}}"
+ skip_git_repo_check: true
approval_policy: never
- network_access_enabled: false
+ additional_directories:
+ - "{{triage_fixture_root}}"
+ - "{{triage_node_root}}"
+ cli_config:
+ default_permissions: triage_runtime_only
+ permissions:
+ triage_runtime_only:
+ filesystem:
+ ":minimal": read
+ ":workspace_roots": read
+ network:
+ enabled: false
+ features:
+ memories: false
+ plugins: false
cli_env:
+ CODEX_MCP_NODE_PATH: "{{triage_node_path}}"
HTTP_PROXY: "{{env.HTTP_PROXY}}"
HTTPS_PROXY: "{{env.HTTPS_PROXY}}"
ALL_PROXY: "{{env.ALL_PROXY}}"
@@ -22,6 +37,8 @@ evaluateOptions:
maxConcurrency: 1
defaultTest:
+ options:
+ transformVars: file://./scripts/runtime-vars.mts
vars:
invocation_preamble: "Use $codex-security:triage-finding to triage the supplied existing security findings."
eval_instructions: |-
diff --git a/evals/triage-finding/prompts/triage-request.txt b/evals/triage-finding/prompts/triage-request.txt
index 40e1cbfec..22a7fc2fe 100644
--- a/evals/triage-finding/prompts/triage-request.txt
+++ b/evals/triage-finding/prompts/triage-request.txt
@@ -1,3 +1,5 @@
+Read and follow skills/triage-finding/SKILL.md from this checkout for the following request.
+
{{invocation_preamble}}
Repository path to inspect:
diff --git a/evals/triage-finding/sastbench/DESIGN.md b/evals/triage-finding/sastbench/DESIGN.md
index 87b40b4f8..75fbcbd62 100644
--- a/evals/triage-finding/sastbench/DESIGN.md
+++ b/evals/triage-finding/sastbench/DESIGN.md
@@ -24,7 +24,7 @@ The SastBench adapter owns only benchmark-specific behavior:
The model-visible prompt contains the affected checkout, an opaque case ID, and the record's `to_analyzer` data. Ground truth is stored in Promptfoo test metadata. The upstream `finding_id`, `ground_truth`, `metadata.source`, and dataset class statistics are not prompt variables.
-The Codex provider runs from a throwaway directory containing only the runtime skill files. A deny-by-default permission profile grants reads only to that directory, the hydrated target repos and their label-free Git cache, and Codex's minimal runtime paths; the label-bearing dataset and Promptfoo harness stay denied.
+The Codex provider runs from the checkout's plugin directory. Its permission profile also grants reads to the selected target, its label-free Git cache, the Node runtime, and Codex's minimal runtime paths; the label-bearing dataset and Promptfoo harness stay denied.
## Verdict mapping
diff --git a/evals/triage-finding/sastbench/README.md b/evals/triage-finding/sastbench/README.md
index 6519a9f38..fdcfe1bcc 100644
--- a/evals/triage-finding/sastbench/README.md
+++ b/evals/triage-finding/sastbench/README.md
@@ -2,17 +2,13 @@
This Promptfoo lane measures how accurately the source version of `plugins/codex-security/skills/triage-finding/SKILL.md` triages supplied scanner findings from public SastBench v0.1. It does not measure repository-wide vulnerability discovery.
-SastBench contains 2,737 findings: 299 labeled true positive and 2,438 labeled false positive. The JavaScript test generator exposes the scanner claim and the exact affected checkout to Codex while keeping SastBench's label-bearing `finding_id` and ground-truth label out of the rendered prompt. The eval runner stages the skill in a throwaway working directory and uses a deny-by-default Codex permission profile so only that directory, hydrated target repos, and their label-free Git cache plus minimal Codex runtime paths are readable to the model.
+SastBench contains 2,737 findings: 299 labeled true positive and 2,438 labeled false positive. The JavaScript test generator exposes the scanner claim and the exact affected checkout to Codex while keeping SastBench's label-bearing `finding_id` and ground-truth label out of the rendered prompt. Codex reads the skill directly from the checkout's plugin directory. Its permission profile also allows the selected target, the label-free Git cache, and the Node runtime; the label-bearing eval harness remains outside those roots.
Run commands from the repository root. The parent `evals/triage-finding/` directory owns the pinned Promptfoo and Codex SDK dependencies and the ignored `artifacts/` tree.
## Prepare the benchmark
-Install dependencies once:
-
-```bash
-pnpm --dir evals/triage-finding run setup
-```
+Follow the [parent setup instructions](../README.md) to install dependencies and build the checkout's policy helper before running model evals.
Install the pinned SastBench checkout and hydrate all 275 repository revisions:
diff --git a/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml b/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml
index b26ea2b25..295915146 100644
--- a/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml
+++ b/evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml
@@ -10,14 +10,13 @@ providers:
- id: openai:codex-sdk:gpt-5.5
label: "Codex SDK triage-finding SastBench (gpt-5.5)"
config:
- # The runner stages only the skill runtime here. The label-bearing
- # dataset and Promptfoo harness stay outside this tree.
- working_dir: "{{env.SASTBENCH_RUNTIME_ROOT}}"
- # Target worktrees and their label-free Git cache are the only
- # additional readable workspace roots.
+ # The plugin checkout excludes the label-bearing dataset and eval harness.
+ working_dir: "{{triage_runtime_root}}"
+ # Read the target, its label-free Git cache, and the helper runtime.
additional_directories:
- - "{{env.SASTBENCH_TARGET_ROOT}}"
- - "{{env.SASTBENCH_GIT_CACHE_ROOT}}"
+ - "{{target_repo}}"
+ - "{{sastbench_git_cache_root}}"
+ - "{{triage_node_root}}"
skip_git_repo_check: true
approval_policy: never
# Pin a value supported by the eval's Codex SDK instead of inheriting a
@@ -36,16 +35,20 @@ providers:
":workspace_roots": read
network:
enabled: false
- # The source skill is loaded explicitly from the staged runtime.
+ # The source skill is loaded explicitly from the checkout.
# Global plugin and memory state would add machine-specific context.
features:
memories: false
plugins: false
+ cli_env:
+ CODEX_MCP_NODE_PATH: "{{triage_node_path}}"
extensions:
- file://./assertions/sastbench-metrics.mts:afterEach
defaultTest:
+ options:
+ transformVars: process.mainModule.require(process.cwd() + '/scripts/runtime-vars.mts').default(vars)
assert:
- type: contains-json
value: file://../schemas/triage-result-v0.schema.json
diff --git a/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt b/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt
index 88cf6eb7d..c33ce67fe 100644
--- a/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt
+++ b/evals/triage-finding/sastbench/prompts/sastbench-triage-request.txt
@@ -1,4 +1,4 @@
-Read and follow plugins/codex-security/skills/triage-finding/SKILL.md to triage the supplied scanner finding.
+Read and follow skills/triage-finding/SKILL.md to triage the supplied scanner finding.
Repository checkout to inspect:
{{target_repo}}
diff --git a/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.mts b/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.mts
deleted file mode 100644
index 3ab8483e4..000000000
--- a/evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.mts
+++ /dev/null
@@ -1,81 +0,0 @@
-#!/usr/bin/env node
-import childProcess from "node:child_process";
-import fs from "node:fs";
-import os from "node:os";
-import path from "node:path";
-
-import { EVAL_ROOT, DEFAULT_TARGET_ROOT } from "./sastbench-lib.mts";
-import { DEFAULT_CACHE_ROOT } from "./hydrate-sastbench-repos.mts";
-const PLUGIN_ROOT = path.resolve(
- EVAL_ROOT,
- "..",
- "..",
- "plugins",
- "codex-security",
-);
-const TRIAGE_SKILL_ROOT = path.join(PLUGIN_ROOT, "skills", "triage-finding");
-const PROMPTFOO_BIN = path.join(EVAL_ROOT, "node_modules", ".bin", "promptfoo");
-
-function copyDirectory(
- sourceRoot: string,
- targetRoot: string,
- excludedNames: Set<string> = new Set(),
-) {
- fs.mkdirSync(targetRoot, { recursive: true });
- for (const entry of fs.readdirSync(sourceRoot, { withFileTypes: true })) {
- if (excludedNames.has(entry.name)) {
- continue;
- }
- const sourcePath = path.join(sourceRoot, entry.name);
- const targetPath = path.join(targetRoot, entry.name);
- if (entry.isDirectory()) {
- copyDirectory(sourcePath, targetPath, excludedNames);
- continue;
- }
- if (!entry.isFile()) {
- throw new Error(
- "Refusing to stage non-file runtime entry: " + sourcePath,
- );
- }
- fs.copyFileSync(sourcePath, targetPath);
- }
-}
-
-/**
- * Give Codex a throwaway working directory that contains only the skill files
- * it needs. The label-bearing dataset and Promptfoo harness stay in EVAL_ROOT.
- */
-const promptfooArgs = process.argv.slice(2);
-if (promptfooArgs.length === 0) {
- throw new Error("Expected Promptfoo arguments");
-}
-const runtimeRoot = fs.mkdtempSync(
- path.join(os.tmpdir(), "codex-security-triage-finding-sastbench-"),
-);
-const stagedPluginRoot = path.join(runtimeRoot, "plugins", "codex-security");
-copyDirectory(
- TRIAGE_SKILL_ROOT,
- path.join(stagedPluginRoot, "skills", "triage-finding"),
- new Set(["evals"]),
-);
-for (const sharedDirectory of ["references", "schemas"]) {
- const sourcePath = path.join(PLUGIN_ROOT, sharedDirectory);
- if (fs.existsSync(sourcePath)) {
- copyDirectory(sourcePath, path.join(stagedPluginRoot, sharedDirectory));
- }
-}
-const env = {
- ...process.env,
- SASTBENCH_RUNTIME_ROOT: runtimeRoot,
- SASTBENCH_TARGET_ROOT: DEFAULT_TARGET_ROOT,
- SASTBENCH_GIT_CACHE_ROOT: DEFAULT_CACHE_ROOT,
-};
-try {
- childProcess.execFileSync(PROMPTFOO_BIN, promptfooArgs, {
- cwd: EVAL_ROOT,
- env,
- stdio: "inherit",
- });
-} finally {
- fs.rmSync(runtimeRoot, { recursive: true, force: true });
-}
diff --git a/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts b/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts
index 8c201d53e..d077d2ad6 100644
--- a/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts
+++ b/evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts
@@ -17,7 +17,7 @@ const packageJson = JSON.parse(
fs.readFileSync(path.join(evalRoot, "package.json"), "utf8"),
);
-assert.match(config, /working_dir:\s+"\{\{env\.SASTBENCH_RUNTIME_ROOT\}\}"/);
+assert.match(config, /working_dir:\s+"\{\{triage_runtime_root\}\}"/);
assert.match(config, /skip_git_repo_check:\s+true/);
assert.doesNotMatch(
config,
@@ -25,7 +25,7 @@ assert.doesNotMatch(
);
assert.match(
config,
- /additional_directories:\n\s+- "\{\{env\.SASTBENCH_TARGET_ROOT\}\}"\n\s+- "\{\{env\.SASTBENCH_GIT_CACHE_ROOT\}\}"/,
+ /additional_directories:\n\s+- "\{\{target_repo\}\}"\n\s+- "\{\{sastbench_git_cache_root\}\}"/,
);
assert.doesNotMatch(config, /sandbox_mode:/);
assert.doesNotMatch(config, /network_access_enabled:/);
@@ -86,10 +86,7 @@ assert.match(
assert.doesNotMatch(sampleConfig, /providers:/);
assert.doesNotMatch(sampleConfig, /derivedMetrics:/);
-assert.match(
- packageJson.scripts["eval:sastbench"],
- /run-sastbench-promptfoo\.mts eval/,
-);
+assert.match(packageJson.scripts["eval:sastbench"], /promptfoo eval/);
assert.match(packageJson.scripts["eval:sastbench"], /--no-cache/);
assert.match(packageJson.scripts["eval:sastbench"], /--no-share/);
assert.doesNotMatch(packageJson.scripts["eval:sastbench"], /--filter-range/);
@@ -102,11 +99,11 @@ assert.doesNotMatch(
);
assert.match(
packageJson.scripts["validate:sastbench:sample"],
- /run-sastbench-promptfoo\.mts validate config -c .\/sastbench\/promptfooconfig\.sastbench-sample\.yaml/,
+ /promptfoo validate config -c .\/sastbench\/promptfooconfig\.sastbench-sample\.yaml/,
);
assert.match(
packageJson.scripts["eval:sastbench:sample"],
- /run-sastbench-promptfoo\.mts eval -c .\/sastbench\/promptfooconfig\.sastbench-sample\.yaml/,
+ /promptfoo eval -c .\/sastbench\/promptfooconfig\.sastbench-sample\.yaml/,
);
assert.match(packageJson.scripts["eval:sastbench:sample"], /--no-cache/);
assert.match(packageJson.scripts["eval:sastbench:sample"], /--no-share/);
@@ -117,7 +114,7 @@ assert.match(
assert.equal("sastbench:generate" in packageJson.scripts, false);
assert.equal(
fs.existsSync(path.join(import.meta.dirname, "run-sastbench-promptfoo.mts")),
- true,
+ false,
);
assert.equal(
fs.existsSync(
diff --git a/evals/triage-finding/scripts/generate-calibration-tests.mts b/evals/triage-finding/scripts/generate-calibration-tests.mts
index c2724e41c..33ae3f570 100644
--- a/evals/triage-finding/scripts/generate-calibration-tests.mts
+++ b/evals/triage-finding/scripts/generate-calibration-tests.mts
@@ -2,6 +2,7 @@
import type { CalibrationCase, CalibrationVariant } from "../types.ts";
import fs from "node:fs";
+import { hash } from "node:crypto";
import path from "node:path";
export const DEFAULT_DATASET = path.join(
@@ -57,29 +58,28 @@ function indentedBlock(value: unknown) {
.replace(/\n/g, "\n ")}`;
}
-function inputId(testCase: CalibrationCase, variant: CalibrationVariant) {
- const base =
- testCase.finding.input_id ||
- testCase.finding.input_id_base ||
- testCase.case_id;
- return `${base}-${variant.variant_id}`;
+export function variantCaseId(
+ testCase: Pick<CalibrationCase, "case_id">,
+ variant: CalibrationVariant,
+) {
+ return `calibration-${hash("sha256", `${testCase.case_id}\0${variant.checkout_ref}`).slice(0, 16)}`;
}
-function evidenceTerms(testCase: CalibrationCase, variant: CalibrationVariant) {
+function evidenceTerms(testCase: CalibrationCase) {
const terms = (testCase.finding.anchor_locations || []).map(
(location) => location.path,
);
- if (variant.variant_id === "fixed" && testCase.finding.fix_patch_ref) {
- terms.push(variant.checkout_ref);
- }
return [...new Set(terms)];
}
-function findingInput(testCase: CalibrationCase, variant: CalibrationVariant) {
+export function findingInput(
+ testCase: CalibrationCase,
+ variant: CalibrationVariant,
+) {
const finding = testCase.finding;
const lines = [
`Source type: ${testCase.source_type}`,
- `1. input_id: ${inputId(testCase, variant)}`,
+ `1. input_id: ${variantCaseId(testCase, variant)}`,
` title: ${finding.title}`,
];
@@ -101,12 +101,6 @@ function findingInput(testCase: CalibrationCase, variant: CalibrationVariant) {
.join(", ");
lines.push(` anchor locations: ${anchors}`);
}
- if (finding.fix_patch_ref) {
- lines.push(` fix evidence: ${finding.fix_patch_ref}`);
- }
- lines.push(
- ` repository state: checked out at ${variant.checkout_ref}. Triage whether the original finding affects this exact state.`,
- );
return lines.join("\n");
}
@@ -116,8 +110,8 @@ function testYaml(
variant: CalibrationVariant,
repoRoot: string,
) {
- const generatedCaseId = `${testCase.case_id}-${variant.variant_id}`;
- const terms = evidenceTerms(testCase, variant);
+ const generatedCaseId = variantCaseId(testCase, variant);
+ const terms = evidenceTerms(testCase);
const lines = [
`- description: ${quote(`calibration ${variant.variant_id}: ${testCase.case_id}`)}`,
" metadata:",
@@ -129,9 +123,10 @@ function testYaml(
` expected_binary_label: ${variant.expected_binary_label}`,
" vars:",
` case_id: ${generatedCaseId}`,
- ` target_repo: ${path.posix.join(repoRoot, testCase.case_id, variant.variant_id)}`,
+ ` calibration_repo: ${generatedCaseId}`,
+ ` calibration_repo_root: ${quote(repoRoot === DEFAULT_REPO_ROOT ? "" : path.resolve(repoRoot))}`,
` source_type_under_test: ${testCase.source_type}`,
- ` expected_ids: ${inputId(testCase, variant)}`,
+ ` expected_ids: ${variantCaseId(testCase, variant)}`,
` expected_source_types: ${testCase.source_type}`,
` expected_verdicts: ${variant.expected_verdict}`,
` expected_binary_label: ${variant.expected_binary_label}`,
@@ -141,7 +136,7 @@ function testYaml(
" eval_instructions: |-",
indentedBlock(
`This is an automated OSS calibration eval. Do not ask follow-up questions.
-Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.`,
diff --git a/evals/triage-finding/scripts/hydrate-calibration-repos.mts b/evals/triage-finding/scripts/hydrate-calibration-repos.mts
index da15c1ddf..f69bd63cd 100644
--- a/evals/triage-finding/scripts/hydrate-calibration-repos.mts
+++ b/evals/triage-finding/scripts/hydrate-calibration-repos.mts
@@ -4,6 +4,7 @@ import { runGit } from "../sastbench/scripts/hydrate-sastbench-repos.mts";
import {
DEFAULT_DATASET,
selectedVariants,
+ variantCaseId,
} from "./generate-calibration-tests.mts";
import fs from "node:fs";
@@ -53,7 +54,7 @@ function gitOutput(args: string[], cwd: string) {
}
}
-function ensureGitCheckout(job: (typeof jobs)[number]) {
+function ensureGitCheckout(job: ReturnType<typeof plannedJobs>[number]) {
fs.mkdirSync(path.dirname(job.targetDir), { recursive: true });
if (!fs.existsSync(job.targetDir)) {
@@ -81,6 +82,13 @@ function ensureGitCheckout(job: (typeof jobs)[number]) {
}
}
+ if (fs.lstatSync(path.join(job.targetDir, ".git")).isDirectory()) {
+ runGit(
+ ["init", "--separate-git-dir", `${job.targetDir}.git`],
+ job.targetDir,
+ );
+ }
+
const currentHead = gitOutput(["rev-parse", "HEAD"], job.targetDir);
if (currentHead === job.checkoutRef) {
return "already current";
@@ -91,24 +99,33 @@ function ensureGitCheckout(job: (typeof jobs)[number]) {
return "hydrated";
}
-const args = parseArgs(process.argv.slice(2));
-const dataset = JSON.parse(fs.readFileSync(args.dataset, "utf8"));
-const jobs = selectedVariants(dataset, args).map(({ testCase, variant }) => ({
- repoUrl: testCase.repo.url,
- checkoutRef: variant.checkout_ref,
- targetDir: path.join(args.repoRoot, testCase.case_id, variant.variant_id),
- label: `${testCase.case_id}/${variant.variant_id}`,
-}));
-const variantWord = jobs.length === 1 ? "variant" : "variants";
-console.log(
- `${args.dryRun ? "would hydrate" : "hydrating"} ${jobs.length} calibration ${variantWord}`,
-);
-for (const job of jobs) {
- if (args.dryRun) {
- console.log(`${job.label} <- ${job.repoUrl} @ ${job.checkoutRef}`);
- console.log(` ${job.targetDir}`);
- } else {
- const status = ensureGitCheckout(job);
- console.log(`${status}: ${job.label} @ ${job.checkoutRef}`);
+export function plannedJobs(
+ dataset: Parameters<typeof selectedVariants>[0],
+ args: Parameters<typeof selectedVariants>[1] & { repoRoot: string },
+) {
+ return selectedVariants(dataset, args).map(({ testCase, variant }) => ({
+ repoUrl: testCase.repo.url,
+ checkoutRef: variant.checkout_ref,
+ targetDir: path.join(args.repoRoot, variantCaseId(testCase, variant)),
+ label: `${testCase.case_id}/${variant.variant_id}`,
+ }));
+}
+
+if (import.meta.filename === fs.realpathSync(process.argv[1])) {
+ const args = parseArgs(process.argv.slice(2));
+ const dataset = JSON.parse(fs.readFileSync(args.dataset, "utf8"));
+ const jobs = plannedJobs(dataset, args);
+ const variantWord = jobs.length === 1 ? "variant" : "variants";
+ console.log(
+ `${args.dryRun ? "would hydrate" : "hydrating"} ${jobs.length} calibration ${variantWord}`,
+ );
+ for (const job of jobs) {
+ if (args.dryRun) {
+ console.log(`${job.label} <- ${job.repoUrl} @ ${job.checkoutRef}`);
+ console.log(` ${job.targetDir}`);
+ } else {
+ const status = ensureGitCheckout(job);
+ console.log(`${status}: ${job.label} @ ${job.checkoutRef}`);
+ }
}
}
diff --git a/evals/triage-finding/scripts/runtime-vars.mts b/evals/triage-finding/scripts/runtime-vars.mts
new file mode 100644
index 000000000..1cda16dd0
--- /dev/null
+++ b/evals/triage-finding/scripts/runtime-vars.mts
@@ -0,0 +1,36 @@
+import fs from "node:fs";
+import path from "node:path";
+
+const evalRoot = path.resolve(import.meta.dirname, "..");
+const sourceRoot = path.resolve(evalRoot, "../..");
+
+// Resolve paths for each execution, including retries of persisted evaluations.
+export default (vars: Record<string, unknown>) => {
+ const nodePath = fs.realpathSync(process.execPath);
+ let targetRepo = vars.target_repo;
+ if (vars.calibration_repo) {
+ targetRepo = path.join(
+ (vars.calibration_repo_root ||
+ path.join(evalRoot, "artifacts", "calibration-repos")) as string,
+ vars.calibration_repo as string,
+ );
+ } else if (
+ typeof targetRepo === "string" &&
+ targetRepo.startsWith("evals/triage-finding/fixtures/")
+ ) {
+ targetRepo = path.join(sourceRoot, targetRepo);
+ }
+ return {
+ ...vars,
+ target_repo: targetRepo,
+ triage_runtime_root: path.join(sourceRoot, "plugins", "codex-security"),
+ triage_fixture_root: path.join(evalRoot, "fixtures"),
+ triage_node_path: nodePath,
+ triage_node_root: path.dirname(nodePath),
+ sastbench_git_cache_root: path.join(
+ evalRoot,
+ "artifacts",
+ "sastbench-git-cache",
+ ),
+ };
+};
diff --git a/evals/triage-finding/scripts/test-generate-calibration-tests.mts b/evals/triage-finding/scripts/test-generate-calibration-tests.mts
index f512906b6..9f8cc7d7e 100644
--- a/evals/triage-finding/scripts/test-generate-calibration-tests.mts
+++ b/evals/triage-finding/scripts/test-generate-calibration-tests.mts
@@ -5,6 +5,9 @@ import { execFileSync } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
+import { findingInput, variantCaseId } from "./generate-calibration-tests.mts";
+import type { CalibrationCase } from "../types.ts";
+import { plannedJobs } from "./hydrate-calibration-repos.mts";
const evalDir = path.resolve(import.meta.dirname, "..");
const generator = path.join(
@@ -15,8 +18,8 @@ const generator = path.join(
const dataset = path.join(evalDir, "datasets", "triage-calibration-seed.json");
const trackedTests = path.join(evalDir, "tests", "calibration-oss.yaml");
-const tmpDir = fs.mkdtempSync(
- path.join(os.tmpdir(), "triage-calibration-tests-"),
+const tmpDir = fs.realpathSync(
+ fs.mkdtempSync(path.join(os.tmpdir(), "triage-calibration-tests-")),
);
const generated = path.join(tmpDir, "calibration-oss.yaml");
const generatedSmoke = path.join(tmpDir, "calibration-smoke.yaml");
@@ -45,17 +48,40 @@ assert.equal(
trackedYaml,
"tracked calibration tests are stale; run calibration:generate",
);
-assert.match(
- generatedYaml,
- /case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerable/,
-);
-assert.match(generatedYaml, /case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed/);
+const cases: CalibrationCase[] = JSON.parse(
+ fs.readFileSync(dataset, "utf8"),
+).cases;
+const identities = new Set();
+for (const testCase of cases) {
+ for (const variant of testCase.variants) {
+ const caseId = variantCaseId(testCase, variant);
+ assert.match(caseId, /^calibration-[a-f0-9]{16}$/);
+ assert.ok(!identities.has(caseId));
+ identities.add(caseId);
+ assert.equal(
+ variantCaseId(testCase, {
+ ...variant,
+ variant_id: "renamed",
+ expected_verdict: "needs_review",
+ }),
+ caseId,
+ );
+ const input = findingInput(testCase, variant);
+ assert.ok(!input.includes(variant.checkout_ref));
+ assert.ok(!input.includes(testCase.finding.fix_patch_ref!));
+ assert.doesNotMatch(input, /-(?:vulnerable|fixed)\b/);
+ const repoRoot = path.join(tmpDir, "targets");
+ const [job] = plannedJobs(
+ { cases: [{ ...testCase, variants: [variant] }] },
+ { repoRoot },
+ );
+ assert.equal(job.targetDir, path.join(repoRoot, caseId));
+ assert.ok(generatedYaml.includes(` calibration_repo: ${caseId}`));
+ }
+}
assert.match(generatedYaml, /expected_verdicts: confirmed/);
assert.match(generatedYaml, /expected_verdicts: not_actionable/);
-assert.match(
- generatedYaml,
- /target_repo: evals\/triage-finding\/artifacts\/calibration-repos\//,
-);
+assert.match(generatedYaml, /calibration_repo_root: ""/);
assert.doesNotMatch(
generatedYaml,
/expected_evidence_terms:\n\s+- /,
@@ -75,23 +101,44 @@ execFileSync(
"oss-dompurify-ghsa-v8jm-5vwx-cfxm",
"--variant",
"vulnerable",
+ "--repo-root",
+ "custom targets # space",
],
{
- cwd: evalDir,
+ cwd: tmpDir,
stdio: "pipe",
},
);
const smokeYaml = fs.readFileSync(generatedSmoke, "utf8");
assert.equal((smokeYaml.match(/^- description:/gm) || []).length, 1);
-assert.match(
- smokeYaml,
- /case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-vulnerable/,
+assert.ok(
+ smokeYaml.includes(
+ `calibration_repo_root: ${JSON.stringify(path.join(tmpDir, "custom targets # space"))}`,
+ ),
);
-assert.doesNotMatch(
- smokeYaml,
- /case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-fixed/,
+const smokeCase = cases.find(
+ (testCase) => testCase.case_id === "oss-dompurify-ghsa-v8jm-5vwx-cfxm",
+)!;
+assert.ok(
+ smokeYaml.includes(
+ `case_id: ${variantCaseId(
+ smokeCase,
+ smokeCase.variants.find(
+ (variant) => variant.variant_id === "vulnerable",
+ )!,
+ )}`,
+ ),
+);
+assert.ok(
+ !smokeYaml.includes(
+ `case_id: ${variantCaseId(
+ smokeCase,
+ smokeCase.variants.find((variant) => variant.variant_id === "fixed")!,
+ )}`,
+ ),
);
+fs.rmSync(tmpDir, { recursive: true, force: true });
console.log(
"calibration test generation matches tracked YAML and supports filtered smoke output",
diff --git a/evals/triage-finding/scripts/test-hydrate-calibration-repos.mts b/evals/triage-finding/scripts/test-hydrate-calibration-repos.mts
index dcf6d6532..429b3b6b1 100644
--- a/evals/triage-finding/scripts/test-hydrate-calibration-repos.mts
+++ b/evals/triage-finding/scripts/test-hydrate-calibration-repos.mts
@@ -1,9 +1,11 @@
#!/usr/bin/env node
+
import assert from "node:assert/strict";
import childProcess from "node:child_process";
-import path from "node:path";
import fs from "node:fs";
import os from "node:os";
+import path from "node:path";
+import { plannedJobs } from "./hydrate-calibration-repos.mts";
const evalDir = path.join(import.meta.dirname, "..");
const scriptPath = path.join(
@@ -89,20 +91,20 @@ try {
const originalOrigin = git(parent, "remote", "get-url", "origin");
const expectedHead = git(source, "rev-parse", "HEAD");
const dataset = path.join(temporary, "dataset.json");
- fs.writeFileSync(
- dataset,
- JSON.stringify({
- cases: [
- {
- case_id: "case",
- repo: { url: source },
- variants: [{ variant_id: "variant", checkout_ref: expectedHead }],
- },
- ],
- }),
- );
- const repoRoot = path.join(parent, "checkouts");
- const target = path.join(repoRoot, "case", "variant");
+ const fixture = {
+ cases: [
+ {
+ case_id: "case",
+ repo: { url: source },
+ variants: [{ variant_id: "variant", checkout_ref: expectedHead }],
+ },
+ ],
+ };
+ fs.writeFileSync(dataset, JSON.stringify(fixture));
+ const repoRoot = path.join(parent, "checkout fixtures");
+ const target = plannedJobs(fixture as Parameters<typeof plannedJobs>[0], {
+ repoRoot,
+ })[0].targetDir;
fs.mkdirSync(path.join(target, ".git"), { recursive: true });
const rejected = childProcess.spawnSync(
process.execPath,
@@ -128,6 +130,11 @@ try {
/hydrated: case\/variant/,
);
assert.equal(git(target, "rev-parse", "HEAD"), expectedHead);
+ assert.ok(fs.lstatSync(path.join(target, ".git")).isFile());
+ assert.equal(
+ fs.realpathSync(git(target, "rev-parse", "--absolute-git-dir")),
+ fs.realpathSync(`${target}.git`),
+ );
assert.equal(git(parent, "remote", "get-url", "origin"), originalOrigin);
assert.equal(git(parent, "rev-parse", "HEAD"), originalHead);
assert.match(
@@ -145,6 +152,50 @@ try {
runHydrator(["--dataset", dataset, "--repo-root", alias]),
/already current/,
);
+
+ const legacyRoot = path.join(parent, "legacy checkouts");
+ const legacyTarget = path.join(legacyRoot, path.basename(target));
+ fs.mkdirSync(legacyRoot);
+ git(parent, "clone", source, legacyTarget);
+ assert.ok(fs.lstatSync(path.join(legacyTarget, ".git")).isDirectory());
+ const legacyAlias = path.join(temporary, "legacy checkout alias");
+ fs.symlinkSync(
+ legacyRoot,
+ legacyAlias,
+ process.platform === "win32" ? "junction" : "dir",
+ );
+ assert.match(
+ runHydrator(["--dataset", dataset, "--repo-root", legacyAlias]),
+ /already current/,
+ );
+ assert.ok(fs.lstatSync(path.join(legacyTarget, ".git")).isFile());
+ assert.equal(git(legacyTarget, "rev-parse", "HEAD"), expectedHead);
+ assert.equal(git(legacyTarget, "remote", "get-url", "origin"), source);
+ assert.equal(
+ fs.readFileSync(path.join(legacyTarget, "fixture.txt"), "utf8"),
+ "source",
+ );
+
+ const worktreeRoot = path.join(parent, "linked checkouts");
+ const worktreeTarget = path.join(worktreeRoot, path.basename(target));
+ git(
+ legacyTarget,
+ "worktree",
+ "add",
+ "--detach",
+ worktreeTarget,
+ expectedHead,
+ );
+ const gitfile = fs.readFileSync(path.join(worktreeTarget, ".git"), "utf8");
+ assert.match(
+ runHydrator(["--dataset", dataset, "--repo-root", worktreeRoot]),
+ /already current/,
+ );
+ assert.equal(
+ fs.readFileSync(path.join(worktreeTarget, ".git"), "utf8"),
+ gitfile,
+ );
+ assert.equal(git(worktreeTarget, "rev-parse", "HEAD"), expectedHead);
} finally {
fs.rmSync(temporary, { recursive: true, force: true });
}
diff --git a/evals/triage-finding/scripts/test-promptfoo-runtime.mts b/evals/triage-finding/scripts/test-promptfoo-runtime.mts
new file mode 100644
index 000000000..9b82ca927
--- /dev/null
+++ b/evals/triage-finding/scripts/test-promptfoo-runtime.mts
@@ -0,0 +1,80 @@
+#!/usr/bin/env node
+
+import assert from "node:assert/strict";
+import fs from "node:fs";
+import path from "node:path";
+import runtimeVars from "./runtime-vars.mts";
+import { DEFAULT_CACHE_ROOT } from "../sastbench/scripts/hydrate-sastbench-repos.mts";
+
+const evalRoot = path.resolve(import.meta.dirname, "..");
+const sourceRoot = path.resolve(evalRoot, "../..");
+const pluginRoot = path.join(sourceRoot, "plugins", "codex-security");
+const vars = runtimeVars({});
+assert.equal(vars.triage_runtime_root, pluginRoot);
+assert.equal(vars.triage_fixture_root, path.join(evalRoot, "fixtures"));
+assert.equal(vars.triage_node_path, fs.realpathSync(process.execPath));
+assert.equal(vars.triage_node_root, path.dirname(vars.triage_node_path));
+assert.equal(vars.sastbench_git_cache_root, DEFAULT_CACHE_ROOT);
+assert.ok(
+ fs.existsSync(path.join(pluginRoot, "skills/triage-finding/SKILL.md")),
+);
+
+for (const fixture of ["repo", "policy-repo"]) {
+ const target = runtimeVars({
+ target_repo: `evals/triage-finding/fixtures/${fixture}`,
+ }).target_repo;
+ assert.equal(target, path.join(evalRoot, "fixtures", fixture));
+ assert.ok(fs.existsSync(target as string));
+}
+for (const target of [
+ path.join(evalRoot, "artifacts", "sastbench-targets", "opaque-target"),
+ "https://github.com/example/repository",
+ "No explicit repository supplied by the user.",
+]) {
+ assert.equal(runtimeVars({ target_repo: target }).target_repo, target);
+}
+
+const calibrationRepo = "calibration-0123456789abcdef";
+for (const customRoot of ["", path.join(evalRoot, "custom targets # space")]) {
+ const replayed = runtimeVars({
+ calibration_repo: calibrationRepo,
+ calibration_repo_root: customRoot,
+ target_repo: "old checkout",
+ triage_runtime_root: "old plugin",
+ });
+ assert.equal(replayed.triage_runtime_root, pluginRoot);
+ assert.equal(
+ replayed.target_repo,
+ path.join(
+ customRoot || path.join(evalRoot, "artifacts", "calibration-repos"),
+ calibrationRepo,
+ ),
+ );
+}
+
+for (const name of [
+ "promptfooconfig.yaml",
+ "promptfooconfig.calibration.yaml",
+ "promptfooconfig.calibration-smoke.yaml",
+ "sastbench/promptfooconfig.sastbench.yaml",
+]) {
+ const config = fs.readFileSync(path.join(evalRoot, name), "utf8");
+ assert.match(config, /working_dir: "\{\{triage_runtime_root\}\}"/);
+ assert.match(config, /":minimal": read\n\s+":workspace_roots": read/);
+ assert.match(config, /plugins: false/);
+ assert.match(config, /memories: false/);
+ assert.ok(config.includes('"{{triage_node_root}}"'));
+ assert.ok(
+ config.includes(
+ name === "promptfooconfig.yaml"
+ ? '"{{triage_fixture_root}}"'
+ : '"{{target_repo}}"',
+ ),
+ );
+}
+assert.ok(
+ fs
+ .readFileSync(path.join(evalRoot, "prompts/triage-request.txt"), "utf8")
+ .startsWith("Read and follow skills/triage-finding/SKILL.md"),
+);
+console.log("triage Promptfoo checkout path tests passed");
diff --git a/evals/triage-finding/tests/calibration-oss.yaml b/evals/triage-finding/tests/calibration-oss.yaml
index 89e3ab0c1..29f004e82 100644
--- a/evals/triage-finding/tests/calibration-oss.yaml
+++ b/evals/triage-finding/tests/calibration-oss.yaml
@@ -1,559 +1,543 @@
- description: "calibration vulnerable: oss-mantisbt-ghsa-73vx-49mv-v8w5"
metadata:
- case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerable
+ case_id: calibration-393a6e303af5b190
suite: calibration-oss
calibration_case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5
calibration_variant: vulnerable
repo: "mantisbt/mantisbt"
expected_binary_label: positive
vars:
- case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-vulnerable
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/vulnerable
+ case_id: calibration-393a6e303af5b190
+ calibration_repo: calibration-393a6e303af5b190
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-73vx-49mv-v8w5-000-vulnerable
+ expected_ids: calibration-393a6e303af5b190
expected_source_types: cve
expected_verdicts: confirmed
expected_binary_label: positive
expected_evidence_terms: "core/classes/IssueTagTimelineEvent.class.php"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-73vx-49mv-v8w5-000-vulnerable
+ 1. input_id: calibration-393a6e303af5b190
title: GHSA-73vx-49mv-v8w5 / CVE-2026-33548
advisory ids: GHSA-73vx-49mv-v8w5, CVE-2026-33548
weakness: xss
severity: high
affected language: PHP
anchor locations: core/classes/IssueTagTimelineEvent.class.php:61
- fix evidence: https://github.com/mantisbt/mantisbt/commit/f32787c14d4518476fe7f05f992dbfe6eaccd815
- repository state: checked out at 80990f43153167c73f11eb4b2bc7108d0c3d6b46. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration fixed: oss-mantisbt-ghsa-73vx-49mv-v8w5"
metadata:
- case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed
+ case_id: calibration-0310e3e9455d0db5
suite: calibration-oss
calibration_case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5
calibration_variant: fixed
repo: "mantisbt/mantisbt"
expected_binary_label: negative
vars:
- case_id: oss-mantisbt-ghsa-73vx-49mv-v8w5-fixed
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-mantisbt-ghsa-73vx-49mv-v8w5/fixed
+ case_id: calibration-0310e3e9455d0db5
+ calibration_repo: calibration-0310e3e9455d0db5
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-73vx-49mv-v8w5-000-fixed
+ expected_ids: calibration-0310e3e9455d0db5
expected_source_types: cve
expected_verdicts: not_actionable
expected_binary_label: negative
- expected_evidence_terms: "core/classes/IssueTagTimelineEvent.class.php, f32787c14d4518476fe7f05f992dbfe6eaccd815"
+ expected_evidence_terms: "core/classes/IssueTagTimelineEvent.class.php"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-73vx-49mv-v8w5-000-fixed
+ 1. input_id: calibration-0310e3e9455d0db5
title: GHSA-73vx-49mv-v8w5 / CVE-2026-33548
advisory ids: GHSA-73vx-49mv-v8w5, CVE-2026-33548
weakness: xss
severity: high
affected language: PHP
anchor locations: core/classes/IssueTagTimelineEvent.class.php:61
- fix evidence: https://github.com/mantisbt/mantisbt/commit/f32787c14d4518476fe7f05f992dbfe6eaccd815
- repository state: checked out at f32787c14d4518476fe7f05f992dbfe6eaccd815. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration vulnerable: oss-imagemagick-ghsa-qpgx-jfcq-r59f"
metadata:
- case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-vulnerable
+ case_id: calibration-89e4ec9f158dc8b9
suite: calibration-oss
calibration_case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f
calibration_variant: vulnerable
repo: "ImageMagick/ImageMagick"
expected_binary_label: positive
vars:
- case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-vulnerable
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/vulnerable
+ case_id: calibration-89e4ec9f158dc8b9
+ calibration_repo: calibration-89e4ec9f158dc8b9
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-qpgx-jfcq-r59f-000-vulnerable
+ expected_ids: calibration-89e4ec9f158dc8b9
expected_source_types: cve
expected_verdicts: confirmed
expected_binary_label: positive
expected_evidence_terms: "MagickCore/visual-effects.c"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-qpgx-jfcq-r59f-000-vulnerable
+ 1. input_id: calibration-89e4ec9f158dc8b9
title: GHSA-qpgx-jfcq-r59f / CVE-2026-27798
advisory ids: GHSA-qpgx-jfcq-r59f, CVE-2026-27798
weakness: memory-corruption
severity: medium
affected language: C/C++
anchor locations: MagickCore/visual-effects.c:3566
- fix evidence: https://github.com/ImageMagick/ImageMagick/commit/0377e60b3c0d766bd7271221c95d9ee54f6a3738
- repository state: checked out at 83847c348aafccbbea89ec8ed44382598aea3f98. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration fixed: oss-imagemagick-ghsa-qpgx-jfcq-r59f"
metadata:
- case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-fixed
+ case_id: calibration-11948efe569f3fb9
suite: calibration-oss
calibration_case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f
calibration_variant: fixed
repo: "ImageMagick/ImageMagick"
expected_binary_label: negative
vars:
- case_id: oss-imagemagick-ghsa-qpgx-jfcq-r59f-fixed
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-imagemagick-ghsa-qpgx-jfcq-r59f/fixed
+ case_id: calibration-11948efe569f3fb9
+ calibration_repo: calibration-11948efe569f3fb9
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-qpgx-jfcq-r59f-000-fixed
+ expected_ids: calibration-11948efe569f3fb9
expected_source_types: cve
expected_verdicts: not_actionable
expected_binary_label: negative
- expected_evidence_terms: "MagickCore/visual-effects.c, 0377e60b3c0d766bd7271221c95d9ee54f6a3738"
+ expected_evidence_terms: "MagickCore/visual-effects.c"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-qpgx-jfcq-r59f-000-fixed
+ 1. input_id: calibration-11948efe569f3fb9
title: GHSA-qpgx-jfcq-r59f / CVE-2026-27798
advisory ids: GHSA-qpgx-jfcq-r59f, CVE-2026-27798
weakness: memory-corruption
severity: medium
affected language: C/C++
anchor locations: MagickCore/visual-effects.c:3566
- fix evidence: https://github.com/ImageMagick/ImageMagick/commit/0377e60b3c0d766bd7271221c95d9ee54f6a3738
- repository state: checked out at 0377e60b3c0d766bd7271221c95d9ee54f6a3738. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration vulnerable: oss-scitokens-ghsa-rh5m-2482-966c"
metadata:
- case_id: oss-scitokens-ghsa-rh5m-2482-966c-vulnerable
+ case_id: calibration-576c7d9cbbc7c018
suite: calibration-oss
calibration_case_id: oss-scitokens-ghsa-rh5m-2482-966c
calibration_variant: vulnerable
repo: "scitokens/scitokens"
expected_binary_label: positive
vars:
- case_id: oss-scitokens-ghsa-rh5m-2482-966c-vulnerable
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/vulnerable
+ case_id: calibration-576c7d9cbbc7c018
+ calibration_repo: calibration-576c7d9cbbc7c018
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-rh5m-2482-966c-000-vulnerable
+ expected_ids: calibration-576c7d9cbbc7c018
expected_source_types: cve
expected_verdicts: confirmed
expected_binary_label: positive
expected_evidence_terms: "src/scitokens/utils/keycache.py"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-rh5m-2482-966c-000-vulnerable
+ 1. input_id: calibration-576c7d9cbbc7c018
title: GHSA-rh5m-2482-966c / CVE-2026-32714
advisory ids: GHSA-rh5m-2482-966c, CVE-2026-32714
weakness: sql-injection
severity: critical
affected language: Python
anchor locations: src/scitokens/utils/keycache.py:80
- fix evidence: https://github.com/scitokens/scitokens/commit/3dba108853f2f4a6c0f2325c03779bf083c41cf2
- repository state: checked out at 66b075c35da18f8364276553513a53c596b4b8e3. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration fixed: oss-scitokens-ghsa-rh5m-2482-966c"
metadata:
- case_id: oss-scitokens-ghsa-rh5m-2482-966c-fixed
+ case_id: calibration-3aaf9d4f40ef518c
suite: calibration-oss
calibration_case_id: oss-scitokens-ghsa-rh5m-2482-966c
calibration_variant: fixed
repo: "scitokens/scitokens"
expected_binary_label: negative
vars:
- case_id: oss-scitokens-ghsa-rh5m-2482-966c-fixed
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-scitokens-ghsa-rh5m-2482-966c/fixed
+ case_id: calibration-3aaf9d4f40ef518c
+ calibration_repo: calibration-3aaf9d4f40ef518c
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-rh5m-2482-966c-000-fixed
+ expected_ids: calibration-3aaf9d4f40ef518c
expected_source_types: cve
expected_verdicts: not_actionable
expected_binary_label: negative
- expected_evidence_terms: "src/scitokens/utils/keycache.py, 3dba108853f2f4a6c0f2325c03779bf083c41cf2"
+ expected_evidence_terms: "src/scitokens/utils/keycache.py"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-rh5m-2482-966c-000-fixed
+ 1. input_id: calibration-3aaf9d4f40ef518c
title: GHSA-rh5m-2482-966c / CVE-2026-32714
advisory ids: GHSA-rh5m-2482-966c, CVE-2026-32714
weakness: sql-injection
severity: critical
affected language: Python
anchor locations: src/scitokens/utils/keycache.py:80
- fix evidence: https://github.com/scitokens/scitokens/commit/3dba108853f2f4a6c0f2325c03779bf083c41cf2
- repository state: checked out at 3dba108853f2f4a6c0f2325c03779bf083c41cf2. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration vulnerable: oss-emissary-ghsa-3p24-9x7v-7789"
metadata:
- case_id: oss-emissary-ghsa-3p24-9x7v-7789-vulnerable
+ case_id: calibration-1561f6cf0aa4fa71
suite: calibration-oss
calibration_case_id: oss-emissary-ghsa-3p24-9x7v-7789
calibration_variant: vulnerable
repo: "NationalSecurityAgency/emissary"
expected_binary_label: positive
vars:
- case_id: oss-emissary-ghsa-3p24-9x7v-7789-vulnerable
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/vulnerable
+ case_id: calibration-1561f6cf0aa4fa71
+ calibration_repo: calibration-1561f6cf0aa4fa71
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-3p24-9x7v-7789-000-vulnerable
+ expected_ids: calibration-1561f6cf0aa4fa71
expected_source_types: cve
expected_verdicts: confirmed
expected_binary_label: positive
expected_evidence_terms: "src/main/java/emissary/util/shell/Executrix.java"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-3p24-9x7v-7789-000-vulnerable
+ 1. input_id: calibration-1561f6cf0aa4fa71
title: GHSA-3p24-9x7v-7789 / CVE-2026-35582
advisory ids: GHSA-3p24-9x7v-7789, CVE-2026-35582
weakness: command-injection
severity: high
affected language: Java
anchor locations: src/main/java/emissary/util/shell/Executrix.java:77
- fix evidence: https://github.com/NationalSecurityAgency/emissary/commit/1faf33f2494c0128f250d7d2e8f2da99bbd32ae8
- repository state: checked out at 082b263a19532454b24dfbed1605311854ea778f. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration fixed: oss-emissary-ghsa-3p24-9x7v-7789"
metadata:
- case_id: oss-emissary-ghsa-3p24-9x7v-7789-fixed
+ case_id: calibration-ca1bddda373398e8
suite: calibration-oss
calibration_case_id: oss-emissary-ghsa-3p24-9x7v-7789
calibration_variant: fixed
repo: "NationalSecurityAgency/emissary"
expected_binary_label: negative
vars:
- case_id: oss-emissary-ghsa-3p24-9x7v-7789-fixed
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-emissary-ghsa-3p24-9x7v-7789/fixed
+ case_id: calibration-ca1bddda373398e8
+ calibration_repo: calibration-ca1bddda373398e8
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-3p24-9x7v-7789-000-fixed
+ expected_ids: calibration-ca1bddda373398e8
expected_source_types: cve
expected_verdicts: not_actionable
expected_binary_label: negative
- expected_evidence_terms: "src/main/java/emissary/util/shell/Executrix.java, 1faf33f2494c0128f250d7d2e8f2da99bbd32ae8"
+ expected_evidence_terms: "src/main/java/emissary/util/shell/Executrix.java"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-3p24-9x7v-7789-000-fixed
+ 1. input_id: calibration-ca1bddda373398e8
title: GHSA-3p24-9x7v-7789 / CVE-2026-35582
advisory ids: GHSA-3p24-9x7v-7789, CVE-2026-35582
weakness: command-injection
severity: high
affected language: Java
anchor locations: src/main/java/emissary/util/shell/Executrix.java:77
- fix evidence: https://github.com/NationalSecurityAgency/emissary/commit/1faf33f2494c0128f250d7d2e8f2da99bbd32ae8
- repository state: checked out at 1faf33f2494c0128f250d7d2e8f2da99bbd32ae8. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration vulnerable: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5"
metadata:
- case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-vulnerable
+ case_id: calibration-2cd95700e4d0032d
suite: calibration-oss
calibration_case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5
calibration_variant: vulnerable
repo: "free5gc/pcf"
expected_binary_label: positive
vars:
- case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-vulnerable
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/vulnerable
+ case_id: calibration-2cd95700e4d0032d
+ calibration_repo: calibration-2cd95700e4d0032d
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-6RGM-GR97-X3J5-vulnerable
+ expected_ids: calibration-2cd95700e4d0032d
expected_source_types: cve
expected_verdicts: confirmed
expected_binary_label: positive
expected_evidence_terms: "internal/sbi/server.go"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-6RGM-GR97-X3J5-vulnerable
+ 1. input_id: calibration-2cd95700e4d0032d
title: GHSA-6RGM-GR97-X3J5 / CVE-2026-42083
advisory ids: GHSA-6RGM-GR97-X3J5, CVE-2026-42083
weakness: authz
severity: high
affected language: Go
anchor locations: internal/sbi/server.go:72
- fix evidence: https://github.com/free5gc/pcf/commit/8c4d457cdf58bb239ee30e88c56b370b22073964
- repository state: checked out at 508d70b8527a6c8c923179dad450ea01e16b6aeb. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration fixed: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5"
metadata:
- case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-fixed
+ case_id: calibration-1b89c72834a4a631
suite: calibration-oss
calibration_case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5
calibration_variant: fixed
repo: "free5gc/pcf"
expected_binary_label: negative
vars:
- case_id: oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5-fixed
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-free5gc-pcf-ghsa-6rgm-gr97-x3j5/fixed
+ case_id: calibration-1b89c72834a4a631
+ calibration_repo: calibration-1b89c72834a4a631
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-6RGM-GR97-X3J5-fixed
+ expected_ids: calibration-1b89c72834a4a631
expected_source_types: cve
expected_verdicts: not_actionable
expected_binary_label: negative
- expected_evidence_terms: "internal/sbi/server.go, 8c4d457cdf58bb239ee30e88c56b370b22073964"
+ expected_evidence_terms: "internal/sbi/server.go"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-6RGM-GR97-X3J5-fixed
+ 1. input_id: calibration-1b89c72834a4a631
title: GHSA-6RGM-GR97-X3J5 / CVE-2026-42083
advisory ids: GHSA-6RGM-GR97-X3J5, CVE-2026-42083
weakness: authz
severity: high
affected language: Go
anchor locations: internal/sbi/server.go:72
- fix evidence: https://github.com/free5gc/pcf/commit/8c4d457cdf58bb239ee30e88c56b370b22073964
- repository state: checked out at 8c4d457cdf58bb239ee30e88c56b370b22073964. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration vulnerable: oss-actualbudget-ghsa-27vg-33gh-4hwg"
metadata:
- case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-vulnerable
+ case_id: calibration-e4d2cc784a81e08c
suite: calibration-oss
calibration_case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg
calibration_variant: vulnerable
repo: "actualbudget/actual"
expected_binary_label: positive
vars:
- case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-vulnerable
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/vulnerable
+ case_id: calibration-e4d2cc784a81e08c
+ calibration_repo: calibration-e4d2cc784a81e08c
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-27vg-33gh-4hwg-000-vulnerable
+ expected_ids: calibration-e4d2cc784a81e08c
expected_source_types: cve
expected_verdicts: confirmed
expected_binary_label: positive
expected_evidence_terms: "packages/sync-server/src/app-sync.ts, packages/sync-server/src/app-sync.test.ts"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-27vg-33gh-4hwg-000-vulnerable
+ 1. input_id: calibration-e4d2cc784a81e08c
title: GHSA-27vg-33gh-4hwg / CVE-2026-3089
advisory ids: GHSA-27vg-33gh-4hwg, CVE-2026-3089
weakness: path-traversal
severity: medium
affected language: TypeScript
anchor locations: packages/sync-server/src/app-sync.ts:49, packages/sync-server/src/app-sync.test.ts:366
- fix evidence: https://github.com/actualbudget/actual/commit/18072e1d8b5281db43ded8b21433ee177bae9dfa
- repository state: checked out at a1e0b3f45d36232a0f4319a60cf264912edac48f. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration fixed: oss-actualbudget-ghsa-27vg-33gh-4hwg"
metadata:
- case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-fixed
+ case_id: calibration-b4528e41b5de5a74
suite: calibration-oss
calibration_case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg
calibration_variant: fixed
repo: "actualbudget/actual"
expected_binary_label: negative
vars:
- case_id: oss-actualbudget-ghsa-27vg-33gh-4hwg-fixed
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-actualbudget-ghsa-27vg-33gh-4hwg/fixed
+ case_id: calibration-b4528e41b5de5a74
+ calibration_repo: calibration-b4528e41b5de5a74
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-27vg-33gh-4hwg-000-fixed
+ expected_ids: calibration-b4528e41b5de5a74
expected_source_types: cve
expected_verdicts: not_actionable
expected_binary_label: negative
- expected_evidence_terms: "packages/sync-server/src/app-sync.ts, packages/sync-server/src/app-sync.test.ts, 18072e1d8b5281db43ded8b21433ee177bae9dfa"
+ expected_evidence_terms: "packages/sync-server/src/app-sync.ts, packages/sync-server/src/app-sync.test.ts"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-27vg-33gh-4hwg-000-fixed
+ 1. input_id: calibration-b4528e41b5de5a74
title: GHSA-27vg-33gh-4hwg / CVE-2026-3089
advisory ids: GHSA-27vg-33gh-4hwg, CVE-2026-3089
weakness: path-traversal
severity: medium
affected language: TypeScript
anchor locations: packages/sync-server/src/app-sync.ts:49, packages/sync-server/src/app-sync.test.ts:366
- fix evidence: https://github.com/actualbudget/actual/commit/18072e1d8b5281db43ded8b21433ee177bae9dfa
- repository state: checked out at 18072e1d8b5281db43ded8b21433ee177bae9dfa. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration vulnerable: oss-dompurify-ghsa-v8jm-5vwx-cfxm"
metadata:
- case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-vulnerable
+ case_id: calibration-6228a29e06832095
suite: calibration-oss
calibration_case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm
calibration_variant: vulnerable
repo: "cure53/DOMPurify"
expected_binary_label: positive
vars:
- case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-vulnerable
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/vulnerable
+ case_id: calibration-6228a29e06832095
+ calibration_repo: calibration-6228a29e06832095
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-v8jm-5vwx-cfxm-gap-fill-000-vulnerable
+ expected_ids: calibration-6228a29e06832095
expected_source_types: cve
expected_verdicts: confirmed
expected_binary_label: positive
expected_evidence_terms: "src/purify.ts, dist/purify.js"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-v8jm-5vwx-cfxm-gap-fill-000-vulnerable
+ 1. input_id: calibration-6228a29e06832095
title: GHSA-v8jm-5vwx-cfxm / CVE-2025-15599
advisory ids: GHSA-v8jm-5vwx-cfxm, CVE-2025-15599
weakness: xss
severity: moderate
affected language: JavaScript
anchor locations: src/purify.ts:1385, dist/purify.js:1130
- fix evidence: https://github.com/cure53/DOMPurify/commit/c861f5a83fb8d90800f1680f855fee551161ac2b
- repository state: checked out at c361baa18dbdcb3344a41110f4c48ad85bf48f80. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration fixed: oss-dompurify-ghsa-v8jm-5vwx-cfxm"
metadata:
- case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-fixed
+ case_id: calibration-6a431e1b299f88f3
suite: calibration-oss
calibration_case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm
calibration_variant: fixed
repo: "cure53/DOMPurify"
expected_binary_label: negative
vars:
- case_id: oss-dompurify-ghsa-v8jm-5vwx-cfxm-fixed
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-dompurify-ghsa-v8jm-5vwx-cfxm/fixed
+ case_id: calibration-6a431e1b299f88f3
+ calibration_repo: calibration-6a431e1b299f88f3
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-v8jm-5vwx-cfxm-gap-fill-000-fixed
+ expected_ids: calibration-6a431e1b299f88f3
expected_source_types: cve
expected_verdicts: not_actionable
expected_binary_label: negative
- expected_evidence_terms: "src/purify.ts, dist/purify.js, c861f5a83fb8d90800f1680f855fee551161ac2b"
+ expected_evidence_terms: "src/purify.ts, dist/purify.js"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-v8jm-5vwx-cfxm-gap-fill-000-fixed
+ 1. input_id: calibration-6a431e1b299f88f3
title: GHSA-v8jm-5vwx-cfxm / CVE-2025-15599
advisory ids: GHSA-v8jm-5vwx-cfxm, CVE-2025-15599
weakness: xss
severity: moderate
affected language: JavaScript
anchor locations: src/purify.ts:1385, dist/purify.js:1130
- fix evidence: https://github.com/cure53/DOMPurify/commit/c861f5a83fb8d90800f1680f855fee551161ac2b
- repository state: checked out at c861f5a83fb8d90800f1680f855fee551161ac2b. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration vulnerable: oss-monetr-ghsa-29v9-frvh-c426"
metadata:
- case_id: oss-monetr-ghsa-29v9-frvh-c426-vulnerable
+ case_id: calibration-4ceeb56f4c1b0b23
suite: calibration-oss
calibration_case_id: oss-monetr-ghsa-29v9-frvh-c426
calibration_variant: vulnerable
repo: "monetr/monetr"
expected_binary_label: positive
vars:
- case_id: oss-monetr-ghsa-29v9-frvh-c426-vulnerable
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/vulnerable
+ case_id: calibration-4ceeb56f4c1b0b23
+ calibration_repo: calibration-4ceeb56f4c1b0b23
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-29v9-frvh-c426-000-vulnerable
+ expected_ids: calibration-4ceeb56f4c1b0b23
expected_source_types: cve
expected_verdicts: confirmed
expected_binary_label: positive
expected_evidence_terms: "compose/monetr.yaml, server/controller/lunch_flow.go, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-29v9-frvh-c426-000-vulnerable
+ 1. input_id: calibration-4ceeb56f4c1b0b23
title: GHSA-29v9-frvh-c426 / CVE-2026-41644
advisory ids: GHSA-29v9-frvh-c426, CVE-2026-41644
weakness: ssrf
severity: high
affected language: YAML/Go/TypeScript
anchor locations: compose/monetr.yaml:34, server/controller/lunch_flow.go:100, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go:213
- fix evidence: https://github.com/monetr/monetr/commit/c260caa3c573a4a396ec2d264c7641a5d958385b
- repository state: checked out at 195a0c94cad26105dc1e9ee9d5ed902af41ed46c. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.
- description: "calibration fixed: oss-monetr-ghsa-29v9-frvh-c426"
metadata:
- case_id: oss-monetr-ghsa-29v9-frvh-c426-fixed
+ case_id: calibration-e364368948a91896
suite: calibration-oss
calibration_case_id: oss-monetr-ghsa-29v9-frvh-c426
calibration_variant: fixed
repo: "monetr/monetr"
expected_binary_label: negative
vars:
- case_id: oss-monetr-ghsa-29v9-frvh-c426-fixed
- target_repo: evals/triage-finding/artifacts/calibration-repos/oss-monetr-ghsa-29v9-frvh-c426/fixed
+ case_id: calibration-e364368948a91896
+ calibration_repo: calibration-e364368948a91896
+ calibration_repo_root: ""
source_type_under_test: cve
- expected_ids: GHSA-29v9-frvh-c426-000-fixed
+ expected_ids: calibration-e364368948a91896
expected_source_types: cve
expected_verdicts: not_actionable
expected_binary_label: negative
- expected_evidence_terms: "compose/monetr.yaml, server/controller/lunch_flow.go, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go, c260caa3c573a4a396ec2d264c7641a5d958385b"
+ expected_evidence_terms: "compose/monetr.yaml, server/controller/lunch_flow.go, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go"
finding_input: |-
Source type: cve
- 1. input_id: GHSA-29v9-frvh-c426-000-fixed
+ 1. input_id: calibration-e364368948a91896
title: GHSA-29v9-frvh-c426 / CVE-2026-41644
advisory ids: GHSA-29v9-frvh-c426, CVE-2026-41644
weakness: ssrf
severity: high
affected language: YAML/Go/TypeScript
anchor locations: compose/monetr.yaml:34, server/controller/lunch_flow.go:100, server/datasources/lunch_flow/lunch_flow_jobs/sync_lunch_flow.go:213
- fix evidence: https://github.com/monetr/monetr/commit/c260caa3c573a4a396ec2d264c7641a5d958385b
- repository state: checked out at c260caa3c573a4a396ec2d264c7641a5d958385b. Triage whether the original finding affects this exact state.
eval_instructions: |-
This is an automated OSS calibration eval. Do not ask follow-up questions.
- Inspect only the supplied repository checkout, the named anchor locations, the fix evidence, and the smallest related static evidence needed for the verdict.
+ Inspect only the supplied repository checkout, the named anchor locations and the smallest related static evidence needed for the verdict.
Do not spawn subagents, run tests, run builds, start applications, run exploit PoCs, modify files, or search for unrelated vulnerabilities.
Return the normal triage-finding result: concise Markdown plus exactly one fenced JSON block.
The JSON block must conform to schema_version "triage-finding/v0" and include source_type, verdict, evidence, counterevidence, proof_gaps, boundary_assessment, and exploitability_stack_rank.