OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security
TypeScript ai-securityapplication-securityclicode-scanningcodexcodex-securitycybersecuritydevsecopsnodejsnpmopenaisdksecuritytypescriptvulnerability-scanning
fix(scan): restore concrete discovery guidance for Standard and Deep scans (#2246)
Michael D'Angelo6942ebf3ef9e
diff --git a/evals/standard-time-budget/README.md b/evals/standard-time-budget/README.mdindex d7666ac7c..dc7a2e2a3 100644--- a/evals/standard-time-budget/README.md+++ b/evals/standard-time-budget/README.md@@ -50,8 +50,11 @@ traversal, reflected HTML, and server-side request forgery (SSRF). Each casehas a safe control. The SSRF pair checks whether a tenant can choose thedestination of a server-side status request. The billing route accepts onlythe two approved URLs. Both routes return the status of the first response,-including redirects. The model receives source and deployment context. The-evaluator holds the gold labels and locations.+including redirects. The statement-download routes both confine file paths,+but only one derives the tenant directory from the session. The grant routes+both verify a signed tenant grant. One consumes the first submitted element+and the other uses the verified element. The model receives source and+deployment context. The evaluator holds the gold labels and locations.Each arm runs in an opaque temporary directory outside the artifact tree; pairand variant labels appear only in evaluator reports and archives. After the@@ -62,6 +65,36 @@ are not next to the model's working directory or included in prompts. This is lasecurity boundary: production API-key authentication and root-read permissionsremain unchanged. This eval does not test resumption or external publication.+## Focused investigator exercise++Materialize the application source in a fresh directory from the repository root:++```sh+node --experimental-strip-types --input-type=module <<'JS'+import { mkdtemp } from "node:fs/promises";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { writeFixture } from "./evals/secret-discovery/fixtures.mts";+import { createFixture } from "./evals/standard-time-budget/fixtures.mts";+const dir = await mkdtemp(join(tmpdir(), "portal-review-"));+await writeFixture(dir, createFixture());+console.log(dir);+JS+```++Start a fresh read-only Codex session in the printed directory. Supply the **Shared Worker+Instructions** and **Focused Investigator Prompt** blocks from the revision of+[`core-scan.md`](../../plugins/codex-security/references/core-scan.md) under test,+then give the investigator this single packet:++> Review `src/statements.py`, using `README.md` and `src/app.py` for context. Can a download request escape the resolved statement directory by path traversal?++Grade without the revision label. Both routes contain the path. Credit the+independent finding if it connects the request-selected tenant to another+tenant’s private statement and distinguishes the session-selected sibling.+For comparisons, keep the fixture, model, effort and packet fixed and record+incomplete attempts separately.+## ResultsArtifacts are saved under ignored `reports/`, updated after every attempt:@@ -82,11 +115,21 @@ scan archival, cleanup, and grading, but not subsequent evaluator JSON writing.A canonical scan that misses a bug still completed successfully. The runnerexits nonzero for execution failures, not low recall.-The grader counts unique gold cases with matching CWE, a `root_control` or-`sink` source location, and authentic source evidence. Other or missing roles-do not earn automatic recall; unmatched findings require adjudication. It-consumes the production contract's repository-relative POSIX paths. Duplicates-do not increase recall.+Automatic recall uses a finite set of accepted CWEs and source locations for+each case. A `root_control` or `sink` location needs authentic source evidence.+When an affected location omits its role or identifies an entrypoint+(`entrypoint` or `entrypoint/wrapper`), an+authentic `root_control`, `sink`, or `entrypoint` snippet must cover a gold line+within that location and the snippet's cited range. An unlabelled location on+a safe sibling is a comparison when verified `expected_control` evidence covers+it and no competing citation claims that line.+The statement routes send the file where they select the tenant.+For the grant case, the discarded verification result and the next line that+uses the unchecked tenant identify the vulnerable route. Both routes call the+same invoice query, so a citation to that query alone needs manual review.+A different CWE, another equivalent source location, or missing roles in both+the location and its evidence can leave a valid finding unmatched. Adjudicate+these reports before counting discovery misses. Duplicates do not increase recall.Safe-control overlaps, unmatched findings, and citation errors are separatesignals; overlaps and unmatched findings require adjudication, not automaticfalse-positive labels. Free-text location roles can flag benign context@@ -123,7 +166,7 @@ capture, and parent-plus-worker usage. CI reuses its host native build, buildsthe plugin with `build_mcp_app.mjs --native host`, and links it at`sdk/typescript/_bundled_plugin`; universal release artifacts are not required.-A second integration test sends synthetic findings for the five fixture cases+A second integration test sends synthetic findings for every positive fixture casethrough the shipped draft tool and the real workbench. It checks that an emptyterminal draft preserves previously accepted findings, source evidence, andvalidation details in the sealed scan. Each accepted finding must appear indiff --git a/evals/standard-time-budget/fixtures.mts b/evals/standard-time-budget/fixtures.mtsindex 162f668fe..5285fe89b 100644--- a/evals/standard-time-budget/fixtures.mts+++ b/evals/standard-time-budget/fixtures.mts@@ -2,6 +2,7 @@ export interface SourceCase {id: string;path: string;line: number;+ endLine?: number;cwes: string[];}@@ -15,7 +16,7 @@ export function createFixture(): Fixture {const files: Record<string, string> = {"README.md": `# Account portal-This Python Flask service serves authenticated tenant accounts. Both the account+This Python Flask 3.1 service serves authenticated tenant accounts. Both the accountand billing routes are currently supported. The sign-in service creates signedFlask sessions containing a server-selected integer tenant_id; clients cannotchoose that value. The session signing key comes from deployment configuration.@@ -28,6 +29,16 @@ Export downloads are available to all authenticated users, but only files underdeployment-managed; account users cannot create or modify files there. Otherserver files, including configuration, must remain private.+Private monthly statements are stored separately under+/srv/portal/statements/<tenant_id>/monthly.csv. A signed-in user may download+only their own tenant's statements. Tenant numbers are present in the account+directory. The operator manages the statement directories and their contents.++Account users also receive short-lived signed report grants for their tenant.+Only the report issuer and the portal hold the report signing key. When a+client presents a pair of grants, the portal may release invoices only for+a tenant covered by a valid grant.+Preview endpoints render user-provided titles in HTML for the signed-in browser.The deployment does not add an HTML sanitizer or Content-Security-Policy header.The database and export files are deployment resources, not checked-in fixtures.@@ -43,10 +54,12 @@ import sqlite3from flask import Flask, abort, g, sessionfrom .downloads import downloads+from .grants import grantsfrom .invoices import invoicesfrom .partners import partnersfrom .preview import previewfrom .search import search+from .statements import statementsapp = Flask(__name__)app.secret_key = os.environ["SESSION_SIGNING_KEY"]@@ -76,6 +89,8 @@ app.register_blueprint(search)app.register_blueprint(downloads)app.register_blueprint(preview)app.register_blueprint(partners)+app.register_blueprint(statements)+app.register_blueprint(grants)`,"src/__init__.py": "","src/invoices.py": `from flask import Blueprint, abort, g, jsonify@@ -148,6 +163,88 @@ def billing_download():if not target.is_relative_to(root):abort(403)return send_file(target, as_attachment=True)+`,+ "src/statements.py": `from pathlib import Path++from flask import Blueprint, abort, g, send_file++statements = Blueprint("statements", __name__)+STATEMENT_ROOT = Path("/srv/portal/statements")+++def statement_path(tenant_id, name):+ root = (STATEMENT_ROOT / str(tenant_id)).resolve()+ target = (root / name).resolve()+ if not target.is_relative_to(root):+ abort(403)+ return target++[email protected]("/account/statements/<int:tenant_id>/<path:name>")+def account_statement(tenant_id, name):+ return send_file(statement_path(tenant_id, name), as_attachment=True)++[email protected]("/billing/statements/<path:name>")+def billing_statement(name):+ return send_file(statement_path(g.tenant_id, name), as_attachment=True)+`,+ "src/grants.py": `import os++from flask import Blueprint, abort, g, jsonify, request+from itsdangerous import BadData, URLSafeTimedSerializer++grants = Blueprint("grants", __name__)+signer = URLSafeTimedSerializer(os.environ["REPORT_SIGNING_KEY"], salt="tenant-report")+++def submitted_grants():+ request.max_content_length = 4096+ envelope = request.get_json()+ batch = envelope.get("grants") if isinstance(envelope, dict) else None+ if not isinstance(batch, list) or len(batch) != 2:+ abort(400)+ for grant in batch:+ if not isinstance(grant, dict):+ abort(400)+ tenant = grant.get("tenant_id")+ proof = grant.get("proof")+ if type(tenant) is not int or not 0 < tenant < 2**63:+ abort(400)+ if not isinstance(proof, str) or not proof.isascii():+ abort(400)+ return batch+++def verified_grant(batch):+ for grant in batch:+ try:+ tenant = signer.loads(grant["proof"], max_age=300)+ except BadData:+ continue+ if tenant == grant["tenant_id"]:+ return grant+ abort(403)+++def invoice_report(tenant_id):+ rows = g.database.execute(+ "SELECT id, recipient, total FROM invoices WHERE tenant_id = ?", (tenant_id,)+ ).fetchall()+ return jsonify([dict(row) for row in rows])++[email protected]("/account/report")+def account_report():+ batch = submitted_grants()+ verified_grant(batch)+ return invoice_report(batch[0]["tenant_id"])++[email protected]("/billing/report")+def billing_report():+ grant = verified_grant(submitted_grants())+ return invoice_report(grant["tenant_id"])`,"src/preview.py": `from html import escape@@ -212,13 +309,28 @@ CREATE TABLE catalog (cwes: string[],occurrence = 0,): SourceCase => {- const matchingLines = files[path]- .split("\n")- .flatMap((line, index) => (line.includes(marker) ? [index + 1] : []));+ const lines = files[path].split("\n");+ const markers = marker.split("\n");+ const matchingLines = lines.flatMap((_, index) =>+ markers.every((part, offset) => lines[index + offset]?.includes(part))+ ? [index + 1]+ : [],+ );const line = matchingLines[occurrence];if (!line) throw new Error(`Fixture marker missing: ${id}`);- return { id, path, line, cwes };+ const expected: SourceCase = { id, path, line, cwes };+ if (markers.length > 1) expected.endLine = line + markers.length - 1;+ return expected;};+ const tenantAccessCwes = ["CWE-639", "CWE-862", "CWE-863", "CWE-284"];+ const grantCwes = [+ ...tenantAccessCwes,+ "CWE-347",+ "CWE-345",+ "CWE-287",+ "CWE-349",+ "CWE-252",+ ];return {files,positives: [@@ -226,7 +338,7 @@ CREATE TABLE catalog ("invoice-tenant-isolation","src/invoices.py",'FROM invoices WHERE id = ?"',- ["CWE-639", "CWE-862", "CWE-863", "CWE-284"],+ tenantAccessCwes,),sourceCase("catalog-sql-injection", "src/search.py", "query = f", ["CWE-89",@@ -249,13 +361,25 @@ CREATE TABLE catalog ('requests.head(request.args["url"]',["CWE-918"],),+ sourceCase(+ "statement-tenant-selection",+ "src/statements.py",+ "send_file(statement_path(tenant_id, name)",+ tenantAccessCwes,+ ),+ sourceCase(+ "report-unverified-grant",+ "src/grants.py",+ 'verified_grant(batch)\nreturn invoice_report(batch[0]["tenant_id"])',+ grantCwes,+ ),],controls: [sourceCase("invoice-scoped-query","src/invoices.py","AND tenant_id = ?",- ["CWE-639", "CWE-862", "CWE-863", "CWE-284"],+ tenantAccessCwes,),sourceCase("catalog-bound-parameters", "src/search.py", "name LIKE ?", ["CWE-89",@@ -279,6 +403,24 @@ CREATE TABLE catalog ("requests.head(url,",["CWE-918"],),+ sourceCase(+ "statement-session-tenant",+ "src/statements.py",+ "send_file(statement_path(g.tenant_id, name)",+ tenantAccessCwes,+ ),+ sourceCase(+ "statement-contained-path",+ "src/statements.py",+ "if not target.is_relative_to(root):",+ ["CWE-22", "CWE-23", "CWE-36", "CWE-73"],+ ),+ sourceCase(+ "report-verified-grant",+ "src/grants.py",+ "return invoice_report(grant[",+ grantCwes,+ ),],};}diff --git a/evals/standard-time-budget/grade.mts b/evals/standard-time-budget/grade.mtsindex 985782d97..28a8f77c9 100644--- a/evals/standard-time-budget/grade.mts+++ b/evals/standard-time-budget/grade.mts@@ -9,7 +9,10 @@ import type { Fixture, SourceCase } from "./fixtures.mts";export type SourceLocation = FindingLocation;type Finding = Pick<ScanFinding, "title" | "locations"> & {taxonomy: Pick<ScanFinding["taxonomy"], "cwe">;- codeEvidence?: Pick<FindingCodeEvidence, "path" | "startLine" | "code">[];+ codeEvidence?: Pick<+ FindingCodeEvidence,+ "path" | "startLine" | "endLine" | "code" | "role"+ >[];};export interface Result {@@ -25,6 +28,11 @@ interface EvidenceError {reason: string;}+interface EvidenceCitation {+ declared: SourceLocation;+ verified?: SourceLocation;+}+export type GradeReport = ReturnType<typeof gradeResult>;function validRange(location: SourceLocation, fixture: Fixture) {@@ -43,7 +51,7 @@ function validRange(location: SourceLocation, fixture: Fixture) {function overlaps(location: SourceLocation, expected: SourceCase) {return (posix.normalize(location.path) === expected.path &&- location.startLine <= expected.line &&+ location.startLine <= (expected.endLine ?? expected.line) &&(location.endLine ?? location.startLine) >= expected.line);}@@ -52,23 +60,61 @@ function matches(finding: Finding,expected: SourceCase,fixture: Fixture,+ citations: EvidenceCitation[],requirePrimaryRole: boolean,) {- return (- finding.taxonomy.cwe.some((cwe) =>+ if (+ !finding.taxonomy.cwe.some((cwe) =>expected.cwes.includes(String(cwe).trim().toUpperCase()),- ) &&- finding.locations.some(- (location) =>- (requirePrimaryRole- ? /^(root_control|sink)$/u.test(location.role ?? "")- : !/^(supporting|support|context|consumer|expected_control)$/u.test(- location.role ?? "",- )) &&- validRange(location, fixture) &&- overlaps(location, expected),)- );+ )+ return false;++ return finding.locations.some((location) => {+ if (!validRange(location, fixture) || !overlaps(location, expected))+ return false;+ if (location.role) {+ if (!requirePrimaryRole) {+ return !/^(supporting|support|context|consumer|expected_control)$/u.test(+ location.role,+ );+ }+ if (location.role === "root_control" || location.role === "sink")+ return true;+ if (+ location.role !== "entrypoint" &&+ location.role !== "entrypoint/wrapper"+ )+ return false;+ }++ // The evidence must identify the role on a gold line the location cites.+ const cited = {+ ...expected,+ line: Math.max(expected.line, location.startLine),+ endLine: Math.min(+ expected.endLine ?? expected.line,+ location.endLine ?? location.startLine,+ ),+ };+ const verified = citations.filter(+ ({ verified }) => verified && overlaps(verified, cited),+ );+ if (requirePrimaryRole) {+ return verified.some(+ ({ declared: { role } }) =>+ role === "entrypoint" || role === "root_control" || role === "sink",+ );+ }+ const comparison = verified.some(+ ({ declared }) => declared.role === "expected_control",+ );+ const competing = citations.some(+ ({ declared }) =>+ declared.role !== "expected_control" && overlaps(declared, cited),+ );+ return !comparison || competing;+ });}function inspectEvidence(@@ -78,6 +124,7 @@ function inspectEvidence() {const entries = finding.codeEvidence ?? [];const ranges: SourceLocation[] = [];+ const citations: EvidenceCitation[] = [];const errors: EvidenceError[] = [];if (!entries.length) {errors.push({ findingIndex, reason: "missing code evidence" });@@ -88,6 +135,10 @@ function inspectEvidence(...entry,endLine: entry.startLine + code.split("\n").length - 1,};+ const citation: EvidenceCitation = {+ declared: { ...entry, endLine: entry.endLine ?? range.endLine },+ };+ citations.push(citation);const source = fixture.files[posix.normalize(entry.path)];if (!code.trim() ||@@ -103,11 +154,21 @@ function inspectEvidence(evidenceIndex,reason: "code does not match source at the cited lines",});+ } else if (!validRange(entry, fixture)) {+ errors.push({+ findingIndex,+ evidenceIndex,+ reason: "code evidence range does not exist in the source",+ });} else {ranges.push(range);+ citation.verified = {+ ...range,+ endLine: Math.min(range.endLine, entry.endLine ?? range.endLine),+ };}}- return { ranges, errors };+ return { ranges, citations, errors };}export function gradeResult(result: Result, fixture: Fixture) {@@ -131,7 +192,7 @@ export function gradeResult(result: Result, fixture: Fixture) {}evidenceErrors.push(...evidence.errors);const candidates = fixture.positives.filter((expected) =>- matches(finding, expected, fixture, true),+ matches(finding, expected, fixture, evidence.citations, true),);const matched = candidates.filter((expected) => {const grounded = evidence.ranges.some((range) =>@@ -150,7 +211,7 @@ export function gradeResult(result: Result, fixture: Fixture) {matchCounts.set(expected.id, matchCounts.get(expected.id)! + 1);}const controls = fixture.controls.filter((expected) =>- matches(finding, expected, fixture, false),+ matches(finding, expected, fixture, evidence.citations, false),);if (controls.length) {safeControlDetails.push({diff --git a/evals/standard-time-budget/test_grading.mts b/evals/standard-time-budget/test_grading.mtsindex ffd04fce2..f5868b30d 100644--- a/evals/standard-time-budget/test_grading.mts+++ b/evals/standard-time-budget/test_grading.mts@@ -1,31 +1,46 @@import assert from "node:assert/strict";import test from "node:test";import { createFixture, type Fixture, type SourceCase } from "./fixtures.mts";-import { gradeResult, type SourceLocation } from "./grade.mts";+import { gradeResult, type Result, type SourceLocation } from "./grade.mts";function findingFor(fixture: Fixture,expected: SourceCase,role = "root_control",) {+ const codeEvidence: NonNullable<Result["findings"][number]["codeEvidence"]> =+ [+ {+ path: expected.path,+ startLine: expected.line,+ code: fixture.files[expected.path].split("\n")[expected.line - 1],+ },+ ];return {title: expected.id,taxonomy: { cwe: [expected.cwes[0]] },locations: [{ path: expected.path, startLine: expected.line, role },] as SourceLocation[],- codeEvidence: [- {- path: expected.path,- startLine: expected.line,- code: fixture.files[expected.path].split("\n")[expected.line - 1],- },- ],+ codeEvidence,};}type FixtureFinding = ReturnType<typeof findingFor>;+function findingAt(+ fixture: Fixture,+ path: string,+ source: string,+ cwe: string,+) {+ const line =+ fixture.files[path].split("\n").findIndex((row) => row.includes(source)) ++ 1;+ assert.ok(line > 0, `Missing fixture source: ${source}`);+ return findingFor(fixture, { id: source, path, line, cwes: [cwe] }, "sink");+}+function completeResult(fixture: Fixture) {return {findings: fixture.positives.map((expected) =>@@ -56,7 +71,10 @@ test("missing SSRF reduces recall even when every other case is found", () => {report.cases.filter((entry) => !entry.found),[{ id: "partner-status-ssrf", found: false }],);- assert.equal(report.recall, 4 / 5);+ assert.equal(+ report.recall,+ (fixture.positives.length - 1) / fixture.positives.length,+ );assert.equal(report.passed, false);assert.deepEqual(report.unmatchedFindings, []);});@@ -64,26 +82,24 @@ test("missing SSRF reduces recall even when every other case is found", () => {test("SSRF earns credit at the user-selected destination and flags the allowlisted destination for review", () => {const fixture = createFixture();const path = "src/partners.py";- function findingAt(sink: string) {- const line =- fixture.files[path].split("\n").findIndex((row) => row.includes(sink)) +- 1;- assert.ok(line > 0, `Missing sink: ${sink}`);- return findingFor(- fixture,- { id: "Outbound status request", path, line, cwes: ["CWE-918"] },- "sink",- );- }const positive = gradeResult(- { findings: [findingAt('requests.head(request.args["url"]')] },+ {+ findings: [+ findingAt(+ fixture,+ path,+ 'requests.head(request.args["url"]',+ "CWE-918",+ ),+ ],+ },fixture,);assert.deepEqual(positive.matchedIds, ["partner-status-ssrf"]);assert.equal(positive.safeControlFindings, 0);const control = gradeResult(- { findings: [findingAt("requests.head(url,")] },+ { findings: [findingAt(fixture, path, "requests.head(url,", "CWE-918")] },fixture,);assert.deepEqual(control.matchedIds, []);@@ -94,6 +110,271 @@ test("SSRF earns credit at the user-selected destination and flags the allowlistassert.equal(control.needsAdjudication, true);});+test("path containment and tenant selection are independent controls", () => {+ const fixture = createFixture();+ const path = "src/statements.py";+ const callerSelected = "send_file(statement_path(tenant_id, name)";+ const report = gradeResult(+ {+ findings: [+ findingAt(fixture, path, callerSelected, "CWE-639"),+ findingAt(+ fixture,+ path,+ "send_file(statement_path(g.tenant_id, name)",+ "CWE-639",+ ),+ findingAt(+ fixture,+ path,+ "if not target.is_relative_to(root):",+ "CWE-22",+ ),+ ],+ },+ fixture,+ );+ assert.deepEqual(report.matchedIds, ["statement-tenant-selection"]);+ assert.deepEqual(+ report.safeControlDetails.map((entry) => entry.controlIds),+ [["statement-session-tenant"], ["statement-contained-path"]],+ );+ const traversal = gradeResult(+ { findings: [findingAt(fixture, path, callerSelected, "CWE-22")] },+ fixture,+ );+ assert.deepEqual(traversal.matchedIds, []);+ assert.equal(traversal.unmatchedFindings.length, 1);+});++test("grant verification distinguishes the selected tenant from the verified tenant", () => {+ const fixture = createFixture();+ const path = "src/grants.py";+ for (const cwe of ["CWE-639", "CWE-862", "CWE-349", "CWE-252"]) {+ const report = gradeResult(+ {+ findings: [+ findingAt(+ fixture,+ path,+ 'return invoice_report(batch[0]["tenant_id"])',+ cwe,+ ),+ findingAt(+ fixture,+ path,+ 'return invoice_report(grant["tenant_id"])',+ cwe,+ ),+ findingAt(+ fixture,+ path,+ "SELECT id, recipient, total FROM invoices WHERE tenant_id",+ cwe,+ ),+ findingAt(+ fixture,+ path,+ 'tenant = signer.loads(grant["proof"],',+ cwe,+ ),+ ],+ },+ fixture,+ );+ assert.deepEqual(report.matchedIds, ["report-unverified-grant"]);+ assert.deepEqual(+ report.safeControlDetails.map((entry) => entry.controlIds),+ [["report-verified-grant"]],+ );+ assert.equal(report.unmatchedFindings.length, 2);+ assert.equal(report.needsAdjudication, true);+ }+});++test("the discarded verification result is a grounded root control for the grant finding", () => {+ const fixture = createFixture();+ const path = "src/grants.py";+ const finding = findingAt(+ fixture,+ path,+ " verified_grant(batch)",+ "CWE-862",+ );+ finding.locations[0].role = "root_control";+ const grade = () => gradeResult({ findings: [finding] }, fixture);+ assert.deepEqual(grade().matchedIds, ["report-unverified-grant"]);++ finding.locations[0].role = "supporting";+ assert.deepEqual(grade().matchedIds, []);+ finding.locations[0].role = "root_control";+ finding.codeEvidence = findingAt(+ fixture,+ path,+ " batch = submitted_grants()",+ "CWE-863",+ ).codeEvidence;+ assert.deepEqual(grade().matchedIds, []);+ assert.equal(grade().evidenceErrors.length, 1);+});++test("an unlabelled affected location can use the role on authentic evidence at the same gold line", () => {+ const fixture = createFixture();+ const finding = findingAt(+ fixture,+ "src/grants.py",+ " verified_grant(batch)",+ "CWE-863",+ );+ const [location] = finding.locations;+ const [snippet] = finding.codeEvidence;+ const grade = () => gradeResult({ findings: [finding] }, fixture);+ delete location.role;+ for (const role of ["root_control", "sink", "entrypoint"]) {+ snippet.role = role;+ assert.deepEqual(grade().matchedIds, ["report-unverified-grant"]);+ }+ for (const role of ["entrypoint", "entrypoint/wrapper"]) {+ location.role = role;+ assert.deepEqual(grade().matchedIds, ["report-unverified-grant"]);+ }+ delete location.role;+ for (const role of [+ undefined,+ "expected_control",+ "supporting",+ "propagation",+ ]) {+ snippet.role = role;+ assert.deepEqual(grade().matchedIds, []);+ }++ // Both lines are gold. A role on one cannot qualify a citation to the other.+ snippet.role = "sink";+ location.startLine++;+ assert.deepEqual(grade().matchedIds, []);+ location.startLine--;+ location.role = "supporting";+ assert.deepEqual(grade().matchedIds, []);+ delete location.role;+ snippet.code = "invented code";+ const fabricated = grade();+ assert.deepEqual(fabricated.matchedIds, []);+ assert.equal(fabricated.evidenceErrors.length, 1);+});++test("a safe sibling cited as a comparison stays clear, while an accusation is flagged", () => {+ const fixture = createFixture();+ const path = "src/statements.py";+ const finding = findingAt(+ fixture,+ path,+ "send_file(statement_path(tenant_id, name)",+ "CWE-639",+ );+ const safe = findingAt(+ fixture,+ path,+ "send_file(statement_path(g.tenant_id, name)",+ "CWE-639",+ );+ delete finding.locations[0].role;+ finding.codeEvidence[0].role = "entrypoint";+ finding.locations.push(...safe.locations);+ finding.codeEvidence.push(...safe.codeEvidence);+ const compare = (evidenceRole?: string, locationRole?: string) => {+ safe.locations[0].role = locationRole;+ safe.codeEvidence[0].role = evidenceRole;+ return gradeResult({ findings: [finding] }, fixture);+ };+ const comparison = compare("expected_control");+ assert.deepEqual(comparison.matchedIds, ["statement-tenant-selection"]);+ assert.deepEqual(comparison.safeControlDetails, []);+ finding.locations.push({+ path: "src/app.py",+ startLine: 999,+ role: "supporting",+ });+ const unrelatedError = compare("expected_control");+ assert.equal(unrelatedError.evidenceErrors.length, 1);+ assert.deepEqual(unrelatedError.safeControlDetails, []);+ finding.locations.pop();+ for (const report of [+ compare("sink"),+ compare("supporting"),+ compare("consumer"),+ compare(undefined),+ compare("expected_control", "root_control"),+ ]) {+ assert.deepEqual(+ report.safeControlDetails.map(({ controlIds }) => controlIds),+ [["statement-session-tenant"]],+ );+ assert.equal(report.needsAdjudication, true);+ }+ const conflictSnippet = {+ ...safe.codeEvidence[0],+ role: "sink",+ endLine: fixture.files[path].split("\n").length + 1,+ };+ finding.codeEvidence.push(conflictSnippet);+ const conflict = compare("expected_control");+ assert.equal(conflict.passed, false);+ assert.equal(conflict.evidenceErrors.length, 1);+ assert.deepEqual(+ conflict.safeControlDetails.flatMap(({ controlIds }) => controlIds),+ ["statement-session-tenant"],+ );+ conflictSnippet.endLine = conflictSnippet.startLine;+ conflictSnippet.startLine--;+ conflictSnippet.code =+ fixture.files[path].split("\n")[conflictSnippet.startLine - 1];+ const partialQuote = compare("expected_control");+ assert.deepEqual(partialQuote.evidenceErrors, []);+ assert.deepEqual(+ partialQuote.safeControlDetails.flatMap(({ controlIds }) => controlIds),+ ["statement-session-tenant"],+ );+});++test("a snippet cannot assign roles to trailing context outside its declared span", () => {+ const fixture = createFixture();+ const findingWithContext = (expected: SourceCase, role: string) => {+ const finding = findingFor(fixture, expected);+ delete finding.locations[0].role;+ const [snippet] = finding.codeEvidence;+ snippet.role = role;+ snippet.startLine--;+ snippet.endLine = snippet.startLine;+ snippet.code = fixture.files[snippet.path]+ .split("\n")+ .slice(snippet.startLine - 1, expected.line)+ .join("\n");+ return finding;+ };+ const [expected] = fixture.positives;+ const finding = findingWithContext(expected, "root_control");+ const grade = () => gradeResult({ findings: [finding] }, fixture);+ assert.deepEqual(grade().matchedIds, []);+ finding.locations[0].role = "sink";+ assert.deepEqual(grade().matchedIds, [expected.id]);+ delete finding.locations[0].role;+ const [snippet] = finding.codeEvidence;+ delete snippet.endLine;+ assert.deepEqual(grade().matchedIds, [expected.id]);+ snippet.endLine = expected.line;+ snippet.code = snippet.code.split("\n")[0];+ assert.deepEqual(grade().matchedIds, []);+ const [control] = fixture.controls;+ const comparison = findingWithContext(control, "expected_control");+ assert.deepEqual(+ gradeResult({ findings: [comparison] }, fixture).safeControlDetails.flatMap(+ ({ controlIds }) => controlIds,+ ),+ [control.id],+ );+});+test("additional fabricated locations invalidate an otherwise grounded finding", () => {const fixture = createFixture();for (const role of ["root_control", "supporting"]) {diff --git a/evals/standard-time-budget/test_validation_survival.mts b/evals/standard-time-budget/test_validation_survival.mtsindex 813d3618b..856a89cf3 100644--- a/evals/standard-time-budget/test_validation_survival.mts+++ b/evals/standard-time-budget/test_validation_survival.mts@@ -27,7 +27,7 @@ async function readCheckpoint(scanDir: string) {}for (const deferLast of [false, true]) {- test(`completion retains checkpoint findings and grades ${deferLast ? "only validated" : "all five"} cases`, async (t) => {+ test(`completion retains checkpoint findings and grades ${deferLast ? "only validated" : "all"} cases`, async (t) => {const root = await temporaryDirectory("eval-validation-survival-", true);t.after(() => rm(root, { recursive: true, force: true }));const pluginRoot = await bundledPluginRoot();@@ -108,7 +108,7 @@ for (const deferLast of [false, true]) {): AsyncGenerator<ThreadEvent> {yield { type: "thread.started", thread_id: threadId };yield { type: "turn.started" };- // Discovery mentions all five candidates, including any deferred case.+ // Discovery mentions every candidate, including any deferred case.yield {type: "item.completed",item: {diff --git a/plugins/codex-security/references/core-scan.md b/plugins/codex-security/references/core-scan.mdindex 6be21d02d..a3c844018 100644--- a/plugins/codex-security/references/core-scan.md+++ b/plugins/codex-security/references/core-scan.md@@ -50,7 +50,14 @@ Return JSON with findings, resolved_questions, and reviewed_files. Include only## Baseline Auditor Prompt```markdown-Audit the repository's architecture, entry points, and security boundaries independently. Follow attacker-controlled input through effective controls to sensitive operations; investigate relevant vulnerability classes rather than relying on keyword matches. Include authentication, authorization, parsing, execution, data exposure, and resource or memory safety where the implementation exposes those boundaries.+Audit the repository's architecture, entry points, and security boundaries independently. Follow attacker-controlled input through effective controls to sensitive operations. Use the surfaces you encounter to choose which vulnerability classes to investigate.++- At object reads and state changes, check authentication, ownership, tenant isolation, and authorization differences between sibling operations.+- At query builders, renderers, and interpreters, check SQL or NoSQL injection, XSS, template or command injection, and unsafe deserialization.+- At file and network boundaries, check traversal, unsafe uploads, open redirects, SSRF, and sensitive data exposure.+- Check XML parsers for external entity resolution, HTTP intermediaries for request smuggling, object merges for prototype pollution, and attacker-controlled sizes or lifetimes for resource exhaustion or memory-safety errors.++Use these examples as starting points. Establish the reachable path, effective control, and consequence for each finding.Explicitly inspect embedded credentials and private keys, including configuration, documentation, tests, examples, and inactive code. Establish their purpose from context. Source-reader access to a credential with additional authority is sufficient to consider a disclosure; execution of its containing code is not required. Public material and demonstrable placeholders are not secrets. Unknown live validity limits confidence, not the evidence of exposure. Do not use the credential or contact a service.```@@ -58,7 +65,9 @@ Explicitly inspect embedded credentials and private keys, including configuratio## Focused Investigator Prompt```markdown-Investigate the assigned source-backed security questions. Packets are leads, not conclusions. Trace actual callers, representations, effective controls, sensitive consumers, and counterevidence. Check sibling operations and concrete implementations when they share the suspected control, preserving independently vulnerable paths. Continue beyond the first issue when assigned questions remain.+Investigate the assigned source-backed security questions. If evidence on the path points to a different broken control, follow that lead within the authorized scope and remaining budget, even if the original hypothesis has been ruled out. Record the disposition of the original question separately from any new finding. Check sibling operations and concrete implementations that share the suspected control, including differences in authorization or state transitions. Preserve independently vulnerable paths and continue beyond the first issue when assigned questions remain.++Trace forward from attacker-controlled input or backward from a sensitive operation. Identify the exact object, value, or destination the operation uses and, for an interpreter, the grammar it applies. Compare that with what the guard checked. Follow decoding, redirects, object selection, or state transitions when they can change the result, and verify the check occurs before the first sensitive effect.-Use forward dataflow, backward tracing from sensitive operations, or authorization/state-transition analysis as useful. Follow related evidence of embedded secrets using the source-disclosure standard: establish purpose and authority from context without testing credentials. Do not infer remote exposure, privileges, or missing controls that the source does not establish.+Follow related evidence of embedded secrets using the source-disclosure standard: establish purpose and authority from context without testing credentials. Do not infer remote exposure, privileges, or missing controls that the source does not establish.```diff --git a/sdk/typescript/src/custom-validation-prompt.ts b/sdk/typescript/src/custom-validation-prompt.tsindex 6d284a394..a1b5a9e3f 100644--- a/sdk/typescript/src/custom-validation-prompt.ts+++ b/sdk/typescript/src/custom-validation-prompt.ts@@ -10,6 +10,7 @@ import { PLUGIN_NAME } from "./runtime.js";const SOURCES = {"references/core-scan.md": ["7690b6c854de07ff15984f6d2c25ddfa1f6ae65f65876cadf2f904054b4f0679",+ "83bafdc4089803e631eaaa86ec57167c7da3f51794514d0dcdb62c5afdced8a7",],"skills/security-scan/SKILL.md": ["b6a69658d48fff6979d6690e95d2096da236bb7e51b96ebb4f92102a9863d66c",