Skip to content

OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security

feat(cli): add read-only security verify-fix (#566)

Kyle Brown05a364fa8707
diff --git a/sdk/typescript/README.md b/sdk/typescript/README.md
index 5079dd090..ee0ff79d5 100644
--- a/sdk/typescript/README.md
+++ b/sdk/typescript/README.md
@@ -251,6 +251,9 @@ npx @openai/codex-security publish scan /path/outside/repository/results --to li
npx @openai/codex-security publish scan --to linear --linear-team TEAM_ID
npx @openai/codex-security validate /path/outside/repository/findings.json "Possible SQL injection in src/query.ts:42"
npx @openai/codex-security validate "Possible SQL injection" --effort high
+npx @openai/codex-security verify-fix --linear-issue SEC-123 --json
+npx @openai/codex-security verify-fix --linear-project "Security backlog" --linear-filter '{"state":{"type":{"eq":"completed"}}}' --json
+npx @openai/codex-security verify-fix --scan SCAN_ID --severity high --json
npx @openai/codex-security patch /path/outside/repository/findings.json "Missing authorization check in src/routes.ts:18"
npx @openai/codex-security patch "Missing authorization check" --effort high
npx @openai/codex-security patch OCCURRENCE_ID
@@ -815,6 +818,23 @@ history. Imported content is always literal, and issue URLs must match the
selected workspace. Linear access is read-only, and its credentials are not
passed to the patch subprocess.
+Use `verify-fix` to check whether an existing security fix actually closes its
+original vulnerability without modifying the repository. Pass a finding
+description, a saved finding identifier, `--scan SCAN_ID`, `--linear-issue ISSUE`,
+or `--linear-project "PROJECT"`. Linear intake uses the same credentials and
+`--linear-filter` options as `patch`; explicitly filter for completed issues when
+verifying a finished backlog. Verification runs the bundled `verify-fix` skill
+in a read-only Codex sandbox and reports `fixed`, `still_vulnerable`, or
+`inconclusive` with supporting evidence for every finding. Use `--json` for
+structured output. Exit code `0` means every finding is fixed, `1` means at
+least one finding remains vulnerable, and `2` means a result is inconclusive or
+verification could not complete.
+
+Interactive verification shows the same live agent-activity dashboard used by
+scans and Linear publication. Reasoning, repository commands, and progress are
+written to stderr; redirected output and CI receive plain progress lines, and
+JSON results on stdout remain machine-readable.
+
Exit codes are `0` for a completed report-only scan or a passing policy, `1`
for a completed policy violation, `2` for invalid input, incomplete coverage, or
a runtime/export error, `130` for interruption, and `143` for termination.
diff --git a/sdk/typescript/_bundled_plugin/skills/verify-fix/SKILL.md b/sdk/typescript/_bundled_plugin/skills/verify-fix/SKILL.md
new file mode 100644
index 000000000..6cd5b221d
--- /dev/null
+++ b/sdk/typescript/_bundled_plugin/skills/verify-fix/SKILL.md
@@ -0,0 +1,44 @@
+---
+name: verify-fix
+description: Use when the user asks whether an existing security fix, patch, finding, or completed issue actually remediates the original vulnerability without modifying the repository. Do not use to validate candidate findings, implement patches, or run full repository scans.
+---
+
+# Verify Fix
+
+## Objective
+
+Determine whether each supplied security finding has been fixed in the current checkout. Operate in standalone verification-only mode; do not create, modify, or delete repository files, apply patches, commit changes, write artifacts, or modify issue trackers.
+
+## Assessment Method
+
+Use `../../references/static-finding-assessment.md` to identify the original attacker-controlled source, security control, sensitive sink, reachable path, trust boundary, counterevidence, and proof gaps. If the caller already supplied that reference in the prompt, use the supplied contents without reading it again.
+
+## Verification Workflow
+
+1. Establish the original vulnerability, its preconditions, affected security boundary, and legitimate behavior that must continue to work.
+2. Confirm the current checkout contains the affected component. Follow moved or refactored code rather than treating a missing file, removed line, or changed function name as proof of remediation.
+3. Trace the original exploit path through the current implementation and check the nearest relevant control, equivalent paths, and plausible bypasses.
+4. Run the original reproducer, focused regression checks, or legitimate-behavior checks only when they can run without modifying the repository. Preserve exact static evidence when runtime checks are unavailable.
+5. Return one result per supplied finding, in the requested order. Treat closed tickets, unrelated passing tests, and the absence of a new scan finding as insufficient proof.
+
+## Result Contract
+
+Return exactly one JSON object:
+
+```json
+{
+ "results": [
+ {
+ "id": "finding-or-issue-id",
+ "status": "fixed|still_vulnerable|inconclusive",
+ "evidence": "specific current source, exploit, test, or proof-gap evidence"
+ }
+ ]
+}
+```
+
+- Use `fixed` only when evidence proves the original security boundary is closed and legitimate behavior remains intact.
+- Use `still_vulnerable` only when evidence proves the original vulnerable path remains reachable.
+- Use `inconclusive` for a repository mismatch, missing original context, unavailable relevant checks, an unproven legitimate control, or another material proof gap.
+
+Never infer a stronger verdict by weakening the read-only boundary, substituting a different vulnerability, or hiding missing evidence.
diff --git a/sdk/typescript/_bundled_plugin/skills/verify-fix/agents/openai.yaml b/sdk/typescript/_bundled_plugin/skills/verify-fix/agents/openai.yaml
new file mode 100644
index 000000000..e6487201f
--- /dev/null
+++ b/sdk/typescript/_bundled_plugin/skills/verify-fix/agents/openai.yaml
@@ -0,0 +1,4 @@
+interface:
+ display_name: "Verify Fix"
+ short_description: "Verify security fixes without changing code"
+ default_prompt: "Use $verify-fix to verify that a reported security vulnerability is fixed without modifying the repository."
diff --git a/sdk/typescript/plugin-files.json b/sdk/typescript/plugin-files.json
index 653840987..668678608 100644
--- a/sdk/typescript/plugin-files.json
+++ b/sdk/typescript/plugin-files.json
@@ -107,6 +107,8 @@
"skills/validation/SKILL.md",
"skills/validation/agents/openai.yaml",
"skills/validation/references/validation-guidance.md",
+ "skills/verify-fix/SKILL.md",
+ "skills/verify-fix/agents/openai.yaml",
"skills/vulnerability-writeup/SKILL.md",
"skills/vulnerability-writeup/agents/openai.yaml",
"skills/vulnerability-writeup/references/report-format.md"
diff --git a/sdk/typescript/src/cli.ts b/sdk/typescript/src/cli.ts
index c4e332b73..19a79398c 100644
--- a/sdk/typescript/src/cli.ts
+++ b/sdk/typescript/src/cli.ts
@@ -548,6 +548,160 @@ class PublicationProgressPresenter {
}
}
+class VerificationProgressPresenter {
+ readonly #stream: Writable;
+ readonly #dependencies: CliDependencies;
+ readonly #repository: string;
+ readonly #total: number;
+ readonly #seenActivities = new Set<string>();
+ readonly #reasoning = new Map<string, string>();
+ #dashboard: ScanDashboard | null = null;
+
+ public constructor(
+ stream: Writable,
+ dependencies: CliDependencies,
+ repository: string,
+ total: number,
+ ) {
+ this.#stream = stream;
+ this.#dependencies = dependencies;
+ this.#repository = repository;
+ this.#total = total;
+ }
+
+ public start(): void {
+ if (
+ this.#stream.isTTY === true &&
+ this.#dependencies.environment["CI"] === undefined &&
+ this.#dependencies.environment["TERM"] !== "dumb"
+ ) {
+ const dashboard = new ScanDashboard(this.#stream, {
+ repository: this.#repository,
+ presentation: "verification",
+ clock: this.#dependencies,
+ color: this.#dependencies.environment["NO_COLOR"] === undefined,
+ sanitize: safeErrorMessage,
+ });
+ dashboard.setPublicationProgress(0, this.#total);
+ dashboard.setStage(`Verifying findings · 0/${this.#total}`);
+ try {
+ dashboard.start();
+ this.#dashboard = dashboard;
+ return;
+ } catch {
+ try {
+ dashboard.stop();
+ } catch {}
+ }
+ }
+
+ this.#write(
+ `Verifying ${this.#total} finding${this.#total === 1 ? "" : "s"} against the current checkout.`,
+ );
+ }
+
+ public observe(event: Readonly<Record<string, unknown>>): void {
+ const method = event["method"];
+ const params = event["params"];
+ if (
+ typeof params !== "object" ||
+ params === null ||
+ Array.isArray(params)
+ ) {
+ return;
+ }
+ const values = params as Record<string, unknown>;
+ let normalized: Record<string, unknown>;
+
+ if (method === "item/reasoning/summaryTextDelta") {
+ const id = values["itemId"];
+ const delta = values["delta"];
+ if (typeof id !== "string" || typeof delta !== "string") return;
+ const text = `${this.#reasoning.get(id) ?? ""}${delta}`;
+ this.#reasoning.set(id, text);
+ normalized = {
+ type: "item.updated",
+ item: { id, type: "reasoning", text },
+ };
+ } else if (method === "item/started" || method === "item/completed") {
+ const item = values["item"];
+ if (typeof item !== "object" || item === null || Array.isArray(item)) {
+ return;
+ }
+ const current = item as Record<string, unknown>;
+ const type = current["type"];
+ let converted: Record<string, unknown>;
+ if (type === "commandExecution") {
+ converted = { ...current, type: "command_execution" };
+ } else if (type === "mcpToolCall") {
+ converted = { ...current, type: "mcp_tool_call" };
+ } else if (type === "agentMessage") {
+ if (current["phase"] !== "commentary") return;
+ converted = { ...current, type: "agent_message" };
+ } else if (type === "reasoning") {
+ const summary = current["summary"];
+ const text = Array.isArray(summary)
+ ? summary
+ .filter((entry): entry is string => typeof entry === "string")
+ .join("\n")
+ : current["text"];
+ if (typeof text !== "string") return;
+ converted = { ...current, text };
+ } else {
+ return;
+ }
+ normalized = {
+ type: method === "item/started" ? "item.started" : "item.completed",
+ item: converted,
+ };
+ } else {
+ return;
+ }
+
+ for (const activity of scanActivitiesFromEvent(
+ normalized,
+ this.#repository,
+ )) {
+ if (this.#dashboard !== null) {
+ this.#dashboard.record(activity);
+ continue;
+ }
+ const key = `${activity.id}\0${activity.description}`;
+ if (this.#seenActivities.has(key)) continue;
+ this.#seenActivities.add(key);
+ const label =
+ activity.kind === "reasoning" || activity.kind === "message"
+ ? "Codex"
+ : "Tool";
+ this.#write(`${label}: ${activity.description}`);
+ }
+ }
+
+ public complete(results: readonly FindingVerification[]): void {
+ if (this.#dashboard === null) return;
+ this.#dashboard.setPublicationProgress(results.length, this.#total);
+ this.#dashboard.setStage(
+ `Verification complete · ${results.length}/${this.#total}`,
+ );
+ for (const result of results) {
+ this.#dashboard.note(`${result.status.toUpperCase()} ${result.id}`);
+ }
+ }
+
+ public stop(): void {
+ try {
+ this.#dashboard?.stop();
+ } catch {}
+ this.#dashboard = null;
+ }
+
+ #write(message: string): void {
+ try {
+ this.#stream.write(`${safePatchText(message)}\n`);
+ } catch {}
+ }
+}
+
function effortOption() {
return z
.enum(MODEL_REASONING_EFFORTS, {
@@ -730,10 +884,15 @@ type MatchingPlan = JsonObject & {
};
interface SkillCommandOutput {
- readonly command: "validate" | "patch";
+ readonly command: "validate" | "patch" | "verify-fix";
readonly stdout: Writable;
readonly stderr: Writable;
- readonly appServer?: { readonly directory: string; readonly prompt: string };
+ readonly appServer?: {
+ readonly directory: string;
+ readonly prompt: string;
+ readonly sandbox?: "read-only" | "workspace-write";
+ readonly onEvent?: (event: Readonly<Record<string, unknown>>) => void;
+ };
}
const findingPatchSchema = z.object({
@@ -746,10 +905,20 @@ const findingPatchSchema = z.object({
type FindingPatch = z.infer<typeof findingPatchSchema>;
+const findingVerificationSchema = z.object({
+ id: z.string(),
+ status: z.enum(["fixed", "still_vulnerable", "inconclusive"]),
+ evidence: z.string().trim().min(1),
+});
+
+type FindingVerification = z.infer<typeof findingVerificationSchema>;
+
interface SkillRunOptions {
directory?: string;
findings?: readonly Finding[];
findingInstructions?: Readonly<Record<string, string>>;
+ verificationIds?: readonly string[];
+ onEvent?: (event: Readonly<Record<string, unknown>>) => void;
provider?: string;
providerConfiguration?: JsonObject;
environment?: NodeJS.ProcessEnv;
@@ -1034,8 +1203,11 @@ export async function runCodexSkillCommand(
output.appServer === undefined
? undefined
: {
+ directory: output.appServer.directory,
prompt: output.appServer.prompt,
input: invocation.stdin!,
+ sandbox: output.appServer.sandbox,
+ onEvent: output.appServer.onEvent,
},
),
new Promise<undefined>((resolve) => {
@@ -1990,7 +2162,7 @@ export async function main(
});
const cli = Cli.create("codex-security", {
description:
- "Run, validate, patch, export, and publish Codex Security findings.",
+ "Run, validate, patch, verify fixes, export, and publish Codex Security findings.",
version: VERSION,
mcp: {
command: "npx --yes @openai/codex-security --mcp",
@@ -2607,6 +2779,223 @@ export async function main(
}
},
})
+ .command("verify-fix", {
+ description:
+ "Verify existing security fixes without changing the repository.",
+ destructive: false,
+ mcp: false,
+ args: z.object({
+ "findings...": z
+ .string()
+ .min(1, "A finding must not be empty.")
+ .optional()
+ .describe("Finding text, a file, or a saved finding identifier."),
+ }),
+ options: z.object({
+ effort: effortOption(),
+ scan: optionValue("--scan")
+ .optional()
+ .describe("Verify open findings from a saved scan."),
+ severity: z
+ .enum(REPORTABLE_SEVERITIES)
+ .optional()
+ .describe("Verify saved findings at or above LEVEL."),
+ linearIssue: z
+ .array(optionValue("--linear-issue"))
+ .default([])
+ .describe("Linear issue identifier or URL; repeat for more issues."),
+ linearProject: optionValue("--linear-project")
+ .optional()
+ .describe("Verify issues in this Linear project."),
+ linearFilter: optionValue("--linear-filter")
+ .optional()
+ .describe("JSON Linear issue filter for --linear-project."),
+ linearApiKey: linearApiKeyOption(),
+ codex: z
+ .array(optionValue("--codex"))
+ .default([])
+ .describe(
+ 'Repeat TOML model="gpt-5.6-terra" or model_reasoning_effort="high" only.',
+ ),
+ }),
+ output: z.record(z.string(), z.unknown()).optional(),
+ async run({ format, options }) {
+ try {
+ const linear =
+ options.linearIssue.length > 0 || !!options.linearProject;
+ if (options.linearIssue.length > 0 && options.linearProject) {
+ throw new CodexSecurityError(
+ "Use either --linear-issue or --linear-project, not both.",
+ );
+ }
+ if (options.linearFilter && !options.linearProject) {
+ throw new CodexSecurityError(
+ "--linear-filter requires --linear-project.",
+ );
+ }
+ if (options.linearApiKey !== undefined && !linear) {
+ throw new CodexSecurityError(
+ "--linear-api-key requires --linear-issue or --linear-project.",
+ );
+ }
+ const savedFindings =
+ options.scan !== undefined ||
+ (positionals.length > 0 && positionals.every(isFindingIdentifier));
+ if (savedFindings && linear) {
+ throw new CodexSecurityError(
+ "Saved findings cannot be combined with Linear issues or projects.",
+ );
+ }
+ if (options.severity !== undefined && !savedFindings) {
+ throw new CodexSecurityError(
+ "--severity requires a saved finding identifier or --scan.",
+ );
+ }
+ if (positionals.length === 0 && !linear && !savedFindings) {
+ throw new CodexSecurityError(
+ "Verify-fix requires a finding, --scan, --linear-issue, or --linear-project.",
+ );
+ }
+
+ const selected = savedFindings
+ ? await selectSavedFindings(
+ positionals,
+ options.scan,
+ options.severity,
+ dependencies,
+ )
+ : undefined;
+ const imports = linear
+ ? await importLinearIssues({
+ issues: options.linearIssue,
+ project: options.linearProject,
+ filter: options.linearFilter,
+ apiKey: options.linearApiKey,
+ environment: dependencies.environment,
+ linearClient: dependencies.linearClient,
+ })
+ : [];
+ const identifiers =
+ selected === undefined
+ ? [
+ ...positionals.map(
+ (_finding, index) => `finding-${index + 1}`,
+ ),
+ ...imports.map(({ id }) => id),
+ ]
+ : selected.findings.map(({ occurrenceId }) => occurrenceId);
+ if (new Set(identifiers).size !== identifiers.length) {
+ throw new CodexSecurityError(
+ "Verification inputs must have unique finding or issue identifiers.",
+ );
+ }
+ const repository =
+ selected?.repository ?? dependencies.currentDirectory();
+ let results: FindingVerification[] = [];
+ if (identifiers.length > 0) {
+ const environment =
+ imports.length === 0
+ ? undefined
+ : Object.fromEntries(
+ Object.entries(dependencies.environment).filter(
+ ([name]) =>
+ !/^(?:CODEX_SECURITY_)?LINEAR_(?:API_KEY|ACCESS_TOKEN)$/iu.test(
+ name,
+ ),
+ ),
+ );
+ let response = "";
+ const verificationOutput: Writable = {
+ write(value: string | Uint8Array): boolean {
+ response += value.toString();
+ return true;
+ },
+ };
+ const progress = new VerificationProgressPresenter(
+ errorOutput,
+ dependencies,
+ repository,
+ identifiers.length,
+ );
+ progress.start();
+ try {
+ exitCode = await runSkill(
+ "verify-fix",
+ selected === undefined ? [...positionals, ...imports] : [],
+ options.codex,
+ options.effort,
+ verificationOutput,
+ errorOutput,
+ dependencies,
+ {
+ directory: repository,
+ ...(selected === undefined
+ ? {}
+ : { findings: selected.findings }),
+ verificationIds: identifiers,
+ environment,
+ onEvent: progress.observe.bind(progress),
+ },
+ );
+ if (exitCode !== 0) {
+ if (exitCode !== 130 && exitCode !== 143) exitCode = 2;
+ return undefined;
+ }
+
+ let reported: unknown;
+ try {
+ reported = JSON.parse(response);
+ } catch {
+ throw new CodexSecurityError(
+ "Verification results were not valid JSON.",
+ );
+ }
+ const parsed = z
+ .object({ results: z.array(findingVerificationSchema) })
+ .safeParse(reported);
+ if (
+ !parsed.success ||
+ parsed.data.results.length !== identifiers.length ||
+ parsed.data.results.some(
+ ({ id }, index) => id !== identifiers[index],
+ )
+ ) {
+ throw new CodexSecurityError(
+ "Codex did not return an evidence-backed verification result for every finding.",
+ );
+ }
+ results = parsed.data.results;
+ progress.complete(results);
+ exitCode = results.some(({ status }) => status === "inconclusive")
+ ? 2
+ : results.some(({ status }) => status === "still_vulnerable")
+ ? 1
+ : 0;
+ } finally {
+ progress.stop();
+ }
+ }
+
+ if (format === "json" || format === "jsonl") {
+ return {
+ repository,
+ ...(selected === undefined ? {} : { scanId: selected.scanId }),
+ results,
+ };
+ }
+ for (const result of results) {
+ output.write(
+ `${result.status.toUpperCase()} ${safePatchText(result.id)}: ${safePatchText(result.evidence)}\n`,
+ );
+ }
+ return undefined;
+ } catch (error) {
+ exitCode = 2;
+ errorOutput.write(`codex-security: ${safeErrorMessage(error)}\n`);
+ return undefined;
+ }
+ },
+ })
.command("patch", {
description: "Patch one or more security issues.",
destructive: true,
@@ -3186,6 +3575,7 @@ function validateCliArguments(
"export",
"publish",
"validate",
+ "verify-fix",
"patch",
"login",
"logout",
@@ -3314,6 +3704,7 @@ function validateCliArguments(
}
if (
command !== "validate" &&
+ command !== "verify-fix" &&
command !== "patch" &&
positionals.length >
(command === "logout" || command === "info"
@@ -3818,7 +4209,7 @@ async function runFindingPatches(
}
async function runSkill(
- skill: "validation" | "fix-finding",
+ skill: "validation" | "fix-finding" | "verify-fix",
inputs: readonly (string | ImportedIssue)[],
codexOverrides: readonly string[],
effort: ScanReasoningEffort | undefined,
@@ -3918,13 +4309,31 @@ async function runSkill(
contents.push(contentsOrLiteral);
}
const plugin = await bundledPluginRoot();
- const inputLabel = skill === "validation" ? "Findings" : "Issues";
+ const verify = skill === "verify-fix";
+ const inputLabel = skill === "validation" || verify ? "Findings" : "Issues";
const prompt = [
- `Use the bundled $codex-security:${skill} skill at ${JSON.stringify(join(plugin, "skills", skill, "SKILL.md"))}.`,
- ...(options.findings === undefined
- ? []
+ ...(verify
+ ? [
+ "Use the bundled $codex-security:verify-fix skill. Its complete instructions and shared assessment reference are provided below; do not reread either file.",
+ await readFile(
+ join(plugin, "skills", "verify-fix", "SKILL.md"),
+ "utf8",
+ ),
+ "Shared static finding assessment reference:",
+ await readFile(
+ join(plugin, "references", "static-finding-assessment.md"),
+ "utf8",
+ ),
+ `Expected result identifiers (JSON array): ${JSON.stringify(options.verificationIds)}`,
+ "Return exactly one evidence-backed result per expected identifier in the same order, following the skill's JSON result contract.",
+ ]
: [
- 'Return exactly one JSON object with a "patches" array. Include one object for every supplied finding: {"occurrenceId":"...","status":"verified|no_change|blocked|failed","files":["relative/path"],"verification":"proof that the original issue is fixed and legitimate behavior still works","reason":"required for blocked or failed outcomes"}. Use "verified" only after the original issue no longer reproduces and relevant checks pass. Preserve unrelated local changes.',
+ `Use the bundled $codex-security:${skill} skill at ${JSON.stringify(join(plugin, "skills", skill, "SKILL.md"))}.`,
+ ...(options.findings === undefined
+ ? []
+ : [
+ 'Return exactly one JSON object with a "patches" array. Include one object for every supplied finding: {"occurrenceId":"...","status":"verified|no_change|blocked|failed","files":["relative/path"],"verification":"proof that the original issue is fixed and legitimate behavior still works","reason":"required for blocked or failed outcomes"}. Use "verified" only after the original issue no longer reproduces and relevant checks pass. Preserve unrelated local changes.',
+ ]),
]),
...(options.findingInstructions === undefined
? []
@@ -3936,12 +4345,13 @@ async function runSkill(
JSON.stringify(contents),
].join("\n");
const patch = skill === "fix-finding";
+ const appServer = patch || verify;
return dependencies.runCodex(
[
- ...(patch ? ["app-server"] : ["exec", "--ignore-user-config"]),
+ ...(appServer ? ["app-server"] : ["exec", "--ignore-user-config"]),
"--disable",
"plugins",
- ...(patch ? [] : ["--ephemeral", "--color", "never", "--json"]),
+ ...(appServer ? [] : ["--ephemeral", "--color", "never", "--json"]),
"--config",
`model=${JSON.stringify(model)}`,
"--config",
@@ -3956,10 +4366,11 @@ async function runSkill(
],
),
"--config",
- 'approval_policy="never"',
+ verify ? 'approval_policy="on-request"' : 'approval_policy="never"',
+ ...(verify ? ["--config", 'approvals_reviewer="auto_review"'] : []),
"--config",
'responses_api_metadata.codex_security_surface="cli"',
- ...(patch
+ ...(appServer
? []
: [
"--sandbox",
@@ -3971,10 +4382,21 @@ async function runSkill(
]),
],
{
- command: patch ? "patch" : "validate",
+ command: verify ? "verify-fix" : patch ? "patch" : "validate",
stdout,
stderr,
- ...(patch ? { appServer: { directory, prompt } } : {}),
+ ...(appServer
+ ? {
+ appServer: {
+ directory,
+ prompt,
+ ...(verify ? { sandbox: "read-only" as const } : {}),
+ ...(options.onEvent === undefined
+ ? {}
+ : { onEvent: options.onEvent }),
+ },
+ }
+ : {}),
},
options.environment,
);
@@ -3983,8 +4405,11 @@ async function runSkill(
export async function readSkillCommandOutput(
stream: AsyncIterable<Buffer | string>,
appServer?: {
+ readonly directory?: string;
readonly prompt: string;
readonly input: NodeJS.WritableStream;
+ readonly sandbox?: "read-only" | "workspace-write";
+ readonly onEvent?: (event: Readonly<Record<string, unknown>>) => void;
},
): Promise<{
message?: string;
@@ -4001,6 +4426,20 @@ export async function readSkillCommandOutput(
const send = (request: JsonObject): void => {
appServer?.input.write(`${JSON.stringify(request)}\n`);
};
+ const startThread = (config?: JsonObject): void => {
+ send({
+ id: 2,
+ method: "thread/start",
+ // An explicit cwd makes Codex persist trust for a new project.
+ // Inherit the child process cwd and preserve the user's decision.
+ params: {
+ approvalPolicy:
+ appServer?.sandbox === "read-only" ? "on-request" : "never",
+ sandbox: appServer?.sandbox ?? "workspace-write",
+ ...(config === undefined ? {} : { config }),
+ },
+ });
+ };
if (appServer !== undefined) {
send({
id: 1,
@@ -4024,6 +4463,19 @@ export async function readSkillCommandOutput(
}
const value = event as Record<string, unknown>;
if (appServer !== undefined) {
+ const params = value["params"];
+ if (
+ typeof params === "object" &&
+ params !== null &&
+ "threadId" in params &&
+ params.threadId === threadId &&
+ "turnId" in params &&
+ params.turnId === turnId
+ ) {
+ try {
+ appServer.onEvent?.(value);
+ } catch {}
+ }
if (value["id"] !== undefined) {
if (typeof value["method"] === "string") {
send({
@@ -4035,13 +4487,83 @@ export async function readSkillCommandOutput(
appServer.input.end();
} else if (value["id"] === 1) {
send({ method: "notifications/initialized" });
- send({
- id: 2,
- method: "thread/start",
- // An explicit cwd makes Codex persist trust for a new project.
- // Inherit the child process cwd and preserve the user's decision.
- params: { approvalPolicy: "never", sandbox: "workspace-write" },
- });
+ if (appServer.sandbox === "read-only") {
+ if (appServer.directory === undefined) {
+ error =
+ "Codex did not receive the repository directory required for read-only verification.";
+ appServer.input.end();
+ continue;
+ }
+ send({
+ id: 4,
+ method: "config/read",
+ params: {
+ cwd: appServer.directory,
+ includeLayers: true,
+ },
+ });
+ } else {
+ startThread();
+ }
+ } else if (value["id"] === 4 && appServer.sandbox === "read-only") {
+ const result = value["result"];
+ const layers =
+ typeof result === "object" && result !== null && "layers" in result
+ ? result.layers
+ : undefined;
+ if (!Array.isArray(layers)) {
+ error =
+ "Codex did not provide the configuration layers required for read-only verification.";
+ appServer.input.end();
+ continue;
+ }
+ const repositoryServers = new Set<string>();
+ const configuredServers = new Set<string>();
+ for (const layer of layers) {
+ if (typeof layer !== "object" || layer === null) continue;
+ const name = layer["name"];
+ const configuration = layer["config"];
+ if (
+ typeof name !== "object" ||
+ name === null ||
+ typeof configuration !== "object" ||
+ configuration === null
+ ) {
+ continue;
+ }
+ const servers = configuration["mcp_servers"];
+ if (typeof servers !== "object" || servers === null) continue;
+ if (name["type"] === "project") {
+ if (layer["disabledReason"] !== undefined) continue;
+ for (const server of Object.keys(servers)) {
+ repositoryServers.add(server);
+ }
+ } else {
+ for (const server of Object.keys(servers)) {
+ configuredServers.add(server);
+ }
+ }
+ }
+ const conflict = [...repositoryServers].find((server) =>
+ configuredServers.has(server),
+ );
+ if (conflict !== undefined) {
+ error = `Repository-local MCP server ${JSON.stringify(conflict)} overrides a configured integration; remove the repository override before verifying fixes.`;
+ appServer.input.end();
+ continue;
+ }
+ startThread(
+ repositoryServers.size === 0
+ ? undefined
+ : {
+ mcp_servers: Object.fromEntries(
+ [...repositoryServers].map((server) => [
+ server,
+ { enabled: false },
+ ]),
+ ),
+ },
+ );
} else if (value["id"] === 2) {
threadId = (value["result"] as { thread: { id: string } }).thread.id;
send({
@@ -4132,7 +4654,7 @@ export async function readSkillCommandOutput(
}
export function skillCommandFailure(
- command: "validate" | "patch",
+ command: "validate" | "patch" | "verify-fix",
status: number,
detail: string,
): string {
diff --git a/sdk/typescript/src/scan-dashboard.ts b/sdk/typescript/src/scan-dashboard.ts
index 11e15c25b..8173f0bd3 100644
--- a/sdk/typescript/src/scan-dashboard.ts
+++ b/sdk/typescript/src/scan-dashboard.ts
@@ -44,7 +44,7 @@ interface DashboardInput {
interface ScanDashboardOptions {
repository: string;
- presentation?: "scan" | "publication";
+ presentation?: "scan" | "publication" | "verification";
mode?: ScanMode;
model?: ScanModelConfiguration;
maxCostUsd?: number;
@@ -132,7 +132,11 @@ export class ScanDashboard {
this.#options.onInterrupt?.();
lines = 0;
} else if (/^[dam1-9]$/u.test(key)) {
- if (this.#options.presentation === "publication") continue;
+ if (
+ this.#options.presentation !== undefined &&
+ this.#options.presentation !== "scan"
+ )
+ continue;
if (key !== "d" && this.#view !== "details") continue;
if (lines !== 0) this.scroll(lines);
if (key === "d") {
@@ -357,6 +361,8 @@ export class ScanDashboard {
#render(): void {
const publication = this.#options.presentation === "publication";
+ const verification = this.#options.presentation === "verification";
+ const findingProgress = publication || verification;
const width = this.#width();
const activityRows = this.#activityRows();
const divider = ` ${"─".repeat(Math.max(0, width - 4))}`;
@@ -390,7 +396,7 @@ export class ScanDashboard {
const activity = history.slice(first, first + activityRows);
if (activity.length === 0) {
activity.push({
- text: ` [${formatLocalTime(this.#options.clock.now())}] · Waiting for ${this.#view === "details" ? "session events" : publication ? "publication activity" : "scan activity"}…`,
+ text: ` [${formatLocalTime(this.#options.clock.now())}] · Waiting for ${this.#view === "details" ? "session events" : publication ? "publication activity" : verification ? "verification activity" : "scan activity"}…`,
kind: "path",
});
}
@@ -401,7 +407,7 @@ export class ScanDashboard {
this.#scrollOffset === 0
? "Ctrl+C to exit"
: `${formatCount(this.#scrollOffset)} ${this.#scrollOffset === 1 ? "line" : "lines"} above live · Ctrl+C to exit`;
- if (!publication && this.#options.input?.isTTY === true) {
+ if (!findingProgress && this.#options.input?.isTTY === true) {
scrollStatus =
this.#view === "details"
? `d activity · a/m/1-9 source · ${scrollStatus}`
@@ -410,11 +416,11 @@ export class ScanDashboard {
const model = this.#options.model;
const lines = [
- ` CODEX SECURITY · ${publication ? "PUBLISH · " : ""}${basename(this.#options.repository)}${model === undefined ? "" : ` · ${model.model} (${model.reasoningEffort})`}${this.#view === "details" ? ` · DETAILS${this.#source === "all" ? "" : ` · ${typeof this.#source === "number" ? `worker ${this.#source}` : this.#source}`}` : ""}`,
+ ` CODEX SECURITY · ${publication ? "PUBLISH · " : verification ? "VERIFY-FIX · " : ""}${basename(this.#options.repository)}${model === undefined ? "" : ` · ${model.model} (${model.reasoningEffort})`}${this.#view === "details" ? ` · DETAILS${this.#source === "all" ? "" : ` · ${typeof this.#source === "number" ? `worker ${this.#source}` : this.#source}`}` : ""}`,
divider,
...activity,
divider,
- ...(publication
+ ...(findingProgress
? [
` STAGE ${this.#stage}`,
` FINDINGS ${this.#publicationProgress === null ? "waiting for findings" : `${formatCount(this.#publicationProgress.completed)} / ${formatCount(this.#publicationProgress.total)} processed`}`,
diff --git a/sdk/typescript/tests-ts/cli-patch-trust.test.ts b/sdk/typescript/tests-ts/cli-patch-trust.test.ts
index c12a060a7..241131011 100644
--- a/sdk/typescript/tests-ts/cli-patch-trust.test.ts
+++ b/sdk/typescript/tests-ts/cli-patch-trust.test.ts
@@ -12,15 +12,22 @@ import { readSkillCommandOutput } from "../src/cli.js";
import { writeCodexConfig } from "../src/config.js";
import { resolveCodexCommand } from "../src/runtime.js";
-test.each([undefined, "untrusted", "trusted"] as const)(
- "preserves project trust when starting a patch (%s)",
- async (trust) => {
+test.each([
+ [undefined, "patch"],
+ ["untrusted", "patch"],
+ ["trusted", "patch"],
+ ["trusted", "verify-fix"],
+ ["trusted", "conflicting-user-server"],
+] as const)(
+ "preserves project trust and protects verification (%s, %s)",
+ async (trust, mode) => {
const root = await realpath(
await mkdtemp(join(tmpdir(), "codex-security-patch-trust-")),
);
const repository = join(root, "repository");
const codexHome = join(root, "codex-home");
const marker = join(root, "mcp-started");
+ const userMarker = join(root, "user-mcp-started");
const projects =
trust === undefined
? undefined
@@ -51,6 +58,20 @@ test.each([undefined, "untrusted", "trusted"] as const)(
requires_openai_auth: false,
},
},
+ ...(mode === "patch"
+ ? {}
+ : {
+ mcp_servers: {
+ [mode === "verify-fix" ? "user-configured" : "synthetic"]: {
+ command: process.execPath,
+ args: [
+ "-e",
+ `require("node:fs").writeFileSync(${JSON.stringify(userMarker)}, "started")`,
+ ],
+ startup_timeout_sec: 1,
+ },
+ },
+ }),
...(projects === undefined ? {} : { projects }),
});
const child = spawn(
@@ -74,13 +95,20 @@ test.each([undefined, "untrusted", "trusted"] as const)(
const closed = once(child, "close");
let servers: string[] | undefined;
const input = new Writable({
+ final(callback) {
+ if (child.stdin.writableEnded) {
+ callback();
+ } else {
+ child.stdin.end(callback);
+ }
+ },
write(chunk, _encoding, callback) {
const request = JSON.parse(chunk.toString());
// Inspect the native task without making a model request.
if (request.method === "turn/start") {
child.stdin.write(
`${JSON.stringify({
- id: 4,
+ id: 5,
method: "mcpServerStatus/list",
params: { threadId: request.params.threadId },
})}\n`,
@@ -94,7 +122,7 @@ test.each([undefined, "untrusted", "trusted"] as const)(
async function* events(): AsyncGenerator<string> {
for await (const line of createInterface({ input: child.stdout })) {
const event = JSON.parse(line);
- if (event.id === 4) {
+ if (event.id === 5) {
servers = event.result?.data.map(
(server: { name: string }) => server.name,
);
@@ -104,18 +132,38 @@ test.each([undefined, "untrusted", "trusted"] as const)(
}
}
try {
- expect(
- await readSkillCommandOutput(events(), {
- prompt: "Synthetic finding",
- input,
- }),
- ).toMatchObject({ completed: false });
+ const output = await readSkillCommandOutput(events(), {
+ directory: repository,
+ prompt: "Synthetic finding",
+ input,
+ ...(mode === "patch" ? {} : { sandbox: "read-only" }),
+ });
+ expect(output).toMatchObject({
+ completed: false,
+ ...(mode === "conflicting-user-server"
+ ? {
+ error:
+ 'Repository-local MCP server "synthetic" overrides a configured integration; remove the repository override before verifying fixes.',
+ }
+ : {}),
+ });
expect(await closed).toEqual([0, null]);
expect(
parseToml(await readFile(configPath, "utf8"))["projects"],
).toEqual(projects);
- expect(servers).toEqual(trust === "trusted" ? ["synthetic"] : []);
- expect(existsSync(marker)).toBe(trust === "trusted");
+ expect(servers?.sort()).toEqual(
+ mode === "conflicting-user-server"
+ ? undefined
+ : mode === "verify-fix"
+ ? ["synthetic", "user-configured"]
+ : trust === "trusted"
+ ? ["synthetic"]
+ : [],
+ );
+ expect(existsSync(marker)).toBe(
+ trust === "trusted" && mode === "patch",
+ );
+ expect(existsSync(userMarker)).toBe(mode === "verify-fix");
} finally {
input.end();
child.stdin.end();
diff --git a/sdk/typescript/tests-ts/cli-skills.test.ts b/sdk/typescript/tests-ts/cli-skills.test.ts
index df4cba4c6..77dce5c80 100644
--- a/sdk/typescript/tests-ts/cli-skills.test.ts
+++ b/sdk/typescript/tests-ts/cli-skills.test.ts
@@ -1161,6 +1161,88 @@ lines.on("line", (line) => {
expect(stderr.text()).toBe("");
});
+ test("starts verification app-server threads with a read-only sandbox", async () => {
+ const source = `
+const assert = require("node:assert/strict");
+const lines = require("node:readline").createInterface({ input: process.stdin });
+const send = (message) => process.stdout.write(JSON.stringify(message) + "\\n");
+lines.on("line", (line) => {
+ const request = JSON.parse(line);
+ if (request.method === "initialize") send({ id: 1, result: {} });
+ if (request.method === "config/read") {
+ assert.deepEqual(request.params, {
+ cwd: ${JSON.stringify(process.cwd())},
+ includeLayers: true,
+ });
+ send({ id: 4, result: { layers: [
+ { name: { type: "project" }, config: { mcp_servers: { repository: { command: "untrusted" } } } },
+ { name: { type: "user" }, config: { mcp_servers: { trusted: { command: "trusted" } } } },
+ ] } });
+ }
+ if (request.method === "thread/start") {
+ assert.deepEqual(request.params, {
+ approvalPolicy: "on-request",
+ sandbox: "read-only",
+ config: { mcp_servers: { repository: { enabled: false } } },
+ });
+ send({ id: 2, result: { thread: { id: "verification" } } });
+ }
+ if (request.method === "turn/start") {
+ send({ id: 3, result: { turn: { id: "verification-turn" } } });
+ send({ method: "item/started", params: {
+ threadId: "another-thread", turnId: "verification-turn",
+ item: { id: "hidden-command", type: "commandExecution", command: "private command" },
+ } });
+ send({ method: "item/started", params: {
+ threadId: "verification", turnId: "verification-turn",
+ item: { id: "verification-command", type: "commandExecution", command: "rg authorization src/guard.ts" },
+ } });
+ send({ method: "item/reasoning/summaryTextDelta", params: {
+ threadId: "verification", turnId: "verification-turn",
+ itemId: "verification-reasoning", delta: "Tracing the original control.",
+ } });
+ send({ method: "item/completed", params: {
+ threadId: "verification", turnId: "verification-turn",
+ item: { type: "agentMessage", text: "Verified without modifying source" },
+ } });
+ send({ method: "turn/completed", params: {
+ threadId: "verification", turn: { id: "verification-turn", status: "completed" },
+ } });
+ }
+});
+`;
+ const stdout = capture();
+ const stderr = capture();
+ const activity: Readonly<Record<string, unknown>>[] = [];
+
+ await expect(
+ runCodexSkillCommand(
+ ["-e", source],
+ {
+ command: "verify-fix",
+ stdout: stdout.stream,
+ stderr: stderr.stream,
+ appServer: {
+ directory: process.cwd(),
+ prompt:
+ "Verify the synthetic finding without editing the repository",
+ sandbox: "read-only",
+ onEvent: (event) => activity.push(event),
+ },
+ },
+ { command: process.execPath },
+ ),
+ ).resolves.toBe(0);
+ expect(stdout.text()).toBe("Verified without modifying source\n");
+ expect(stderr.text()).toBe("");
+ expect(activity.map((event) => event["method"])).toEqual([
+ "item/started",
+ "item/reasoning/summaryTextDelta",
+ "item/completed",
+ ]);
+ expect(JSON.stringify(activity)).not.toContain("private command");
+ });
+
test.each([
["EOF after a final answer", "final_answer", false],
["EOF after commentary", "commentary", false],
diff --git a/sdk/typescript/tests-ts/cli-verify-fix.test.ts b/sdk/typescript/tests-ts/cli-verify-fix.test.ts
new file mode 100644
index 000000000..d25883334
--- /dev/null
+++ b/sdk/typescript/tests-ts/cli-verify-fix.test.ts
@@ -0,0 +1,540 @@
+import { describe, expect, test } from "bun:test";
+import { readFile } from "node:fs/promises";
+import { stripVTControlCharacters } from "node:util";
+import { main } from "../src/cli.js";
+import type { Finding, JsonObject } from "../src/index.js";
+import type { LinearClientFactory } from "../src/linear.js";
+import { capture, dependencies, fakeResult } from "./cli-fixtures.js";
+
+function linearIssue(identifier: string) {
+ return {
+ identifier,
+ title: `Verify ${identifier}`,
+ description: `Synthetic security evidence for ${identifier}`,
+ url: `https://linear.app/example/issue/${identifier}`,
+ };
+}
+
+describe("read-only finding verification", () => {
+ test("verifies imported Linear issues in a read-only sandbox without exposing credentials", async () => {
+ const stdout = capture();
+ const stderr = capture();
+ let prompt = "";
+ let environment: NodeJS.ProcessEnv | undefined;
+
+ expect(
+ await main(
+ ["verify-fix", "--linear-issue", "SEC-123", "--json"],
+ stdout.stream,
+ stderr.stream,
+ dependencies({
+ environment: {
+ CODEX_SECURITY_LINEAR_API_KEY: "lin_api_SYNTHETIC_SECRET",
+ LINEAR_ACCESS_TOKEN: "SYNTHETIC_OAUTH_SECRET",
+ OPENAI_API_KEY: "sk-proj-SYNTHETIC_MODEL_KEY",
+ },
+ linearClient: () =>
+ ({
+ issue: async (id: string) => linearIssue(id),
+ }) as ReturnType<LinearClientFactory>,
+ onCodex: (args, output, processEnvironment) => {
+ expect(args[0]).toBe("app-server");
+ expect(args).toContain('approval_policy="on-request"');
+ expect(args).toContain('approvals_reviewer="auto_review"');
+ expect(args).not.toContain('approval_policy="never"');
+ expect(output?.command).toBe("verify-fix");
+ expect(output?.appServer?.sandbox).toBe("read-only");
+ prompt = output!.appServer!.prompt;
+ environment = processEnvironment;
+ output?.stdout.write(
+ JSON.stringify({
+ results: [
+ {
+ id: "SEC-123",
+ status: "fixed",
+ evidence:
+ "The authorization check rejects the original request and the legitimate request succeeds.",
+ },
+ ],
+ }),
+ );
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(0);
+
+ expect(JSON.parse(stdout.text())).toEqual({
+ repository: "/current/repository",
+ results: [
+ {
+ id: "SEC-123",
+ status: "fixed",
+ evidence:
+ "The authorization check rejects the original request and the legitimate request succeeds.",
+ },
+ ],
+ });
+ expect(stderr.text()).not.toContain("lin_api_SYNTHETIC_SECRET");
+ expect(prompt).toContain("standalone verification-only mode");
+ expect(prompt).toContain("$codex-security:verify-fix");
+ expect(prompt).toContain(
+ await readFile(
+ new URL(
+ "../_bundled_plugin/skills/verify-fix/SKILL.md",
+ import.meta.url,
+ ),
+ "utf8",
+ ),
+ );
+ expect(prompt).toContain(
+ await readFile(
+ new URL(
+ "../_bundled_plugin/references/static-finding-assessment.md",
+ import.meta.url,
+ ),
+ "utf8",
+ ),
+ );
+ expect(prompt).not.toContain("$codex-security:fix-finding");
+ expect(prompt).not.toContain("$codex-security:validation");
+ expect(prompt).toContain("Synthetic security evidence for SEC-123");
+ expect(prompt).not.toContain("lin_api_SYNTHETIC_SECRET");
+ expect(environment?.["OPENAI_API_KEY"]).toBe("sk-proj-SYNTHETIC_MODEL_KEY");
+ expect(environment).not.toHaveProperty("CODEX_SECURITY_LINEAR_API_KEY");
+ expect(environment).not.toHaveProperty("LINEAR_ACCESS_TOKEN");
+ });
+
+ test("shows live agent progress in the verification dashboard while keeping JSON clean", async () => {
+ const stdout = capture();
+ const stderr = capture(true);
+
+ expect(
+ await main(
+ ["verify-fix", "A previously reported authorization bypass", "--json"],
+ stdout.stream,
+ stderr.stream,
+ dependencies({
+ onCodex: (_args, output) => {
+ const emit = output?.appServer?.onEvent;
+ emit?.({
+ method: "item/started",
+ params: {
+ item: {
+ id: "command-1",
+ type: "commandExecution",
+ command: "rg authorization src/guard.ts",
+ },
+ },
+ });
+ emit?.({
+ method: "item/reasoning/summaryTextDelta",
+ params: {
+ itemId: "reasoning-1",
+ delta: "Checking the original authorization guard.",
+ },
+ });
+ emit?.({
+ method: "item/started",
+ params: {
+ item: {
+ id: "tool-1",
+ type: "mcpToolCall",
+ tool: "read_file",
+ arguments: { path: "src/guard.ts" },
+ },
+ },
+ });
+ emit?.({
+ method: "item/completed",
+ params: {
+ item: {
+ id: "commentary-1",
+ type: "agentMessage",
+ phase: "commentary",
+ text: "The bypass now reaches the shared guard.",
+ },
+ },
+ });
+ output?.stdout.write(
+ JSON.stringify({
+ results: [
+ {
+ id: "finding-1",
+ status: "fixed",
+ evidence: "The original authorization bypass is rejected.",
+ },
+ ],
+ }),
+ );
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(0);
+
+ const activity = stripVTControlCharacters(stderr.text());
+ expect(activity).toContain("VERIFY-FIX");
+ expect(activity).toContain("rg authorization src/guard.ts");
+ expect(activity).toContain("Checking the original authorization guard.");
+ expect(activity).toContain("read_file");
+ expect(activity).toContain("The bypass now reaches the shared guard.");
+ expect(activity).not.toContain("FILES");
+ expect(stderr.text()).toContain("\u001B[?1049h\u001B[?25l");
+ expect(stderr.text()).toContain("\u001B[?25h\u001B[?1049l");
+ expect(JSON.parse(stdout.text())).toMatchObject({
+ results: [{ id: "finding-1", status: "fixed" }],
+ });
+ expect(stdout.text()).not.toContain("\u001B");
+ });
+
+ test("uses plain verification progress in CI and dumb terminals", async () => {
+ for (const environment of [{ CI: "1" }, { TERM: "dumb" }]) {
+ const stdout = capture();
+ const stderr = capture(true);
+
+ expect(
+ await main(
+ [
+ "verify-fix",
+ "A previously reported authorization bypass",
+ "--json",
+ ],
+ stdout.stream,
+ stderr.stream,
+ dependencies({
+ environment,
+ onCodex: (_args, output) => {
+ output?.appServer?.onEvent?.({
+ method: "item/completed",
+ params: {
+ item: {
+ id: "reasoning-1",
+ type: "reasoning",
+ summary: ["Checking the original authorization guard."],
+ },
+ },
+ });
+ output?.stdout.write(
+ JSON.stringify({
+ results: [
+ {
+ id: "finding-1",
+ status: "fixed",
+ evidence:
+ "The original authorization bypass is rejected.",
+ },
+ ],
+ }),
+ );
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(0);
+
+ expect(stderr.text()).toContain("Verifying");
+ expect(stderr.text()).toContain(
+ "Codex: Checking the original authorization guard.",
+ );
+ expect(stderr.text()).not.toContain("\u001B");
+ expect(JSON.parse(stdout.text())).toMatchObject({
+ results: [{ id: "finding-1", status: "fixed" }],
+ });
+ }
+ });
+
+ test("verifies repeated Linear issues together in one agent invocation", async () => {
+ const stdout = capture();
+ let agentCalls = 0;
+ let prompt = "";
+
+ expect(
+ await main(
+ [
+ "verify-fix",
+ "--linear-issue",
+ "SEC-123",
+ "--linear-issue",
+ "SEC-456",
+ "--json",
+ ],
+ stdout.stream,
+ capture().stream,
+ dependencies({
+ environment: { LINEAR_ACCESS_TOKEN: "SYNTHETIC_OAUTH_TOKEN" },
+ linearClient: () =>
+ ({
+ issue: async (id: string) => linearIssue(id),
+ }) as ReturnType<LinearClientFactory>,
+ onCodex: (_args, output) => {
+ agentCalls += 1;
+ prompt = output!.appServer!.prompt;
+ output?.stdout.write(
+ JSON.stringify({
+ results: [
+ {
+ id: "SEC-123",
+ status: "fixed",
+ evidence: "The original exploit is rejected.",
+ },
+ {
+ id: "SEC-456",
+ status: "still_vulnerable",
+ evidence:
+ "The original unauthenticated route remains reachable.",
+ },
+ ],
+ }),
+ );
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(1);
+
+ expect(agentCalls).toBe(1);
+ const suppliedFindings = JSON.parse(prompt.split("\n").at(-1)!) as string[];
+ expect(suppliedFindings).toHaveLength(2);
+ expect(suppliedFindings[0]).toContain("SEC-123");
+ expect(suppliedFindings[1]).toContain("SEC-456");
+ expect(JSON.parse(stdout.text())).toMatchObject({
+ results: [
+ { id: "SEC-123", status: "fixed" },
+ { id: "SEC-456", status: "still_vulnerable" },
+ ],
+ });
+ });
+
+ test("verifies saved findings in their original repository", async () => {
+ const result = fakeResult(["high", "medium"]);
+ result.findings.findings.forEach((finding, index) => {
+ Object.assign(finding, {
+ findingId: `csf_${index + 1}`,
+ occurrenceId: `occ_${index + 1}`,
+ title: `Finding ${index + 1}`,
+ });
+ });
+ const stdout = capture();
+
+ expect(
+ await main(
+ ["verify-fix", "--scan", "scan-1", "--severity", "high", "--json"],
+ stdout.stream,
+ capture().stream,
+ dependencies({
+ onWorkbench: (args) => {
+ expect(args).toEqual(["get-scan", "--scan-id", "scan-1"]);
+ return {
+ scan: {
+ scanId: "scan-1",
+ targetPath: "/saved/repository",
+ findings: result.findings.findings as unknown as JsonObject[],
+ },
+ };
+ },
+ onCodex: (_args, output) => {
+ expect(output?.appServer?.directory).toBe("/saved/repository");
+ expect(output?.appServer?.sandbox).toBe("read-only");
+ const findings = JSON.parse(
+ output!.appServer!.prompt.split("\n").at(-1)!,
+ ) as Finding[];
+ expect(findings.map(({ occurrenceId }) => occurrenceId)).toEqual([
+ "occ_1",
+ ]);
+ output?.stdout.write(
+ JSON.stringify({
+ results: [
+ {
+ id: "occ_1",
+ status: "fixed",
+ evidence:
+ "The original exploit fails and regression checks pass.",
+ },
+ ],
+ }),
+ );
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(0);
+
+ expect(JSON.parse(stdout.text())).toMatchObject({
+ repository: "/saved/repository",
+ scanId: "scan-1",
+ results: [{ id: "occ_1", status: "fixed" }],
+ });
+ });
+
+ test("reports inconclusive findings without treating them as fixed", async () => {
+ const stdout = capture();
+ expect(
+ await main(
+ ["verify-fix", "A previously reported authorization bypass"],
+ stdout.stream,
+ capture().stream,
+ dependencies({
+ onCodex: (_args, output) => {
+ output?.stdout.write(
+ JSON.stringify({
+ results: [
+ {
+ id: "finding-1",
+ status: "inconclusive",
+ evidence: "The original entrypoint cannot be identified.",
+ },
+ ],
+ }),
+ );
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(2);
+
+ expect(stdout.text()).toContain(
+ "INCONCLUSIVE finding-1: The original entrypoint cannot be identified.",
+ );
+ });
+
+ test.each([
+ [1, 2],
+ [7, 2],
+ [130, 130],
+ [143, 143],
+ ] as const)(
+ "maps Codex exit %i to verification exit %i",
+ async (codexStatus, expectedStatus) => {
+ const stdout = capture();
+
+ expect(
+ await main(
+ [
+ "verify-fix",
+ "A previously reported authorization bypass",
+ "--json",
+ ],
+ stdout.stream,
+ capture().stream,
+ dependencies({ onCodex: () => codexStatus }),
+ ),
+ ).toBe(expectedStatus);
+ expect(stdout.text()).toBe("");
+ },
+ );
+
+ test("sanitizes model-controlled evidence in human-readable output", async () => {
+ const stdout = capture();
+
+ expect(
+ await main(
+ ["verify-fix", "A previously reported authorization bypass"],
+ stdout.stream,
+ capture().stream,
+ dependencies({
+ onCodex: (_args, output) => {
+ output?.stdout.write(
+ JSON.stringify({
+ results: [
+ {
+ id: "finding-1",
+ status: "fixed",
+ evidence:
+ "\u001b[31mOriginal exploit rejected.\nForged result.",
+ },
+ ],
+ }),
+ );
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(0);
+
+ expect(stdout.text()).toBe(
+ "FIXED finding-1: Original exploit rejected. Forged result.\n",
+ );
+ });
+
+ test.each([
+ ["missing result", { results: [] }],
+ [
+ "wrong finding",
+ {
+ results: [
+ { id: "SEC-124", status: "fixed", evidence: "Different finding." },
+ ],
+ },
+ ],
+ [
+ "missing evidence",
+ { results: [{ id: "SEC-123", status: "fixed", evidence: " " }] },
+ ],
+ [
+ "unsupported outcome",
+ { results: [{ id: "SEC-123", status: "no_change", evidence: "Safe." }] },
+ ],
+ ] as const)(
+ "rejects %s instead of reporting an unverified fix",
+ async (_name, result) => {
+ const stdout = capture();
+ const stderr = capture();
+
+ expect(
+ await main(
+ ["verify-fix", "--linear-issue", "SEC-123", "--json"],
+ stdout.stream,
+ stderr.stream,
+ dependencies({
+ environment: { CODEX_SECURITY_LINEAR_API_KEY: "synthetic-key" },
+ linearClient: () =>
+ ({
+ issue: async (id: string) => linearIssue(id),
+ }) as ReturnType<LinearClientFactory>,
+ onCodex: (_args, output) => {
+ output?.stdout.write(JSON.stringify(result));
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(2);
+
+ expect(stdout.text()).toBe("");
+ expect(stderr.text()).toContain(
+ "an evidence-backed verification result for every finding",
+ );
+ },
+ );
+
+ test.each([
+ [["verify-fix"], "Verify-fix requires a finding"],
+ [
+ ["verify-fix", "--scan", "scan-1", "--linear-issue", "SEC-123"],
+ "Saved findings cannot be combined with Linear issues or projects",
+ ],
+ ] as const)(
+ "rejects invalid verification selection %j",
+ async (args, expected) => {
+ const stderr = capture();
+ let started = false;
+
+ expect(
+ await main(
+ args,
+ capture().stream,
+ stderr.stream,
+ dependencies({
+ environment: { CODEX_SECURITY_LINEAR_API_KEY: "synthetic-key" },
+ onCodex: () => {
+ started = true;
+ return 0;
+ },
+ }),
+ ),
+ ).toBe(2);
+ expect(stderr.text()).toContain(expected);
+ expect(started).toBe(false);
+ },
+ );
+});
diff --git a/sdk/typescript/tests-ts/cli.test.ts b/sdk/typescript/tests-ts/cli.test.ts
index 6c297b621..7a8517e1c 100644
--- a/sdk/typescript/tests-ts/cli.test.ts
+++ b/sdk/typescript/tests-ts/cli.test.ts
@@ -202,6 +202,9 @@ describe("CLI", () => {
expect(manifest.text()).toContain("codex-security bulk-scan [input]");
expect(manifest.text()).toContain("codex-security export [scanDir]");
expect(manifest.text()).toContain("codex-security validate <findings...>");
+ expect(manifest.text()).toContain(
+ "codex-security verify-fix [findings...]",
+ );
expect(manifest.text()).toContain("codex-security patch [issues...]");
expect(manifest.text()).toContain(
"codex-security findings false-positive <occurrenceId>",
diff --git a/sdk/typescript/tests-ts/patch-tui.test.ts b/sdk/typescript/tests-ts/patch-tui.test.ts
index e81210d99..bef2a644a 100644
--- a/sdk/typescript/tests-ts/patch-tui.test.ts
+++ b/sdk/typescript/tests-ts/patch-tui.test.ts
@@ -81,7 +81,7 @@ function findings(severities: readonly SeverityLevel[]): Finding[] {
}
async function settle(): Promise<void> {
- await new Promise<void>((resolve) => setTimeout(resolve, 30));
+ await new Promise<void>((resolve) => setTimeout(resolve, 60));
}
describe("interactive patch finding browser", () => {