OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security
TypeScript ai-securityapplication-securityclicode-scanningcodexcodex-securitycybersecuritydevsecopsnodejsnpmopenaisdksecuritytypescriptvulnerability-scanning
- description: "Unavailable Linear connector stops with setup guidance"
metadata:
case_id: ticket-connector-unavailable
source_type: scanner_ticket
vars:
case_id: ticket-connector-unavailable
target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear issue SEC-293. Connector observation: the Linear app is not connected and no other authenticated connection can read this Linear workspace."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
expected_ticket_failure: unavailable
eval_instructions: &failedReadInstructions |-
This is an automated eval. Treat the connector observation as the result
of the requested read. Respond exactly as the skill requires after this
intake failure. Do not inspect code or fabricate the issue contents.
options: &intakeOnly
disableDefaultAsserts: true
assert: &ticketIntake
- type: javascript
value: file://./assertions/ticket-intake.mts
- description: "Linear permission failure stops with access guidance"
metadata:
case_id: ticket-connector-permission
source_type: scanner_ticket
vars:
case_id: ticket-connector-permission
target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear issue SEC-293. Connector observation: access was denied because the authenticated user lacks permission."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
expected_ticket_failure: permission
eval_instructions: *failedReadInstructions
options: *intakeOnly
assert: *ticketIntake
- description: "Missing Linear issue stops with identifier guidance"
metadata:
case_id: ticket-not-found
source_type: scanner_ticket
vars:
case_id: ticket-not-found
target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear issue SEC-999999. Connector observation: the issue was not found or is inaccessible."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-999999."
expected_ticket_failure: not_found
eval_instructions: *failedReadInstructions
options: *intakeOnly
assert: *ticketIntake
- description: "Repeated transient Linear failure stops after one retry"
metadata:
case_id: ticket-transient-retry-exhausted
source_type: scanner_ticket
vars:
case_id: ticket-transient-retry-exhausted
target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear issue SEC-293. Connector observation: the first read returned a transient timeout; the identical retry also timed out."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
expected_ticket_failure: transient
eval_instructions: |-
This is an automated eval. Treat the connector observation as the result
of the requested read and its one retry. Respond exactly as the skill
requires after retry exhaustion. Do not inspect code or fabricate the
issue contents.
options: *intakeOnly
assert: *ticketIntake
- description: "Linear parent asks before importing direct sub-issues"
metadata:
case_id: linear-direct-subissue-confirmation
source_type: scanner_ticket
vars:
case_id: linear-direct-subissue-confirmation
target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear parent SEC-293 was fetched. Child listing returned SEC-294 'Auth bypass' and SEC-295 'Unsafe redirect'. Their full bodies have not been fetched."
invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
expected_linear_subissues: direct_confirmation
eval_instructions: |-
This is an automated eval. Treat the parent fetch and child listing as
completed connector observations. Follow the skill before importing any
child issue. Do not inspect code or fabricate child issue contents.
options: *intakeOnly
assert: *ticketIntake
- description: "Linear child-only selection asks before expanding to grandchildren"
metadata:
case_id: linear-nested-subissue-confirmation
source_type: scanner_ticket
vars:
case_id: linear-nested-subissue-confirmation
target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "The user approved direct child SEC-294. Its full body was fetched. The next child listing found SEC-296 'Nested token leak'; its full body has not been fetched."
invocation_preamble: "Continue $codex-security:triage-finding for the approved Linear issue tree."
expected_linear_subissues: next_depth
eval_instructions: |-
This is an automated eval. Treat the connector observations as complete.
Follow the skill before importing the next depth. Do not inspect code or
fabricate the nested issue contents.
options: *intakeOnly
assert: *ticketIntake
- description: "Linear all-descendants selection proceeds across depths"
metadata:
case_id: linear-all-descendants
source_type: scanner_ticket
vars:
case_id: linear-all-descendants
target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
invocation_preamble: "Use $codex-security:triage-finding to triage every finding under Linear parent SEC-293, including all descendants."
expected_ids: SEC-294,SEC-295,SEC-296
expected_source_types: scanner_ticket,scanner_ticket,scanner_ticket
expected_verdicts: confirmed,not_actionable,not_actionable
finding_input: |-
Connector observations: all child listings and pages are complete. SEC-293
is an organizing summary with no independent claim. Its children, in
creation order, are SEC-294 and SEC-295. SEC-294 has child SEC-296. There
are no other descendants. The full selected issue bodies were fetched:
SEC-294: User-controlled next query parameter reaches res.redirect in
src/server.js route /redirect without allowlist validation.
SEC-295: User-controlled next query parameter may reach res.redirect in
src/server.js route /safe-redirect.
SEC-296: SQL injection in product search: req.query.q is concatenated into
a SQL string in src/server.js.
eval_instructions: |-
This is an automated eval. Treat the connector observations as completed
reads, including the full issue bodies above. Follow the skill to decide
whether the requested collection is ready for static triage. Do not make
live connector calls or invent additional issues. If triage can proceed,
inspect only the fixture repository and return concise Markdown plus one
fenced triage-finding/v0 JSON block.
- description: "Ambiguous Linear parent asks whether it is a finding"
metadata:
case_id: linear-ambiguous-parent
source_type: scanner_ticket
vars:
case_id: linear-ambiguous-parent
target_repo: evals/triage-finding/fixtures/repo
source_type_under_test: scanner_ticket
finding_input: "Linear parent SEC-293 contains collection notes plus a possible standalone authorization claim. The user approved importing its child findings."
invocation_preamble: "Continue $codex-security:triage-finding for Linear parent SEC-293."
expected_linear_subissues: ambiguous_parent
eval_instructions: |-
This is an automated eval. The parent may or may not be an independent
finding. Follow the skill before normalizing the parent. Do not inspect
code or emit triage JSON.
options: *intakeOnly
assert: *ticketIntake