Skip to content

OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security

7.6 KB
Raw Blame
- description: "Unavailable Linear connector stops with setup guidance"
  metadata:
    case_id: ticket-connector-unavailable
    source_type: scanner_ticket
  vars:
    case_id: ticket-connector-unavailable
    target_repo: evals/triage-finding/fixtures/repo
    source_type_under_test: scanner_ticket
    finding_input: "Linear issue SEC-293. Connector observation: the Linear app is not connected and no other authenticated connection can read this Linear workspace."
    invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
    expected_ticket_failure: unavailable
    eval_instructions: &failedReadInstructions |-
      This is an automated eval. Treat the connector observation as the result
      of the requested read. Respond exactly as the skill requires after this
      intake failure. Do not inspect code or fabricate the issue contents.
  options: &intakeOnly
    disableDefaultAsserts: true
  assert: &ticketIntake
    - type: javascript
      value: file://./assertions/ticket-intake.mts

- description: "Linear permission failure stops with access guidance"
  metadata:
    case_id: ticket-connector-permission
    source_type: scanner_ticket
  vars:
    case_id: ticket-connector-permission
    target_repo: evals/triage-finding/fixtures/repo
    source_type_under_test: scanner_ticket
    finding_input: "Linear issue SEC-293. Connector observation: access was denied because the authenticated user lacks permission."
    invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
    expected_ticket_failure: permission
    eval_instructions: *failedReadInstructions
  options: *intakeOnly
  assert: *ticketIntake

- description: "Missing Linear issue stops with identifier guidance"
  metadata:
    case_id: ticket-not-found
    source_type: scanner_ticket
  vars:
    case_id: ticket-not-found
    target_repo: evals/triage-finding/fixtures/repo
    source_type_under_test: scanner_ticket
    finding_input: "Linear issue SEC-999999. Connector observation: the issue was not found or is inaccessible."
    invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-999999."
    expected_ticket_failure: not_found
    eval_instructions: *failedReadInstructions
  options: *intakeOnly
  assert: *ticketIntake

- description: "Repeated transient Linear failure stops after one retry"
  metadata:
    case_id: ticket-transient-retry-exhausted
    source_type: scanner_ticket
  vars:
    case_id: ticket-transient-retry-exhausted
    target_repo: evals/triage-finding/fixtures/repo
    source_type_under_test: scanner_ticket
    finding_input: "Linear issue SEC-293. Connector observation: the first read returned a transient timeout; the identical retry also timed out."
    invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
    expected_ticket_failure: transient
    eval_instructions: |-
      This is an automated eval. Treat the connector observation as the result
      of the requested read and its one retry. Respond exactly as the skill
      requires after retry exhaustion. Do not inspect code or fabricate the
      issue contents.
  options: *intakeOnly
  assert: *ticketIntake

- description: "Linear parent asks before importing direct sub-issues"
  metadata:
    case_id: linear-direct-subissue-confirmation
    source_type: scanner_ticket
  vars:
    case_id: linear-direct-subissue-confirmation
    target_repo: evals/triage-finding/fixtures/repo
    source_type_under_test: scanner_ticket
    finding_input: "Linear parent SEC-293 was fetched. Child listing returned SEC-294 'Auth bypass' and SEC-295 'Unsafe redirect'. Their full bodies have not been fetched."
    invocation_preamble: "Use $codex-security:triage-finding to import Linear issue SEC-293."
    expected_linear_subissues: direct_confirmation
    eval_instructions: |-
      This is an automated eval. Treat the parent fetch and child listing as
      completed connector observations. Follow the skill before importing any
      child issue. Do not inspect code or fabricate child issue contents.
  options: *intakeOnly
  assert: *ticketIntake

- description: "Linear child-only selection asks before expanding to grandchildren"
  metadata:
    case_id: linear-nested-subissue-confirmation
    source_type: scanner_ticket
  vars:
    case_id: linear-nested-subissue-confirmation
    target_repo: evals/triage-finding/fixtures/repo
    source_type_under_test: scanner_ticket
    finding_input: "The user approved direct child SEC-294. Its full body was fetched. The next child listing found SEC-296 'Nested token leak'; its full body has not been fetched."
    invocation_preamble: "Continue $codex-security:triage-finding for the approved Linear issue tree."
    expected_linear_subissues: next_depth
    eval_instructions: |-
      This is an automated eval. Treat the connector observations as complete.
      Follow the skill before importing the next depth. Do not inspect code or
      fabricate the nested issue contents.
  options: *intakeOnly
  assert: *ticketIntake

- description: "Linear all-descendants selection proceeds across depths"
  metadata:
    case_id: linear-all-descendants
    source_type: scanner_ticket
  vars:
    case_id: linear-all-descendants
    target_repo: evals/triage-finding/fixtures/repo
    source_type_under_test: scanner_ticket
    invocation_preamble: "Use $codex-security:triage-finding to triage every finding under Linear parent SEC-293, including all descendants."
    expected_ids: SEC-294,SEC-295,SEC-296
    expected_source_types: scanner_ticket,scanner_ticket,scanner_ticket
    expected_verdicts: confirmed,not_actionable,not_actionable
    finding_input: |-
      Connector observations: all child listings and pages are complete. SEC-293
      is an organizing summary with no independent claim. Its children, in
      creation order, are SEC-294 and SEC-295. SEC-294 has child SEC-296. There
      are no other descendants. The full selected issue bodies were fetched:

      SEC-294: User-controlled next query parameter reaches res.redirect in
      src/server.js route /redirect without allowlist validation.

      SEC-295: User-controlled next query parameter may reach res.redirect in
      src/server.js route /safe-redirect.

      SEC-296: SQL injection in product search: req.query.q is concatenated into
      a SQL string in src/server.js.
    eval_instructions: |-
      This is an automated eval. Treat the connector observations as completed
      reads, including the full issue bodies above. Follow the skill to decide
      whether the requested collection is ready for static triage. Do not make
      live connector calls or invent additional issues. If triage can proceed,
      inspect only the fixture repository and return concise Markdown plus one
      fenced triage-finding/v0 JSON block.

- description: "Ambiguous Linear parent asks whether it is a finding"
  metadata:
    case_id: linear-ambiguous-parent
    source_type: scanner_ticket
  vars:
    case_id: linear-ambiguous-parent
    target_repo: evals/triage-finding/fixtures/repo
    source_type_under_test: scanner_ticket
    finding_input: "Linear parent SEC-293 contains collection notes plus a possible standalone authorization claim. The user approved importing its child findings."
    invocation_preamble: "Continue $codex-security:triage-finding for Linear parent SEC-293."
    expected_linear_subissues: ambiguous_parent
    eval_instructions: |-
      This is an automated eval. The parent may or may not be an independent
      finding. Follow the skill before normalizing the parent. Do not inspect
      code or emit triage JSON.
  options: *intakeOnly
  assert: *ticketIntake