Skip to content

OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security

16.1 KB
Raw Blame

Testing the SDK and CLI

Install dependencies

Use the repository’s pinned pnpm version and CI’s Bun version. From the repository root, enter the SDK directory and install both packages:

cd sdk/typescript
pnpm install --frozen-lockfile
pnpm --dir ../../plugins/codex-security/mcp-app install --frozen-lockfile

The commands below run from sdk/typescript unless a different directory is named. For a test that builds or loads the bundled plugin, first download the universal native payload. The SDK’s test, test:ci, test:mcp and prepack scripts all build that bundle.

Run the CLI from your checkout

From the repository root, use the source entry point to iterate on commands and help:

bun sdk/typescript/src/cli.ts scan --help

For the Node launcher that npm installs as cs, build the SDK first:

pnpm --dir sdk/typescript run build
node sdk/typescript/bin/codex-security.mjs scan --help
node sdk/typescript/bin/codex-security.mjs scan . --dry-run --json

Saved history, exports, and commands that run the bundled plugin also need the native payload described above and pnpm --dir sdk/typescript run build:plugin. The build:ci script compiles the repository’s CI helpers.

Run a focused test

Run the file for the behavior you changed. This example exercises progress parsing:

bun test --timeout 30000 ./tests-ts/worker-progress.test.ts

For an MCP change, follow the MCP test guide. To see which suites this repository requires for a PR, see How this repository tests pull requests.

Native Windows sandbox tests

Two tests in tests-ts/config.test.ts start the real Codex sandbox to check workspace writes and policy-source reads. Sandbox setup can change machine-wide Windows state. These tests run by default on GitHub-hosted Windows Actions runners. To include them in a local or self-hosted PowerShell run, set $env:CODEX_SECURITY_TEST_WINDOWS_SANDBOX = '1' before running Bun. The value true also enables them. Set the variable to 0 or false to skip both, even on a hosted runner. The PowerShell machine-policy test has its own CODEX_SECURITY_ALLOW_MACHINE_POLICY_TEST setting.

Check the package

Run static checks, the MCP suite and the SDK suite after installing the dependencies and native payload:

pnpm run check:plugin-source
pnpm run types
pnpm run format
pnpm run test:mcp
pnpm run test

The release checker rejects tar archives with sparse entries. If you created a candidate archive with a sparse-aware tar tool, regenerate it from the source checkout with pnpm pack before signing or publishing it.

To collect JUnit and coverage while running the SDK suite, use pnpm run test:ci in place of pnpm run test. To test what an npm consumer receives, pack the archive and pass its exact path to the inspection command:

pnpm pack --pack-destination ../../dist
pnpm run check:package ../../dist/openai-codex-security-X.Y.Z.tgz
pnpm run test:package

Replace X.Y.Z with the version in the filename printed by pnpm pack. Packing regenerates SDK defaults and project schemas before building the MCP runtime. On Alpine Linux, install GNU tar with apk add tar before running check:package. The BusyBox tar applet lacks the archive options used by the release checker.

The authored plugin lives in plugins/codex-security. pnpm pack generates the ignored _bundled_plugin runtime payload from plugins/codex-security/plugin-files.json during prepack, including the MCP runtime built from mcp-app; do not edit generated files there. Run pnpm run build:plugin when you need to inspect the staged payload locally. Rebuilding replaces the entire generated directory, including any numbered Brotli parts from earlier builds. Each MCP entrypoint now loads its matching .mjs.br file. pnpm run check:plugin-source fails if any file beneath _bundled_plugin is tracked by Git. Required CI runs the same check before installing dependencies.

These checks include native Node contracts for SDK completion, cancellation, close cleanup, and CLI terminal sanitization. They use the installed package, controlled model output, and the real workbench without live model credentials. The installed-package checks also verify hosted SDK exports, input validation, and the scan --host protocol without preparing a repository or requiring Git. The hosted-scan tests cover real repository scans, scope, artifact seals, and executor receipts. To rerun just the installed-package check, use node scripts/smoke-hosted-package.mjs /absolute/path/to/node_modules/@openai/codex-security.

Tests run in random order by default. To replay a failure, pass the seed from Bun's summary to pnpm run test --seed 12345.

The SDK shards in node-ci use the workflow run number as their seed, so rerunning a job repeats its test order. To replay a shard locally, use its platform, shard selector, and seed, for example node --experimental-strip-types scripts/run-ci-tests.mts 2/6 --seed=12345. On Linux, if the original workflow ran the separate MCP tests, prefix that command with CODEX_SECURITY_TEST_MCP_WORKERS_SEPARATE=true to skip the four worker groups they cover. Leave it unset to replay a run that skipped MCP.

The local test commands pass a 30-second per-test timeout explicitly. Windows CI and the Windows runner diagnostics allow 120 seconds for slower native credential and document checks. test:ci writes reports/junit.xml and coverage/lcov.info. Coverage measures loaded JavaScript and TypeScript, not the Python helpers or child processes. It is diagnostic for now. Use several successful CI runs to establish a baseline before proposing a coverage floor.

Native artifacts in fresh development environments

Before running the full SDK tests or Ona’s Run tests task in a fresh checkout, download the verified native package inputs for that checkout. Refresh them whenever the native source or build toolchain changes.

Python test dependencies

Use Python 3.10 or later. From the repository root, install the Python test and lint dependencies with:

python -m pip install --disable-pip-version-check --no-input -r plugins/codex-security/requirements-test.txt

Tests load the plugin scripts from source; the repository does not need an installed Python distribution. The requirements file is the single list of pinned Python test dependencies. Ruff and pytest settings remain in plugins/codex-security/pyproject.toml; the Ruff dependency and its required-version setting must agree.

Writing tests

  • Test observable results, failures, cancellation, and cleanup. Prefer a regression case that fails before a fix over assertions about private calls or exact prose.
  • Pair a fault case with a healthy control when that shows the failure's cause. For ordering bugs, coordinate the relevant events explicitly rather than relying on sleeps: for example, persist a checkpoint before cancellation or fail an operation before its cleanup runs.
  • Keep fixtures synthetic and independent. Use real temporary directories, Git repositories, SQLite databases, and installed packages when those boundaries are the behavior under test. Do not use live model credentials.
  • Follow the changed behavior through its consumer. Parser checks alone do not prove public API output, persisted state, child-process settings, or installed package behavior. Use the relevant boundary without repeating every test at every layer. See the source map.
  • When a result can be partial, assert what was retained and what remains unknown. Distinguish empty results from unavailable results, and checkpointed worker output from accepted results and published scan artifacts.
  • Use the typed TestClient and createApiTestFixtures helpers for API tests. Do not add a production abstraction solely to support a mock.
  • Test workflow retry, resume, and checkpoint permutations with the existing injected workbench dependency. scriptedWorkbench rejects unexpected requests; checkpointWorkbench stores review results without reproducing the Python workflow engine. Keep real workbench calls in the focused workflow integration file for process termination, migration, and source snapshots.
  • Test Python database behavior by calling the production functions in pytest. The workbench_db fixture copies a migrated, empty schema into a fresh in-memory SQLite database for each test, with foreign keys enabled. Use file-backed databases for migrations, reopening, locking, and crash recovery; an in-memory database cannot exercise those boundaries.
  • Restore spies, timers, and environment changes. Tests that change the process cwd or install persistent ESM module mocks use runTestInSubprocess. Per-file Bun isolation does not isolate process-wide state inside one file.
  • Keep shared behavior enabled on Linux, macOS, and Windows. The constrained PowerShell test changes machine-wide policy and runs alone, only on an explicitly enabled GitHub-hosted Windows runner.
  • Add property tests for meaningful invariants, with accepted and rejected inputs. Keep example-based regression tests for readable failure cases.
  • Give parameterized cases distinct names. Use %p, %j, or %# for values that are not strings. Duplicate identities remain valid; the JUnit comparison checks their occurrence and outcome counts.

Property tests use a fixed default seed. Fast-check prints the seed, shrink path, and counterexample on failure. To replay one property, select its file and test name, then set CODEX_SECURITY_PROPERTY_SEED and CODEX_SECURITY_PROPERTY_PATH to the reported values. Set CODEX_SECURITY_PROPERTY_RUNS to increase the case count. Pure properties default to 100 cases; filesystem contract properties default to 20.

GitHub Actions

node-ci reports one ci gate for all selected required checks. The CI guide describes documentation-only routing, platform/version selection, and full scheduled qualification.

For source qualification, CI builds and inspects one package archive, then passes that archive to jobs in the same workflow run, using the commit SHA in the artifact name. Selected Node runtimes install and inspect the package, including a strict NodeNext TypeScript consumer, the actual CLI, credential locking, and a nested Codex worker. The installed Deep Scan smoke exercises a real retry on one runtime per OS for PRs and main pushes, and on every scheduled or manual qualification lane. All selected runtimes still complete direct and detached scans. Native plugin-build tests remain in the shared Bun suite. Each supported OS exercises the maintainer release preview, Git branch and commit updates, concurrent ref conflicts, and preservation of maintainer edits. Typechecking and formatting run once in an independent required job, so package consumers do not wait for those checks. Package compilation and archive validation still finish before the test jobs start.

Source qualification runs the full Bun suite once per OS under Node 22: six file shards on Linux and macOS, and fourteen on Windows. GitHub runs the shards as separate jobs; each job uses Bun's default serial runner. The other selected Node versions run the installed-package checks instead of repeating the same Bun suite. MCP CI assigns files by measured duration and starts the longest files first, using two isolated Node test processes per job. From plugins/codex-security/mcp-app, replay shard 1 with node --experimental-strip-types ../../../sdk/typescript/scripts/run-mcp-test-shard.mts 1 2. Local and release test:mcp commands run the full suite. Python tests run separately and use four isolated pytest-xdist workers with work stealing; worker crashes fail the run without automatic restarts.

scripts/run-ci-tests.mts discovers every shared test file and balances whole files using the platform's recorded durations in scripts/test-timings.json. It assigns longer files first to the shard with the least estimated work, then passes each shard's files directly to Bun. Every job uses the same checked-in snapshot. New files are included automatically with the mean known-file duration; deleted files in the timing snapshot are ignored. Timing hints never select or exclude tests, and adding or splitting tests does not require updating them. To reproduce one Windows shard locally after building the plugin, run node --experimental-strip-types scripts/run-ci-tests.mts 3/14 --seed=12345.

Refresh timing hints from successful CI JUnit artifacts when shard runtimes become uneven. From the SDK directory, pass reports for one platform to node --experimental-strip-types scripts/update-test-timings.mts win32 <junit.xml> ... (use linux or darwin for the other platforms). The updater sums testcase durations for executed cases per file and uses their median across reports. A report that skipped every case in a file leaves its timing hint unchanged. Other platforms retain their hints. Review and commit the generated snapshot. Keep runner versions, coverage, timeouts, seeds, and platform-specific checks unchanged when comparing scheduling changes; compare complete JUnit inventories and outcomes as well as wall time.

Every Bun lane uploads JUnit; Linux lanes also upload LCOV per shard. Python reports include case durations, and the MCP runner can upload its JUnit report. Coverage is diagnostic and split across shards, not a combined percentage. Failed tests and missing package artifacts block CI; failed diagnostic uploads do not. Use python -m pytest -n 0 to reproduce Python failures serially.

Each week test-quality runs the mutation trial on Linux. On Linux, macOS, and Windows it runs the test files that import the shared property options, using the workflow run number as the property seed. The filesystem properties use the same bundled plugin as the SDK tests to read the example scan and validate its sealed artifacts. Each platform uploads its JUnit report. To replay the weekly jobs, dispatch with compare-layouts disabled. Ordinary source pull requests run focused contracts for the shard runner, timing metadata, report comparator, and Windows test-root helper in the required SDK suite.

For a scheduling comparison, run test-quality with compare-layouts enabled. It runs the same serial Bun test inventory under two shard layouts. Compare the aggregate JUnit identities, outcomes, occurrence counts, and assertion counts for each OS, then review Linux LCOV artifacts for coverage differences. The job summary includes failed shard timings. Compare both elapsed time to a green gate and total runner time.

Use the same Bun version, seed, timeouts and coverage settings across layouts. Run the full local suite separately to exercise cross-file ordering. Qualify changes to Bun or its execution mode separately, including how test failures propagate and whether JUnit and coverage finish writing.

Keep the machine-policy test serial. Do not replace the full required suite with --changed: Python files, schemas, fixtures, and workflows loaded at runtime are not necessarily part of Bun's import graph.

Mutation testing

Mutation tests require Node 22.18+ or 24.11+ because Stryker 10 uses Babel 8. Use the Node version pinned by the mutation job. The SDK’s supported runtimes are listed in its package manifest. The commands below install Stryker from the separate, frozen sdk/mutation lockfile and run it from the SDK directory.

pnpm run test:mutation
pnpm run test:mutation --mutate src/worker-progress.ts

The Stryker configuration covers progress parsing, safe error messages, and pure cost arithmetic. It uses the fixed default property-test seed and writes HTML and JSON under reports/mutation. Review surviving mutants for missing behavior assertions or equivalent changes. There is no score gate yet; set one only after the trial has a stable, useful baseline. Do not hide surviving mutants with assertions about implementation details.