Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
The SSE-replay format-translation harness, the shared resources a test must never reach, hermetic pane tests against a real magmux, and the opt-in live pane tests.
Extracted from
CLAUDE.md(v7.64.0). Indexed inREADME.md.
Test Infrastructure
Format Translation Test Harness
packages/cli/src/format-translation.test.ts — SSE replay tests for the full translation pipeline.
Fixture-based: Each .sse file in test-fixtures/sse-responses/ is a captured SSE stream from a real provider response. Tests replay fixtures through the stream parser and assert correct Claude SSE output.
Helpers: parseClaudeSseStream(), extractText(), extractToolNames(), extractStopReason(), fixtureToResponse()
Adding regression tests: After extracting fixtures from a debug log, add a describe("Regression: <model>") block. Template is at the bottom of the test file.
Three shared resources a test must never reach, and why prevention beats restore
scripts/guard-real-config.ts — the thing bun run test:safe runs the suite
under — does two separate jobs, and conflating them is how each keeps decaying.
It SNAPSHOTS files. GUARDED lists ~/.claudish/config.json and
~/.claudish/all-models.json. Each is read before the run and compared after.
Present and changed → restored from the snapshot. Absent before and present
after → DELETED, because on a clean machine a test fixture that survives becomes
the machine's first real catalog, and a cold cache is indistinguishable from a
poisoned one to every caller. Present but unreadable → it refuses and says so,
which is why snapshot() establishes presence with existsSync separately from
reading the bytes: folding "unreadable" into "absent" would make the delete
branch destroy the file.
It PREVENTS three things outright, via env vars set on the child:
| Variable | Stops | Why prevention, not restore |
|---|---|---|
CLAUDISH_DISABLE_KEYCHAIN=1 |
security against the login keychain |
A keychain mutation cannot be snapshotted the way JSON bytes can |
CLAUDISH_DISABLE_OP=1 |
1Password handshakes | Denials suppress authorization machine-wide for 15s, for every process |
CLAUDISH_DISABLE_CATALOG_WARM=1 |
the live hosted-catalog fetch | The restore returns the bytes; it cannot return the hermeticity |
The third was added 2026-09-15 and is gated inside refreshCatalog() rather than
warmCatalog(), because ensureCatalogReady() refreshes through the same
function — a gate on one of two entry points is how a gate stops being true. It
returns reason: "disabled", never "network": a caller that logs "the catalog
could not be reached" when nothing tried to reach it sends the reader to debug a
working connection.
What it caught, and how the leak hid. createProxyServer fires
warmCatalog().catch(() => {}) unconditionally (proxy-server.ts:1193), so
handlers/explicit-spec-no-credential.test.ts live-fetched the hosted catalog
and rewrote the real all-models.json with a fresh lastUpdated — even though
that file's own header says "no network call happens", which was true of the
upstream model call and wrong about the proxy.
It hid itself by RACING. A sibling file cleaned up with
_setCatalogEntriesForTest(null), and null is not a reset: the read path
short-circuits on !== undefined, so it is a sticky process-wide "catalog is
empty" override. Every later lookup returned instantly, the process exited
before the ~2s fetch resolved, and the write never landed. Whether the leak was
visible depended on which files ran alongside it. _resetCatalogClient() is the
real teardown; _setCatalogEntriesForTest(null) belongs only inside a test body
that wants the cold-catalog state.
Evidence: that file tripped the guard 3/3 times alone at ~2.0s, ran clean at 55ms against a dead catalog host, and runs at 66ms with the kill switch. This is the same non-hermeticity that turned two DeepSeek tests red mid-release when the hosted catalog changed its answer.
A gate must not also gate its own diagnostic
e2e-sessions.test.ts Group 2 spawns a real claude -p, so it needs a working credential.
It gates on a genuine probe — one tiny headless prompt, treating a login/credential error as
"not usable" — which is the right design. But the probe itself lived inside
if (!SKIP_LIVE_E2E) { … }.
So running with CLAUDISH_SKIP_LIVE_E2E=1 — which is the normal, recommended way to run this
suite — meant the probe never executed, claudeUsable stayed false, and Group 2 reported:
[e2e-sessions] Group 2 SKIPPED — `claude -p` is unavailable or not authenticated
That message is a claim about the environment that the environment was never asked. It is indistinguishable from the same message on a machine with genuinely no credential, and it was repeated in status reports for hours as "environment-gated" when the real answer was "not asked". Removing the flag ran the probe and recovered the tests; with a credential present, all 15 in the file pass with zero skips.
The general form: a mechanism that reports state must not be disabled by the same switch that disables the work. Where a skip is conditional, its REASON must be computed unconditionally, or the skip message must say "not checked" rather than naming a cause. A diagnostic that is silenced along with the feature cannot tell you why the feature is off — it can only repeat its default.
Same shape as team's exit 0 (team-capture.md): a status whose failure mode is to look
like a confident answer. Prefer "unknown" to a plausible guess in any automated report.
A fixture must live where the test lives, never in scratch space
channel/test-helpers/captured-stream-json.ts (the directory is now mcp-session/) read real
captured stream-json frames at MODULE LOAD, via an IIFE. Its PROBE_DIR originally pointed at
ai-docs/sessions/dev-arch-*/probes/ — where the probe that recorded them happened to write.
.gitignore:56 excludes ai-docs/sessions/. So the files existed on exactly one machine and
in no clone. In CI the readFileSync threw during module init, the exported consts were never
assigned, and every test in the file died with:
ReferenceError: Cannot access 'CAPTURED_ASSISTANT_FRAME' before initialization.
which names a symbol, not a missing file — the real cause is two frames up the stack.
Fixed by copying both captures VERBATIM into packages/cli/src/channel/test-helpers/captures/
and pointing PROBE_DIR at resolve(import.meta.dir, "captures"). Byte-identical: fixtures
come from real logs and must never be regenerated or reformatted in a move. (That helper and its
captures were later deleted with the stream-json child transport; the rule stands, and the pane
fixtures under pane/test-fixtures/ follow it.)
The rule, stated generally: a fixture is BY DEFINITION something meant to outlive the
session that produced it, so a session directory is never its home — however convenient that
is at capture time. CLAUDE.md already warns that ai-docs/sessions/ "does not survive a fresh
clone or git worktree remove" and that "three write-ups already died this way". This was the
fourth, and the first to take a test gate down rather than a document.
The trap is that it is invisible locally in the only direction that matters: the suite is green on the machine that recorded the captures and red everywhere else, so local green is not weaker evidence than CI red — it is ACTIVELY MISLEADING. Same family as the gated diagnostic above: a signal that reports success because the question could only be asked where the answer was already yes.
Cheap check before trusting any new fixture: git ls-files <path> must list it, and
git check-ignore -v <path> must say nothing.
A red local suite may just be a newer bun than CI pins
.github/workflows/test.yml pins bun-version: "1.3.10". A developer machine on a
newer bun runs different bundled Unicode tables, and tui/viz/color.test.ts measures
claudish's displayWidth fallback against a RUNTIME oracle. So the oracle moves with
the runtime while the fallback table does not.
Measured 2026-09-04 on bun 1.4.0: two displayWidth tests fail locally on
U+2630–U+2637 and U+268A–U+268F (Yijing trigram and monogram symbols), whose East
Asian Width classification differs between the two bun releases. The same commit's
Tests run on main is GREEN on the pinned 1.3.10.
Do not "fix" the width table to match the newer bun — that inverts the failure and breaks the pinned CI. Before treating any hermetic red as a regression, check the local runtime against the pinned one and look at whether CI is green on the same SHA. A test that compares against a runtime-provided oracle is only as stable as the runtime.
Distinct from the credential-gated live tests, which skip in CI and are documented as non-blocking; this one is hermetic and still environmental.
The reverse holds too: a GREEN local suite on a newer bun can hide a red CI. Measured
2026-10-06: the pane suites passed 4964/36/0 on bun 1.4.0 and failed 118 tests under the pinned
1.3.10. On 1.3.10, net.connect(path) to a unix socket that does not exist yet emits ENOENT
synchronously inside connect(), before any listener can be attached, and bun test reports it as
an unhandled error; 1.4.0 defers the event. MagmuxClient's dial now attaches its listeners to a
new Socket() before calling connect. Run bunx bun@<pinned> test before calling a change that
touches sockets, timers or child processes green.
Since 2026-09-15 the two budgeted sweeps enforce this themselves. They read
bun-version out of test.yml at test time and test.skipIf when Bun.version
differs, naming both versions in the skip. Three details are load-bearing:
- The pin is PARSED, never copied. A second hardcoded
"1.3.10"in the test file would drift from the workflow, which is the failure being fixed. - A parse failure FAILS OPEN and runs the tests. A gate that silently disables coverage when it cannot find its input is worse than no gate.
- Only the two oracle-calibrated sweeps are gated. The first test in the block asserts five literal widths, is not oracle-dependent, and runs everywhere.
The 2026-09-04 measurement above named U+2630–U+2637 and U+268A–U+268F. The
full sweep on bun 1.4.0 is wider: total 3017, of which the other bucket —
budgeted at 17 — is 576, almost all U+1160..U+11FF, conjoining Hangul Jamo that
1.4.0 measures as zero-width. Same cause, larger blast radius than first recorded.
When the pin is bumped, RE-BASELINE the budgets against the new oracle. Never widen them to clear a red run: the budget is the entire assertion.
A green local suite can be reading the developer's own ~/.claudish
The guard snapshots and prevents writes. It does nothing about READS. A production
fallback that asks "what is on disk under homedir()" answers from the developer's
real state, and Bun keeps homedir() on the real home even when a test assigns
process.env.HOME. The test is then green on every developer machine that has the
file and red on the clean runner, which has none.
Measured 2026-10-20 on PR #301, two cases, both green locally under test:safe and
red on CI; each reproduced locally by making the read return what a clean runner
returns:
| Test | Real file it read | Local | CI |
|---|---|---|---|
openai-codex-refresh-outage.test.ts C2/E2 "unknown refresh" |
~/.claudish/codex-oauth.json, through the billing probe's CodexOAuth.hasCredentials() fallback (billing-probe.ts) |
subscription, 503 | metered, 401 |
model-selector.muse-code.test.ts N3 |
~/.claudish/cloud-models-catalog-v3.json, through isChatCapable → classifyChatCapability with no cachePath |
muse-spark-1.3 is chat |
unknown, row dropped |
The injected dependency was right in both. The read happened several layers below
it, in code the test never named. Two seams now exist for this:
registerSubscriptionCredentialProbe makes the billing probe agree with a fake
credential, and _setChatCapabilityCachePathForTest points the chat filter at a
test-owned catalog.
Before calling a test that reaches routing, billing or the picker green, ask what it
would read with an empty ~/.claudish. CI is the only place that state is empty.
The same CI run showed a third red that was not this class:
team-cli-signal.test.ts exited 0 instead of 130 on SIGINT in 5 of 6 local runs,
on main's code as well. shutdown() in pane/pane-registry.ts awaited a reap
whose last waits (the pane watcher, waitForExit's timer) are unref'd, so Bun
drained the event loop and exited 0 before process.exit(code). The shutdown now
holds the loop with an interval until the reap ends. A test of an exit code under
signals needs --rerun-each: one run hides a race that loses more often than it wins.
A test run never writes into a real session's records (the preload)
A suite run inside a claudish pane (a create_session runner, a team slot) inherits the pane's
session variables, and captureUpstreamError appended the suite's test doubles (stub,
fixture-upstream-error) to ~/.claudish/sessions/<id>/upstream-errors.jsonl, so that session's
get_diagnostics listed them as upstream failures (model-failures-2026-10-21.md, D5).
scripts/test-preload.ts deletes SESSION_ENV_VARS from process.env before the first test file
loads. The six names live in the import-free scripts/session-env-vars.ts, each the literal of the
production constant whose writer follows it: CLAUDISH_UPSTREAM_ERROR_LOG, CLAUDISH_HOOK_INBOX,
CLAUDISH_HOOK_SESSION, CLAUDISH_TOKEN_FILE, CLAUDISH_PUBLISHED_TOKEN_FILE,
CLAUDISH_PANE_CHILD. Neither file imports production code: that would load hooks/, config.ts
and the token tracker ahead of the module-scope snapshots tests rely on. A new session-record
writer means a new name in that list.
Three bunfig.toml files register the preload: the repo root, packages/cli and
packages/macos-bridge. bun reads bunfig.toml from the working directory only and never walks
up (measured on 1.3.10 and 1.4.0), and the suite runs from all three.
The trap: a child spawned with no env option gets the ORIGINAL environment back. Measured
2026-10-22: Bun.spawn/Bun.spawnSync on 1.3.10 and 1.4.0, and node:child_process on 1.3.10 (the
CI pin; 1.4.0's child_process sees the deletion). So the preload covers in-process writers (D5's
observed records) and children given env: { ...process.env }, and two more layers close the rest:
scripts/guard-real-config.tsdeletes the same names from the OS environment of atest:saferun, so no descendant can inherit them.scripts/spawn-env.test.tsfails on anyBun.spawn,Bun.spawnSyncornode:child_processspawn/exec/fork in test code (*.test.ts(x)and anything under atest-helpers/directory) whose options are not an object literal with anenvkey. It parses with the TypeScript compiler API, so a comment cannot satisfy it. A site whose options cannot be read statically goes in its allowlist with the file, the line text and the reason. 42 sites lackedenvwhen it landed; each gotenv: { ...process.env }.
Two pane tests "fail inside a pane by design" (launcher/magmux-wrapper.test.ts,
team-grid.e2e.test.ts). With CLAUDISH_PANE_CHILD, CLAUDISH_PANE_CWD and MAGMUX_SOCK set,
magmux-wrapper measured 16 pass / 9 fail both before and after the preload: the 9 come from
MAGMUX_SOCK (the ambient-multiplexer branch), which is not a session-record variable and stays
inherited. team-grid.e2e was 7 / 0 both times in that simulation.
Clearing provider keys: derive the list, never hand-write it
routing-rules.test.ts cleared a hand list of 24 key names that missed POE_API_KEY and every
bundled vendor's key (D4). test-helpers/provider-credential-env.ts derives the list:
providerCredentialEnvVars() is every apiKeyEnvVar/apiKeyAliases in BUILTIN_PROVIDERS and
PREDEFINED_ENDPOINTS (providerKeyEnvVars()), the credential names no definition declares
(taken from the reader's constant where one exists: DEVIN_API_KEY_ENV,
VERTEX_PROJECT_ENV_VARS), and every CUSTOM_<NAME>_KEY present; providerBaseUrlEnvVars() is
every declared baseUrlEnvVars. clearProviderCredentialEnv() deletes both and returns the
restore function. A test that needs "no provider key" uses it, so a new provider cannot reopen the
hole.
Credential files: seams and a redirected HOME
GrokOAuth, KimiOAuth and the Devin reader re-read their credential files when they change
(credential-files.md), so a test that reaches one without its path seam reads, and through a
refresh may write, the developer's real login. Use setGrokOAuthPathForTesting,
setKimiOAuthPathForTesting, setKimiDeviceIdPathForTesting, KimiOAuth._resetForTests() and
setDevinCredentialsPathForTesting. A test file that imports auth/credentials/authority.ts or
auth/kimi-oauth.ts should also run in a child spawned with HOME set to a temp directory in the
child's environment (Bun resolves homedir() from the environment a process starts with, never
from an in-process assignment), so a missed seam lands there.
Pane tests: a real magmux, a fake child, nothing left behind
MCP team slots and create_session run as interactive Claude Code in headless magmux panes
(pane-session.md), so every suite that starts one — pane/*.integration.test.ts,
pane/pane-registry.test.ts, the MCP session and team suites, mcp-contract.e2e.test.ts,
mcp-shutdown.e2e.test.ts — runs a REAL magmux --headless. The transport's behaviour (frames,
exit events, close_pane, slow-subscriber drops, socket paths) is magmux's, and a simulated
magmux would test our model of it instead.
The child is the fake, never Claude Code. pane/test-helpers/fake-interactive-child.ts is
started through CLAUDISH_BIN (a *.ts value becomes bun --no-env-file run <file>). It draws
the REPL with lines copied from the phase-2 screen fixtures, reads raw keystrokes with the key semantics measured
from the real REPL, and writes its transcript by TEMPLATING real records — never hand-written ones.
A scenario is chosen by --model fake-<scenario>, or, in the contract suites' marker mode
(contract-fake-model), by markers in the prompt: @@TOOL@@, @@HANG@@ (accepted, never
answered), @@LINGER@@ (answers, holds the end-of-turn record), @@LATE@@ (a re-wake after the
answer). A file-delivered prompt is answered ANSWER <model> <sha1 prefix of the reconstructed file>, so a test can prove nothing was lost in delivery. pane-child-real-claudish.integration.test.ts
is the one place the REAL child claudish runs: this tree's src/index.ts launches the fake in
"claude" mode as Claude Code (via CLAUDE_PATH) with a native model, so no proxy request is made,
and the fake dumps the argv, env, cwd and --settings it was given.
The environment is built by ALLOWLIST (makePaneTestEnv(), and serverEnv(layout) for a
real MCP server), never by deleting from process.env: no ANTHROPIC_* or provider key survives
because none is listed, and HOME, CLAUDE_CONFIG_DIR, ZDOTDIR and XDG_CONFIG_HOME all point
into a temp directory. CLAUDISH_PANE_ROOT is a fresh /tmp/cpt-<8 hex> per test or layout,
never the user's /tmp/claudish-mux-<uid>, whose startup sweep and 48-pane limit belong to the user's own
sessions.
Drop CLAUDISH_PANE_CHILD from any env spread from process.env. The flag reaches everything a
claudish-started session runs (pane-session.md § One level of nesting), so a
suite launched from inside one (a runner in a create_session pane) inherits it. A claudish child
spawned with ...process.env then refuses to run a model (exit 64, nested session refused), and
an MCP server spawned that way lists 4 tools instead of 14. Build the env by allowlist, or set
CLAUDISH_PANE_CHILD: undefined explicitly. scripts/guard-real-config.ts deletes the flag for
bun run test:safe as a second layer; a hand-run bun test <file> from such a session gets no
such help.
Every test asserts that nothing outlived it (assertNoOrphans, paneOrphans(layout)): no
process whose argv contains the test's root or a session uuid, no process left in any pane
process group, and no socket, pane record or launcher directory in the root. That assertion is
what proves the reap; a green suite without it would hide a REPL left billing.
A missing magmux is "not checked", never a cause. Without magmux these suites skip with
magmux not installed — not checked (the gated-diagnostic rule above). CI installs it
(brew install MadAppGang/tap/magmux in test.yml), so the pane suites run there on macOS.
Time is a seam, and a budget measures a code path, not wall-clock under contention.
PaneSessionOptions.timings scales the quiet windows, the degraded-mode entry and the resend delay
for tests (production never passes it), and the integration suite runs its groups with
describe.concurrent (43 tests in ~50 s instead of ~250 s), each test owning its env and root. Two
measured traps:
- At a load average of 85–88 from another session, one full run produced 14 timing failures across
the pane, registry and session-manager suites — and also exposed a real defect (
shutdownAllskipped starts still in flight), so a red run under load still needs reading before it is dismissed. - REQ-18 asserted the team record's
meta.jsonwithin 1 s ofteam(mode:"run")returning. With panes,runreturns once each prompt is accepted, so that interval also held the fake's turn and the 500 ms settle corroboration, and it failed at 1,013–1,014 ms under load while the record path took milliseconds. It now measures from the run's settle (the last slot'scompletedAt) to the record'scompletedAt, and still fails (1,505 ms) when the end is delayed onto a later poll. Measure the thing the requirement is about; never just raise the number.
Fixtures are real captures, and a test greps them. pane/test-fixtures/ holds screens,
transcripts, frames, token files and a ps table from Claude Code 2.1.290 in a headless pane
(reports/mcp-magmux-panes/phase2-captures.md); the two corpus slices were redacted with
scripts/redact-transcript-fixture.ts. fixtures-redaction.test.ts fails on a home path, a
credential or an e-mail address in any of them.
Live pane tests are opt-in. pane-live-claude.integration.test.ts drives the REAL Claude Code
(native haiku) through PaneSession and spends money, so it runs only when asked:
CLAUDISH_PANE_LIVE=1 CLAUDE_CODE_OAUTH_TOKEN=<token> \
bun test packages/cli/src/pane/pane-live-claude.integration.test.ts
The config directory is hermetic, and a hermetic config directory is "Not logged in": Claude Code
keys its keychain item by config directory, so redirecting HOME or CLAUDE_CONFIG_DIR loses the
login. The token therefore comes from the operator's environment; the test never reads the
keychain or ~/.claude. Likewise bun run test:mcp (mcp-e2e/) needs live 1Password and the real
config and is not part of test:safe. To drive one session by hand, use scripts/pane-drive.ts
(--fake <scenario> for the hermetic fake).