Skip to content

armgit / Orkas

Public Imported from GitHub
0

Orkas is an open-source (MIT), local-first AI team for your desktop: a Commander turns your goal into to-dos and coordinates 16 built-in specialist agents plus Claude Code, Codex, OpenCode, OpenClaw and Hermes, with projects, scheduled automations and connectors. Free app for macOS & Windows; built-in models or your own keys.

Resume deterministic test lanes with validated evidence

wytfafb721703a7
diff --git a/docs/test/test-infrastructure-regression.md b/docs/test/test-infrastructure-regression.md
index 37f9a0fe..3297ed46 100644
--- a/docs/test/test-infrastructure-regression.md
+++ b/docs/test/test-infrastructure-regression.md
@@ -19,3 +19,80 @@ not evidence of coverage.
| Staged language completeness | Locale-resource or language-registry edits run the existing Main/Renderer i18n cases for the full supported registry; failed cases block the commit. Unstaged validation inputs and inputs changed during checks are rejected, preserving unrelated work | `test/main/util/staged-i18n-gate.test.ts` exercises real Git/hook execution, incomplete translations, placeholder mismatch, partial staging, unavailable runtime and changed inputs; `test/main/i18n.test.ts` and `test/renderer/i18n.test.ts` own language semantics | Not applicable; local commit workflow | Not applicable |
| Windows CLI fixture runtime | A desktop test process that can see `git.exe` but resolves `bash` to the System32 WSL placeholder must still execute Git for Windows shell contracts; command fixtures with `.cmd` launchers must be invoked through their real Windows entry point | `test/main/util/test-runtime-env.test.ts`, the separately authorized model canary, and the separately authorized model canary | Not applicable | Not applicable |
+
+## Full non-model runs and recovery
+
+From the application root, run the existing typecheck, unit/resource, E2E,
+platform-native and smoke lanes in sequence:
+
+```bash
+npm run test:non-model -- --checkpoint-dir=/tmp/orkas-non-model
+npm run test:non-model -- --checkpoint-dir=/tmp/orkas-non-model --resume
+```
+
+Native, Office and Electron tests share desktop processes. Do not run another
+native or GUI suite concurrently on the same host. The unified entry preserves
+its existing lanes and stops at a failed lane. No live model evaluation is added.
+Checkpoints cover JS, builtin Python resources and desktop E2E; typechecks,
+platform-native and smoke lanes execute afresh. Each family uses a separate
+checkpoint namespace so its evidence cannot overwrite another family's progress.
+The JS runner uses native Vitest discovery and eight sequential shards by default,
+with the existing per-file isolation and worker limit. Use `--batch-count=N`
+(1–32) to change granularity; it does not increase concurrency. A direct JS run
+can also use `npm run test:js -- --checkpoint-dir=/tmp/orkas-unit-run`.
+For JS, file filters and `--maxWorkers=N` are supported; coverage, watch mode,
+custom configs/reporters and test-name filters use the ordinary runner.
+
+Python and E2E also use sequential file batches, preserving parameterized cases,
+module fixtures and serial tests within a file. Each batch creates a fresh process;
+a partial file restarts from its beginning. Every invocation rediscovers actual
+case IDs, and their final union must exactly match discovery. Python fingerprints
+include its interpreter, standard library and external import dependencies.
+Python accepts test paths and `-q`/`-v`; E2E accepts file filters with the default
+Playwright config. Debugging, custom configs and other flags use the ordinary entry.
+For example, from the application root:
+
+```bash
+node scripts/run-python-tests.mjs resources/test -q --checkpoint-dir=test-results/checkpoint --resume
+npm run test:e2e -- app_e2e_smoke.spec.ts --checkpoint-dir=test-results/checkpoint --resume
+```
+
+Python captures per-case setup/call/teardown output, logging and warnings. E2E
+retains worker output, Electron log snapshots and attachments in unique batch
+output directories, with structured JSON reports instead of the ordinary HTML
+report. The existing log policy gates unexpected errors and unclassified stderr;
+warnings remain reviewable. Retry attempts do not qualify an E2E batch for reuse.
+Missing or modified reports, captured events, logs or attachments force a rerun.
+Tests that consume their own captured output, or dependencies that never send logs
+to these captured streams, still need their owning assertions/log review; a
+checkpoint does not create evidence for an unobserved stream.
+
+`--resume` reuses only completed batches with intact stdout/stderr and structured
+log-analysis artifacts. Failed, interrupted or uncertified batches run again;
+warnings remain visible, and skips remain distinct from passes. Each invocation
+rediscovers files and verifies that the final batch union exactly covers that
+inventory. Source metadata, installed dependency metadata, runtime, environment,
+selection or inventory changes invalidate reuse. Private configuration contents
+are never read for the fingerprint. Checkpoints expire after 12 hours and are
+local to the same host. External service state, native application updates or
+OS permission changes are not certified by this fingerprint: start a fresh run
+without `--resume` after such changes.
+
+Each invocation keeps a separate report directory, including original failed
+attempts, file setup/collection/test durations and log summaries. Successful-case
+console replay is suppressed in checkpoint mode, but every record still passes
+through the existing worker log analyzer. Exact-line aggregate digests reconcile
+raw process stderr with worker-analyzed output; missing or additional bytes fail
+the batch without storing raw text in the structured summary. Inspect the reported
+warning signatures and raw failure logs; a timing
+report alone is not verification evidence. `summary.json` separates newly executed
+and reused results, wall time, input-scan overhead and avoided batch execution
+time. Input scanning adds overhead, so use checkpoints for long/full runs rather
+than tiny local selections. The root entry also records per-lane durations under
+`test-results/non-model-*.json`.
+
+Only one invocation may write a checkpoint directory. Cancellation releases its
+lock; after a forced kill, confirm no associated run remains before removing
+`run.lock`. Old attempt directories are deliberately retained for diagnosis;
+remove the whole checkpoint directory when that evidence is no longer needed.
+
diff --git a/package.json b/package.json
index e0445c77..5c58baa4 100644
--- a/package.json
+++ b/package.json
@@ -22,7 +22,7 @@
"typecheck:main": "tsc --noEmit",
"typecheck:core-agent": "tsc --noEmit -p src/core-agent/tsconfig.json",
"test": "node scripts/run-tests.mjs suite",
- "test:non-model": "npm run typecheck && npm test && npm run test:e2e:typecheck && npm run test:e2e && npm run test:platform-native && npm run smoke",
+ "test:non-model": "node scripts/run-non-model.mjs",
"test:js": "node scripts/run-tests.mjs run",
"test:e2e": "node scripts/run-e2e.cjs test --config playwright.config.ts",
"test:e2e:debug": "node scripts/run-e2e.cjs test --config playwright.config.ts --debug",
diff --git a/scripts/checkpoint-reporter.ts b/scripts/checkpoint-reporter.ts
new file mode 100644
index 00000000..e2137267
--- /dev/null
+++ b/scripts/checkpoint-reporter.ts
@@ -0,0 +1,69 @@
+import fs from 'node:fs';
+import path from 'node:path';
+import { createHash } from 'node:crypto';
+import type { Reporter, FullConfig, Suite, TestCase, TestResult, FullResult } from '@playwright/test/reporter';
+import { analyzeWebAppLogs } from '../test/e2e/fixtures/web-app-log-analysis';
+
+/** Structured outcomes, not console text, determine completion. */
+export default class CheckpointReporter implements Reporter {
+ private report = process.env.ORKAS_TEST_BATCH_REPORT!;
+ private events = this.report.replace(/\.json$/, '.events.jsonl');
+ private inventory: string[] = [];
+ private files: Array<{ file: string; state: string; passed: number; failed: number; skipped: number; durationMs: number }> = [];
+ private testFiles: string[] = [];
+ private errors = 0;
+ private appAnalyses: Array<ReturnType<typeof analyzeWebAppLogs> & { case: string }> = [];
+ private artifacts: string[] = [];
+ private outputDir = '';
+
+ onBegin(config: FullConfig, suite: Suite) {
+ this.outputDir = config.projects[0].outputDir;
+ this.inventory = suite.allTests().map(test => test.id);
+ this.testFiles = [...new Set(suite.allTests().map(test => test.location.file))].sort();
+ fs.writeFileSync(this.events, '', { mode: 0o600 });
+ }
+ onStdOut(chunk: string | Buffer, test?: TestCase) { this.event('stdout', chunk, test); }
+ onStdErr(chunk: string | Buffer, test?: TestCase) { this.event('stderr', chunk, test); }
+ private event(stream: string, chunk: string | Buffer, test?: TestCase) {
+ fs.appendFileSync(this.events, JSON.stringify({ case: test?.id || 'setup', phase: 'worker', stream, text: chunk.toString() }) + '\n');
+ }
+ onError() { this.errors++; }
+ onTestEnd(test: TestCase, result: TestResult) {
+ const failed = result.status !== test.expectedStatus && result.status !== 'skipped';
+ // A retry is preserved as a finding and never certified for reuse.
+ const state = failed || result.retry > 0 ? 'failed' : result.status === 'skipped' ? 'skipped' : 'passed';
+ this.files = this.files.filter(file => file.file !== test.id);
+ this.files.push({ file: test.id, state, passed: Number(state === 'passed'), failed: Number(state === 'failed'),
+ skipped: Number(state === 'skipped'), durationMs: result.duration });
+ const requiredLogs = result.attachments.filter(item => item.name === 'electron-log-required').length;
+ if (requiredLogs !== result.attachments.filter(item => item.name === 'electron-logs').length) this.errors++;
+ let recovered: Parameters<typeof analyzeWebAppLogs>[1] = [];
+ for (const [index, original] of result.attachments.entries()) {
+ const attachment = { ...original };
+ if (!attachment.path && attachment.body) {
+ const id = createHash('sha256').update(`${test.id}/${result.retry}/${index}`).digest('hex');
+ attachment.path = path.join(this.outputDir, 'checkpoint-attachments', id);
+ fs.mkdirSync(path.dirname(attachment.path), { recursive: true });
+ fs.writeFileSync(attachment.path, attachment.body, { mode: 0o600 });
+ }
+ if (!attachment.path) continue;
+ this.artifacts.push(attachment.path);
+ if (attachment.name === 'electron-log-required') {
+ try { recovered = JSON.parse(fs.readFileSync(attachment.path, 'utf8')); }
+ catch { this.errors++; }
+ }
+ if (attachment.name === 'electron-logs') {
+ try {
+ const analysis = analyzeWebAppLogs(fs.readFileSync(attachment.path, 'utf8'), recovered);
+ this.appAnalyses.push({ ...analysis, case: test.id });
+ recovered = [];
+ } catch { this.errors++; }
+ }
+ }
+ }
+ onEnd(result: FullResult) {
+ fs.writeFileSync(this.report, JSON.stringify({ schema: 1, reason: result.status === 'passed' ? 'passed' : 'failed',
+ batchUnits: this.testFiles.length, testFiles: this.testFiles, unhandledErrors: this.errors, inventory: this.inventory, files: this.files, appAnalyses: this.appAnalyses,
+ artifactPaths: this.artifacts, outputDir: this.outputDir }), { mode: 0o600 });
+ }
+}
diff --git a/scripts/pytest_checkpoint.py b/scripts/pytest_checkpoint.py
new file mode 100644
index 00000000..cc64d69c
--- /dev/null
+++ b/scripts/pytest_checkpoint.py
@@ -0,0 +1,121 @@
+"""Checkpoint protocol for pytest; keeps module/session fixtures within a shard."""
+import json
+import logging
+import os
+import signal
+from pathlib import Path
+
+import pytest
+
+
+def pytest_addoption(parser):
+ parser.addoption('--orkas-checkpoint-shard', default='1/1')
+
+
+class Checkpoint:
+ def __init__(self, config):
+ self.config = config
+ self.path = Path(os.environ['ORKAS_TEST_BATCH_REPORT'])
+ self.events = self.path.with_suffix('.events.jsonl')
+ self.events.write_text('', encoding='utf-8')
+ self.inventory = []
+ self.results = {}
+ self.errors = 0
+ self.current = 'collection'
+ self.phase = 'collection'
+ self.signal_handlers = {}
+ for name in ('SIGTERM', 'SIGHUP'):
+ value = getattr(signal, name, None)
+ if value is not None:
+ self.signal_handlers[value] = signal.getsignal(value)
+ signal.signal(value, self.interrupt)
+ owner = self
+
+ class Handler(logging.Handler):
+ def emit(self, record):
+ level = 'error' if record.levelno >= 40 else 'warn' if record.levelno >= 30 else 'info'
+ owner.event('stdout', f'[00:00:00.000] [{level}] [(python)] {record.getMessage()}\n')
+ self.handler = Handler()
+ logging.getLogger().addHandler(self.handler)
+
+ @staticmethod
+ def interrupt(signum, frame):
+ # Let pytest unwind fixtures and stop its children before the parent exits.
+ raise KeyboardInterrupt
+
+ def event(self, stream, text):
+ with self.events.open('a', encoding='utf-8') as output:
+ output.write(json.dumps({'case': self.current, 'phase': self.phase, 'stream': stream, 'text': text}) + '\n')
+
+ @pytest.hookimpl(trylast=True)
+ def pytest_collection_modifyitems(self, session, config, items):
+ part, total = map(int, config.getoption('--orkas-checkpoint-shard').split('/'))
+ files = sorted({str(item.path) for item in items})
+ selected_files = set(files[part - 1::total])
+ kept = [item for item in items if str(item.path) in selected_files]
+ removed = [item for item in items if str(item.path) not in selected_files]
+ if removed:
+ config.hook.pytest_deselected(items=removed)
+ items[:] = kept
+ self.inventory = [item.nodeid for item in items]
+ self.batch_units = len(files)
+
+ def pytest_collectreport(self, report):
+ if report.failed:
+ self.errors += 1
+ for name, text in report.sections:
+ self.event('stderr' if name == 'Captured stderr' else 'stdout', text)
+
+ def pytest_runtest_logstart(self, nodeid, location):
+ self.current = nodeid
+
+ def pytest_runtest_setup(self, item):
+ self.phase = 'setup'
+
+ def pytest_runtest_call(self, item):
+ self.phase = 'call'
+
+ def pytest_runtest_teardown(self, item):
+ self.phase = 'teardown'
+
+ def pytest_runtest_logreport(self, report):
+ result = self.results.setdefault(report.nodeid, {'phases': {}, 'durationMs': 0})
+ result['phases'][report.when] = report.outcome
+ result['durationMs'] += round(report.duration * 1000)
+ for name, text in report.sections:
+ # pytest includes previous phases in later reports. Read each once.
+ if name == f'Captured stdout {report.when}':
+ self.event('stdout', text)
+ elif name == f'Captured stderr {report.when}':
+ self.event('stderr', text)
+
+ def pytest_warning_recorded(self, warning_message, when, nodeid, location):
+ previous = self.current
+ self.current = nodeid or 'collection'
+ self.event('stdout', f'[00:00:00.000] [warn] [(python-warning)] {warning_message.message}\n')
+ self.current = previous
+
+ @pytest.hookimpl(trylast=True)
+ def pytest_sessionfinish(self, session, exitstatus):
+ files = []
+ for nodeid, result in self.results.items():
+ phases = result['phases']
+ skipped = 'skipped' in phases.values()
+ passed = phases.get('setup') == 'passed' and phases.get('call') == 'passed' and phases.get('teardown') == 'passed'
+ failed = 'failed' in phases.values() or phases.get('teardown') != 'passed'
+ state = 'failed' if failed else 'passed' if passed else 'skipped' if skipped else 'failed'
+ files.append({'file': nodeid, 'state': state, 'passed': int(state == 'passed'),
+ 'failed': int(state == 'failed'), 'skipped': int(state == 'skipped'),
+ 'durationMs': result['durationMs']})
+ self.path.write_text(json.dumps({'schema': 1, 'reason': 'passed' if exitstatus == 0 else 'failed',
+ 'unhandledErrors': self.errors, 'inventory': self.inventory,
+ 'files': files, 'batchUnits': getattr(self, 'batch_units', 0)}), encoding='utf-8')
+
+ def pytest_unconfigure(self, config):
+ logging.getLogger().removeHandler(self.handler)
+ for value, handler in self.signal_handlers.items():
+ signal.signal(value, handler)
+
+
+def pytest_configure(config):
+ config.pluginmanager.register(Checkpoint(config), 'orkas-checkpoint-protocol')
diff --git a/scripts/run-e2e.cjs b/scripts/run-e2e.cjs
index 675cda8e..5795eb5c 100644
--- a/scripts/run-e2e.cjs
+++ b/scripts/run-e2e.cjs
@@ -20,4 +20,11 @@ if (process.env.NO_COLOR !== undefined) {
delete process.env.NO_COLOR;
}
-require('playwright/lib/program').program.parse(process.argv);
+if (args.some(arg => arg === '--resume' || arg.startsWith('--checkpoint-dir') || arg.startsWith('--batch-count'))) {
+ import('./runner-checkpoint.mjs').then(({ runRunnerCheckpoint }) =>
+ runRunnerCheckpoint('e2e', args, { cwd: process.cwd(), env: process.env }))
+ .then(code => { process.exitCode = code; })
+ .catch(error => { console.error(`[run-e2e] ${error.message}`); process.exitCode = 1; });
+} else {
+ require('playwright/lib/program').program.parse(process.argv);
+}
diff --git a/scripts/run-non-model.mjs b/scripts/run-non-model.mjs
new file mode 100644
index 00000000..d00f54ef
--- /dev/null
+++ b/scripts/run-non-model.mjs
@@ -0,0 +1,72 @@
+#!/usr/bin/env node
+
+import { existsSync, mkdirSync, writeFileSync } from 'node:fs';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { spawnSync } from 'node:child_process';
+import { randomUUID } from 'node:crypto';
+import { checkpointOptions } from './test-checkpoint.mjs';
+import { withWindowsGitOnPath } from './test-runtime-env.mjs';
+
+const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..');
+let checkpointArgs;
+try {
+ const checkpoint = checkpointOptions(process.argv.slice(2));
+ if (checkpoint.args.length) throw new Error('Only --checkpoint-dir, --batch-count and --resume are supported; run coverage independently');
+ checkpointArgs = checkpoint.directory ? [
+ `--checkpoint-dir=${resolve(checkpoint.directory)}`,
+ `--batch-count=${checkpoint.batches}`,
+ ...(checkpoint.resume ? ['--resume'] : []),
+ ] : [];
+} catch (error) {
+ console.error(`[non-model] ${error.message}`);
+ process.exit(2);
+}
+
+const testEnvironment = withWindowsGitOnPath(process.env);
+function npmInvocation() {
+ if (process.platform !== 'win32') return { command: 'npm', prefix: [] };
+ const candidates = [testEnvironment.npm_execpath];
+ const where = spawnSync('where.exe', ['npm.cmd'], { encoding: 'utf8', env: testEnvironment });
+ if (where.status === 0) {
+ for (const launcher of where.stdout.split(/\r?\n/).map(line => line.trim()).filter(Boolean)) {
+ candidates.push(resolve(dirname(launcher), 'node_modules', 'npm', 'bin', 'npm-cli.js'));
+ }
+ }
+ const cli = candidates.find(candidate => candidate && existsSync(candidate));
+ if (!cli) throw new Error('npm CLI not found; install npm or invoke this runner from npm');
+ // Node cannot directly spawn a .cmd launcher on Windows. Keep arguments out
+ // of a command shell and execute the same installed npm JavaScript entry.
+ return { command: process.execPath, prefix: [cli] };
+}
+
+let npm;
+try { npm = npmInvocation(); }
+catch (error) {
+ console.error(`[non-model] ${error.message}`);
+ process.exit(2);
+}
+
+const reportPath = resolve(repoRoot, 'test-results', `non-model-${randomUUID()}.json`);
+const stages = [];
+console.log(`[non-model] stage timing report: ${reportPath}`);
+
+// Preserve the flat repository's existing six-stage shell chain. In particular,
+// npm test already owns both the JS and builtin Python suites. Their launchers
+// and the E2E launcher own checkpoint discovery, reuse and log certification.
+// Native Office/Electron lanes share host resources and must remain serial.
+for (const script of ['typecheck', 'test', 'test:e2e:typecheck', 'test:e2e', 'test:platform-native', 'smoke']) {
+ const args = [...npm.prefix, ...(script === 'test' ? ['test'] : ['run', script]),
+ ...(['test', 'test:e2e'].includes(script) && checkpointArgs.length ? ['--', ...checkpointArgs] : [])];
+ const started = performance.now();
+ const result = spawnSync(npm.command, args, { cwd: repoRoot, stdio: 'inherit', env: testEnvironment });
+ const durationMs = Math.round(performance.now() - started);
+ const execution = result.error ? 'unavailable' : result.signal ? 'interrupted' : 'completed';
+ stages.push({ script, command: npm.command, args, durationMs, exitCode: result.status, signal: result.signal, execution });
+ mkdirSync(dirname(reportPath), { recursive: true });
+ writeFileSync(reportPath, `${JSON.stringify({ schema: 1, stages,
+ logReview: 'Inspect each owning runner report and all emitted logs' }, null, 2)}\n`);
+ console.log(`[non-model] ${script} duration=${durationMs}ms exit=${result.status ?? 'unavailable'} execution=${execution}`);
+ if (result.error) console.error(`[non-model] failed to start ${script}: ${result.error.message}`);
+ if (result.error || result.signal || result.status !== 0) process.exit(result.status || 1);
+}
diff --git a/scripts/run-python-tests.mjs b/scripts/run-python-tests.mjs
index 9a6313b7..c2d9081a 100644
--- a/scripts/run-python-tests.mjs
+++ b/scripts/run-python-tests.mjs
@@ -7,13 +7,11 @@ import { spawnSync } from 'node:child_process';
import { withWindowsGitOnPath } from './test-runtime-env.mjs';
const here = dirname(fileURLToPath(import.meta.url));
-// The open-source app is a flat repository (unlike the private monorepo, which
-// nests the desktop app under PC/), so resolve both the test venv and pytest
-// working directory from this repository root.
-const appRoot = resolve(here, '..');
+// The open-source app and its test virtual environment share this flat root.
+const repoRoot = resolve(here, '..');
const localPython = process.platform === 'win32'
- ? resolve(appRoot, 'venv', 'Scripts', 'python.exe')
- : resolve(appRoot, 'venv', 'bin', 'python');
+ ? resolve(repoRoot, 'venv', 'Scripts', 'python.exe')
+ : resolve(repoRoot, 'venv', 'bin', 'python');
const testEnvironment = withWindowsGitOnPath(process.env);
const candidates = [testEnvironment.ORKAS_TEST_PYTHON, localPython, 'python3', 'python']
.filter((value, index, all) => value && all.indexOf(value) === index);
@@ -32,15 +30,27 @@ if (!python) {
testEnvironment.ORKAS_TEST_PYTHON = python;
const args = process.argv.slice(2);
+const cwdFlag = args.findIndex(arg => arg.startsWith('--cwd='));
+const cwd = cwdFlag < 0 ? repoRoot : resolve(args.splice(cwdFlag, 1)[0].slice(6));
if (args.length === 0) args.push('resources/builtin');
-const result = spawnSync(python, ['-m', 'pytest', ...args], {
- cwd: appRoot,
- stdio: 'inherit',
- env: testEnvironment,
-});
+if (args.some(arg => arg === '--resume' || arg.startsWith('--checkpoint-dir') || arg.startsWith('--batch-count'))) {
+ try {
+ const { runRunnerCheckpoint } = await import('./runner-checkpoint.mjs');
+ process.exitCode = await runRunnerCheckpoint('python', args, { cwd, env: testEnvironment, python });
+ } catch (error) {
+ console.error(`[run-python-tests] ${error.message}`);
+ process.exitCode = 1;
+ }
+} else {
+ const result = spawnSync(python, ['-m', 'pytest', ...args], {
+ cwd,
+ stdio: 'inherit',
+ env: testEnvironment,
+ });
-if (result.error) {
- console.error(`[run-python-tests] failed to start pytest: ${result.error.message}`);
- process.exit(1);
+ if (result.error) {
+ console.error(`[run-python-tests] failed to start pytest: ${result.error.message}`);
+ process.exit(1);
+ }
+ process.exit(result.status ?? 1);
}
-process.exit(result.status ?? 1);
diff --git a/scripts/run-tests.mjs b/scripts/run-tests.mjs
index 1ff88cea..1bb70db5 100644
--- a/scripts/run-tests.mjs
+++ b/scripts/run-tests.mjs
@@ -26,12 +26,28 @@ const require_ = createRequire(import.meta.url);
const electronBin = require_('electron');
const vitestBin = resolve(here, '..', 'node_modules', 'vitest', 'vitest.mjs');
const testEnvironment = withTestPython(withWindowsGitOnPath(process.env));
+// Nested launcher calls must not overwrite their parent's checkpoint report.
+delete testEnvironment.ORKAS_TEST_BATCH_REPORT;
// npm appends arguments to the final shell command of a compound script.
// Own the suite here so filters reach Vitest without first running every JS
// test or being interpreted by npm/pytest. No arguments retains the full suite.
const suiteMode = process.argv[2] === 'suite';
-const args = process.argv.slice(suiteMode ? 3 : 2);
+let args = process.argv.slice(suiteMode ? 3 : 2);
+let checkpoint;
+let runCheckpoint;
+if (args.some(arg => arg === '--resume' || arg.startsWith('--checkpoint-dir') || arg.startsWith('--batch-count'))) {
+ try {
+ const helper = await import('./test-checkpoint.mjs');
+ checkpoint = helper.checkpointOptions(!suiteMode && args[0] === 'run' ? args.slice(1) : args);
+ if (!suiteMode && args[0] !== 'run') throw new Error('Checkpoint mode requires run or suite');
+ args = suiteMode ? checkpoint.args : ['run', ...checkpoint.args];
+ runCheckpoint = helper.runCheckpoint;
+ } catch (error) {
+ console.error(`[run-tests] ${error.message}`);
+ process.exit(2);
+ }
+}
const commands = [{
executable: electronBin,
args: [vitestBin, ...(suiteMode ? ['run', ...args] : args)],
@@ -44,11 +60,14 @@ const commands = [{
// outer npm/node executable as the explicit test-helper runtime.
ORKAS_TEST_NODE: process.execPath,
},
+ checkpoint,
}];
if (suiteMode && args.length === 0) {
commands.push({
executable: process.execPath,
- args: [resolve(here, 'run-python-tests.mjs'), 'resources/builtin', 'resources/test', '-q'],
+ args: [resolve(here, 'run-python-tests.mjs'), 'resources/builtin', 'resources/test', '-q',
+ ...(checkpoint?.directory ? [`--checkpoint-dir=${resolve(checkpoint.directory)}`, `--batch-count=${checkpoint.batches}`,
+ ...(checkpoint.resume ? ['--resume'] : [])] : [])],
env: testEnvironment,
});
}
@@ -74,12 +93,33 @@ for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP', 'SIGQUIT']) {
process.on(signal, handler);
}
-function runNext() {
+async function runNext() {
const command = commands.shift();
if (!command) {
removeSignalHandlers();
return;
}
+ if (command.checkpoint?.directory) {
+ let code;
+ try {
+ code = await runCheckpoint({ ...command.checkpoint, executable: electronBin, vitestBin,
+ env: command.env, onChild: active => { child = active; }, cancelled: () => forwardedSignal });
+ } catch (error) {
+ if (error.signal) forwardedSignal ||= error.signal;
+ else console.error(`[run-tests] ${error.message}`);
+ code = 1;
+ }
+ if (forwardedSignal) {
+ removeSignalHandlers();
+ process.kill(process.pid, forwardedSignal);
+ return;
+ }
+ if (code !== 0) {
+ removeSignalHandlers();
+ process.exit(code);
+ }
+ return runNext();
+ }
child = spawn(command.executable, command.args, { stdio: 'inherit', env: command.env });
child.once('error', (error) => {
removeSignalHandlers();
diff --git a/scripts/runner-checkpoint.mjs b/scripts/runner-checkpoint.mjs
new file mode 100644
index 00000000..1496f29a
--- /dev/null
+++ b/scripts/runner-checkpoint.mjs
@@ -0,0 +1,152 @@
+/** pytest and Playwright adapters for the shared checkpoint lifecycle. */
+import fs from 'node:fs';
+import path from 'node:path';
+import { createHash } from 'node:crypto';
+import { createRequire } from 'node:module';
+import { fileURLToPath } from 'node:url';
+import { spawnSync } from 'node:child_process';
+import { createInterface } from 'node:readline';
+import { checkpointOptions, runCheckpoint } from './test-checkpoint.mjs';
+import { createStderrReceipt } from './test-stderr-receipt.mjs';
+import { createModelEvalLogCollector, mergeModelEvalLogAnalyses } from './model-eval-log-policy.mjs';
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+const require = createRequire(import.meta.url);
+const maxBytes = 16 * 1024 * 1024;
+const readJson = file => {
+ if (fs.statSync(file).size > maxBytes) throw new Error('Checkpoint report exceeds its size limit');
+ return JSON.parse(fs.readFileSync(file, 'utf8'));
+};
+const digest = async file => {
+ const hash = createHash('sha256');
+ for await (const chunk of fs.createReadStream(file)) hash.update(chunk);
+ return hash.digest('hex');
+};
+
+async function finish(reportPath, attempt, directory, discovery) {
+ if (!fs.existsSync(reportPath)) return;
+ const report = readJson(reportPath);
+ const analyses = [];
+ const receipt = createStderrReceipt();
+ for (const stream of ['stdout', 'stderr']) {
+ const collector = createModelEvalLogCollector(`process-${stream}`);
+ for await (const chunk of fs.createReadStream(path.join(directory, attempt[stream]), { encoding: 'utf8' })) {
+ collector.ingest(stream, chunk);
+ if (stream === 'stderr') receipt.ingest(chunk);
+ }
+ analyses.push(collector.finish());
+ }
+ const eventsPath = reportPath.replace(/\.json$/, '.events.jsonl');
+ const scopes = new Map();
+ const input = fs.createReadStream(eventsPath, { encoding: 'utf8' });
+ try {
+ for await (const line of createInterface({ input, crlfDelay: Infinity })) {
+ if (line.length > maxBytes) throw new Error('Checkpoint event exceeds its size limit');
+ const event = JSON.parse(line);
+ if (!['stdout', 'stderr'].includes(event.stream) || typeof event.text !== 'string') throw new Error('Invalid checkpoint event');
+ const key = JSON.stringify([event.case, event.phase]);
+ if (!scopes.has(key)) {
+ if (scopes.size >= 100_000) throw new Error('Checkpoint event inventory exceeds its size limit');
+ scopes.set(key, createModelEvalLogCollector(key));
+ }
+ scopes.get(key).ingest(event.stream, event.text);
+ }
+ } finally { input.destroy(); }
+ analyses.push(...[...scopes.values()].map(scope => scope.finish()), ...(report.appAnalyses || []));
+ const analysis = mergeModelEvalLogAnalyses(analyses);
+ report.logs = { status: !analysis.passed ? 'FAIL' : analysis.requiresReview ? 'WARN' : 'PASS',
+ cases: report.files.reduce((sum, file) => sum + file.passed + file.failed, 0),
+ files: report.files.filter(file => file.passed + file.failed > 0).length,
+ unexpected: analysis.findings.filter(item => item.severity === 'error').reduce((sum, item) => sum + item.count, 0),
+ unmetExpectations: 0, coverageGaps: analysis.coveragePassed ? [] : ['capture'], analysis };
+ report.stderr = receipt.finish();
+ report.discoveryPassed = discovery && report.reason === 'passed' && report.unhandledErrors === 0 && analysis.passed;
+ const artifactPaths = [eventsPath, ...(report.artifactPaths || [])];
+ // Playwright owns attachments, traces and its last-run metadata. Each batch
+ // has a unique output directory so a later batch cannot erase failed evidence.
+ if (!discovery && report.outputDir && fs.existsSync(report.outputDir)) {
+ for (const file of fs.readdirSync(report.outputDir, { recursive: true, withFileTypes: true })) {
+ if (file.isFile()) artifactPaths.push(path.join(file.parentPath || file.path, file.name));
+ }
+ }
+ report.artifacts = [];
+ for (const file of new Set(artifactPaths)) {
+ const relative = path.relative(directory, file);
+ if (path.isAbsolute(relative) || relative.split(path.sep).includes('..')) throw new Error('Checkpoint artifact escaped its output directory');
+ report.artifacts.push({ file: relative, hash: await digest(file) });
+ }
+ delete report.artifactPaths;
+ delete report.outputDir;
+ fs.writeFileSync(reportPath, JSON.stringify(report, null, 2), { mode: 0o600 });
+}
+
+export async function runRunnerCheckpoint(lane, argv, { cwd, env, python } = {}) {
+ const options = checkpointOptions(argv, lane);
+ let args = options.args;
+ if (lane === 'python' && args.length === 0) args = ['resources/builtin'];
+ env = { ...env };
+ delete env.ORKAS_TEST_BATCH_REPORT;
+ let executable = process.execPath;
+ let prefix;
+ let extraRoots = [];
+ let discoveryArgs;
+ let batchArgs;
+ let testFiles;
+ const inventory = report => { testFiles = report?.testFiles; return report?.inventory?.map(file => ({ file })); };
+ if (lane === 'python') {
+ if (args.some(arg => arg.startsWith('-') && !['-q', '-v'].includes(arg))) throw new Error('Python checkpoints accept paths and -q/-v only');
+ env.PYTHONDONTWRITEBYTECODE = '1';
+ env.PYTHONPATH = [here, env.PYTHONPATH].filter(Boolean).join(path.delimiter);
+ const probe = spawnSync(python, ['-c', 'import sys, json, sysconfig, pytest; print(json.dumps({"executable":sys.executable,"prefix":sys.prefix,"stdlib":sysconfig.get_path("stdlib"),"version":sys.version,"pytest":pytest.__version__,"paths":sys.path}))'],
+ { cwd, env, encoding: 'utf8', timeout: 15_000 });
+ if (probe.status !== 0) throw new Error('Cannot verify the project Python interpreter and pytest');
+ const runtime = JSON.parse(probe.stdout);
+ executable = runtime.executable;
+ const repository = path.resolve(here, '..');
+ extraRoots = [...new Set([runtime.prefix, runtime.stdlib, ...runtime.paths.filter(Boolean)])]
+ .filter(root => { const relative = path.relative(repository, path.resolve(root));
+ return relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative); });
+ env.ORKAS_CHECKPOINT_PYTHON_RUNTIME = JSON.stringify(runtime);
+ prefix = ['-m', 'pytest', '-p', 'pytest_checkpoint'];
+ discoveryArgs = selection => [...selection, '--collect-only'];
+ batchArgs = (batch, batches, selection) => [...selection, `--orkas-checkpoint-shard=${batch}/${batches}`];
+ } else {
+ if (args[0] === 'test') args = args.slice(1);
+ // Keep the normal product config, reporter and output ownership explicit.
+ if (args[0] === '--config' && args[1] === 'playwright.config.ts') args = args.slice(2);
+ if (args.some(arg => arg.startsWith('-'))) throw new Error('E2E checkpoints accept file filters only with the default config');
+ env.ORKAS_E2E_KEEP_LOGS = '1';
+ prefix = [path.join(path.dirname(require.resolve('playwright/package.json')), 'cli.js')];
+ const reporter = path.join(here, 'checkpoint-reporter.ts');
+ discoveryArgs = (selection, report) => ['test', '--config', 'playwright.config.ts', '--list', `--reporter=${reporter}`, ...selection];
+ batchArgs = (batch, batches, selection, report) => ['test', '--config', 'playwright.config.ts',
+ `--reporter=line,${reporter}`, `--output=${report}.artifacts`,
+ ...testFiles.filter((_, index) => index % batches === batch - 1)
+ .map(file => `^${file.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`)];
+ }
+ // A Python lane is keyed by cwd and selected suites, not by a caller label.
+ // Thus root suites and builtin resources cannot overwrite each other's state.
+ const key = createHash('sha256').update(JSON.stringify([lane, cwd, args])).digest('hex').slice(0, 16);
+ const directory = path.join(path.resolve(options.directory), `${lane}-${key}`);
+ let child;
+ let signal;
+ const handlers = new Map();
+ for (const name of ['SIGINT', 'SIGTERM', 'SIGHUP', 'SIGQUIT']) {
+ const handler = () => { signal ||= name; if (child && !child.killed) child.kill(name); };
+ handlers.set(name, handler); process.on(name, handler);
+ }
+ let code;
+ try {
+ code = await runCheckpoint({ ...options, args, directory, executable, env,
+ onChild: value => { child = value; }, cancelled: () => signal,
+ adapter: { lane, cwd, prefix, extraRoots, discoveryArgs, batchArgs, finish,
+ inventory } });
+ } catch (error) {
+ if (error.signal) signal ||= error.signal;
+ else throw error;
+ } finally {
+ for (const [name, handler] of handlers) process.off(name, handler);
+ }
+ if (signal) { process.kill(process.pid, signal); return 1; }
+ return code;
+}
diff --git a/scripts/test-checkpoint.mjs b/scripts/test-checkpoint.mjs
new file mode 100644
index 00000000..917417ab
--- /dev/null
+++ b/scripts/test-checkpoint.mjs
@@ -0,0 +1,288 @@
+/** Resumable Vitest batches. No application/test module state crosses batches. */
+import fs from 'node:fs';
+import path from 'node:path';
+import os from 'node:os';
+import { createHash, randomUUID } from 'node:crypto';
+import { spawn, spawnSync } from 'node:child_process';
+import { createStderrReceipt } from './test-stderr-receipt.mjs';
+
+const MAX_REPORT_BYTES = 16 * 1024 * 1024;
+const MAX_AGE_MS = 12 * 60 * 60 * 1000;
+const slash = value => value.replaceAll('\\', '/');
+
+export function checkpointOptions(argv, lane = 'vitest') {
+ const args = [];
+ const options = { directory: '', resume: false, batches: 8 };
+ for (let i = 0; i < argv.length; i++) {
+ const arg = argv[i];
+ if (arg === '--resume') options.resume = true;
+ else if (arg === '--checkpoint-dir' || arg.startsWith('--checkpoint-dir=')) {
+ options.directory = arg === '--checkpoint-dir' ? argv[++i] : arg.slice('--checkpoint-dir='.length);
+ if (!options.directory || options.directory.startsWith('--')) throw new Error('--checkpoint-dir needs a directory');
+ } else if (arg === '--batch-count' || arg.startsWith('--batch-count=')) {
+ options.batches = Number(arg === '--batch-count' ? argv[++i] : arg.slice('--batch-count='.length));
+ } else args.push(arg);
+ }
+ if (!Number.isInteger(options.batches) || options.batches < 1 || options.batches > 32) throw new Error('--batch-count must be between 1 and 32');
+ if (!options.directory && (options.resume || argv.some(arg => arg === '--batch-count' || arg.startsWith('--batch-count=')))) {
+ throw new Error('Checkpoint options require --checkpoint-dir');
+ }
+ // Keep collection and execution equivalent. Watch, coverage, custom reporters,
+ // alternate projects/configs and test-name filters need their own evidence.
+ if (options.directory && lane === 'vitest' && args.some(arg => arg.startsWith('-') && !/^--maxWorkers=[1-9]\d*$/.test(arg))) {
+ throw new Error('Checkpoint mode accepts file filters and --maxWorkers=N only; use the ordinary runner for other options');
+ }
+ return { args, ...options };
+}
+
+function readJson(file) {
+ try {
+ if (fs.statSync(file).size > MAX_REPORT_BYTES) return null;
+ return JSON.parse(fs.readFileSync(file, 'utf8'));
+ } catch { return null; }
+}
+
+function writeJson(file, value) {
+ const temporary = `${file}.tmp`;
+ fs.writeFileSync(temporary, `${JSON.stringify(value, null, 2)}\n`, { mode: 0o600 });
+ fs.renameSync(temporary, file);
+}
+
+async function hashFile(file) {
+ const hash = createHash('sha256');
+ for await (const chunk of fs.createReadStream(file)) hash.update(chunk);
+ return hash.digest('hex');
+}
+
+async function stderrMatches(file, expected) {
+ if (!expected?.valid) return false;
+ const receipt = createStderrReceipt();
+ for await (const chunk of fs.createReadStream(file)) receipt.ingest(chunk);
+ const actual = receipt.finish();
+ return actual.valid && actual.bytes === expected.bytes && actual.lines === expected.lines && actual.digest === expected.digest;
+}
+
+// Metadata, not file bodies: tracked private configuration must never be read.
+// ctime also invalidates restored timestamps. Scan once at each batch boundary,
+// never per case; output directories are outside the source inventory.
+export function inputFingerprint(repoRoot, env, selection, inventory, executable, extraRoots = []) {
+ const git = spawnSync('git', ['ls-files', '-z', '--cached', '--others', '--exclude-standard', '--deduplicate'], {
+ cwd: repoRoot, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024,
+ });
+ if (git.status !== 0) throw new Error('Cannot establish the repository input inventory');
+ const hash = createHash('sha256');
+ const add = value => hash.update(JSON.stringify(value) + '\n');
+ add({ schema: 1, host: os.hostname(), release: os.release(), platform: process.platform, arch: process.arch,
+ versions: process.versions, executable, cwd: repoRoot, selection, inventory,
+ env: Object.entries(env).sort(([a], [b]) => a.localeCompare(b)) });
+ let entries = 0;
+ const record = (file, knownStat) => {
+ if (++entries > 500_000) throw new Error('Input inventory exceeds the checkpoint safety limit');
+ try {
+ const stat = knownStat || fs.lstatSync(file, { bigint: true });
+ add([file, stat.mode.toString(), stat.size.toString(), stat.mtimeNs.toString(), stat.ctimeNs.toString(), stat.ino.toString()]);
+ if (stat.isSymbolicLink()) {
+ const target = fs.realpathSync(file);
+ const resolved = fs.statSync(target, { bigint: true });
+ add([target, resolved.size.toString(), resolved.mtimeNs.toString(), resolved.ctimeNs.toString()]);
+ }
+ } catch (error) {
+ if (error.code !== 'ENOENT') throw new Error('Cannot inspect a required checkpoint input');
+ add([file, 'missing']);
+ }
+ };
+ for (const file of git.stdout.split('\0').filter(Boolean).sort()) record(path.join(repoRoot, file));
+ const visited = new Set();
+ const walk = file => {
+ let stat;
+ try { stat = fs.lstatSync(file, { bigint: true }); }
+ catch (error) { if (error.code === 'ENOENT') { record(file); return; } throw error; }
+ if (stat.isSymbolicLink()) {
+ record(file, stat);
+ // Follow dependency links as well as recording the link itself.
+ if (fs.existsSync(file)) walk(fs.realpathSync(file));
+ return;
+ }
+ if (!stat.isDirectory()) { record(file, stat); return; }
+ if (visited.has(file)) return;
+ visited.add(file);
+ for (const entry of fs.readdirSync(file).sort()) {
+ // Vitest writes these caches while collecting/executing unchanged inputs.
+ if (path.basename(file) === 'node_modules' && ['.vite', '.vite-temp', '.cache'].includes(entry)) continue;
+ walk(path.join(file, entry));
+ }
+ };
+ for (const root of ['node_modules', 'src/core-agent/node_modules', 'venv',
+ 'resources/runtime', 'resources/embedding-model', 'resources/officecli', 'src/main/native/build', 'out', ...extraRoots]) {
+ walk(path.resolve(repoRoot, root));
+ }
+ record(process.execPath);
+ record(executable);
+ if (env.ORKAS_TEST_PYTHON && path.isAbsolute(env.ORKAS_TEST_PYTHON)) record(env.ORKAS_TEST_PYTHON);
+ return hash.digest('hex');
+}
+
+function validReport(report) {
+ return report?.schema === 1 && report.reason === 'passed' && report.unhandledErrors === 0
+ && report.stderr?.valid === true
+ && Array.isArray(report.inventory) && Array.isArray(report.files)
+ && new Set(report.inventory).size === report.inventory.length
+ && JSON.stringify([...report.inventory].sort()) === JSON.stringify(report.files.map(file => file.file).sort())
+ && report.files.every(file => ['passed', 'skipped'].includes(file.state) && file.failed === 0
+ && [file.passed, file.failed, file.skipped].every(count => Number.isInteger(count) && count >= 0))
+ && report.logs?.cases === report.files.reduce((count, file) => count + file.passed + file.failed, 0)
+ && report.logs?.files === report.files.filter(file => file.passed + file.failed > 0).length
+ && ['PASS', 'WARN'].includes(report.logs?.status) && report.logs.unexpected === 0
+ && report.logs.unmetExpectations === 0 && report.logs.coverageGaps?.length === 0;
+}
+
+export async function runCheckpoint({ directory, resume, batches, args, executable, vitestBin, env, onChild, cancelled = () => null, adapter }) {
+ const started = performance.now();
+ let inputScanMs = 0;
+ const pcRoot = adapter?.cwd || path.resolve(path.dirname(vitestBin), '../..');
+ const git = spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: pcRoot, encoding: 'utf8' });
+ if (git.status !== 0) throw new Error('Checkpoint mode requires a Git checkout');
+ const repoRoot = git.stdout.trim();
+ directory = path.resolve(directory);
+ const relative = path.relative(repoRoot, directory);
+ if (!relative.startsWith(`..${path.sep}`) && !path.isAbsolute(relative)) {
+ const ignored = spawnSync('git', ['check-ignore', '-q', path.join(directory, 'checkpoint.json')], { cwd: repoRoot });
+ if (ignored.status !== 0) throw new Error('Checkpoint directory must be Git-ignored or outside the checkout');
+ }
+ fs.mkdirSync(directory, { recursive: true, mode: 0o700 });
+ const lock = path.join(directory, 'run.lock');
+ let lockFd;
+ try { lockFd = fs.openSync(lock, 'wx', 0o600); }
+ catch { throw new Error('Checkpoint is locked; stop the other run before removing a stale run.lock'); }
+ fs.writeSync(lockFd, String(process.pid));
+ const session = randomUUID();
+ const sessionDir = path.join(directory, session);
+ fs.mkdirSync(sessionDir, { mode: 0o700 });
+ const run = async (label, commandArgs, report) => {
+ const started = performance.now();
+ const stdout = path.join(session, `${label}.stdout.log`);
+ const stderr = path.join(session, `${label}.stderr.log`);
+ const out = fs.openSync(path.join(directory, stdout), 'wx', 0o600);
+ const err = fs.openSync(path.join(directory, stderr), 'wx', 0o600);
+ let result;
+ try {
+ result = await new Promise((resolve, reject) => {
+ const child = spawn(executable, [...(adapter?.prefix || [vitestBin]), ...commandArgs], {
+ cwd: pcRoot, env: { ...env, ORKAS_TEST_BATCH_REPORT: report || '' }, stdio: ['inherit', out, err],
+ });
+ onChild?.(child);
+ child.once('error', () => reject(new Error('Cannot start the checkpoint test runtime')));
+ child.once('close', (exitCode, signal) => resolve({ exitCode, signal }));
+ });
+ } finally { fs.closeSync(out); fs.closeSync(err); }
+ const attempt = { ...result, durationMs: Math.round(performance.now() - started), stdout, stderr,
+ stdoutHash: await hashFile(path.join(directory, stdout)), stderrHash: await hashFile(path.join(directory, stderr)) };
+ writeJson(path.join(sessionDir, `${label}.attempt.json`), attempt);
+ if (result.signal) throw Object.assign(new Error('Checkpoint run interrupted'), { signal: result.signal });
+ return attempt;
+ };
+ try {
+ const discoveryPath = path.join(sessionDir, 'inventory.json');
+ const discovery = await run('discovery', adapter
+ ? adapter.discoveryArgs(args, discoveryPath)
+ : ['list', '--filesOnly', `--json=${discoveryPath}`, ...args], adapter ? discoveryPath : undefined);
+ if (adapter) await adapter.finish(discoveryPath, discovery, directory, true);
+ const discoveryReport = readJson(discoveryPath);
+ const discovered = adapter ? adapter.inventory(discoveryReport) : discoveryReport;
+ if (discovery.exitCode !== 0 || (adapter ? !discoveryReport?.discoveryPassed : fs.statSync(path.join(directory, discovery.stderr)).size !== 0)
+ || !Array.isArray(discovered) || !discovered.length
+ || discovered.some(item => typeof item.file !== 'string' || item.projectName)) {
+ throw new Error('Test discovery failed or returned an unsupported inventory; inspect discovery logs');
+ }
+ if (adapter) console.log(`[test-checkpoint] discovery log analysis ${discoveryReport.logs.status}`);
+ const inventory = discovered.map(item => adapter ? item.file : slash(path.relative(pcRoot, item.file))).sort();
+ if (new Set(inventory).size !== inventory.length) throw new Error('Duplicate test discovery entries');
+ batches = Math.min(batches, adapter ? discoveryReport.batchUnits : inventory.length);
+ if (!Number.isInteger(batches) || batches < 1) throw new Error('Invalid checkpoint batch inventory');
+ const fingerprint = () => {
+ const scanStarted = performance.now();
+ const result = inputFingerprint(repoRoot, env, { args, batches, ...(adapter ? { lane: adapter.lane } : {}) }, inventory, executable, adapter?.extraRoots);
+ inputScanMs += performance.now() - scanStarted;
+ return result;
+ };
+ const initial = fingerprint();
+ const statePath = path.join(directory, 'checkpoint.json');
+ const previous = resume ? readJson(statePath) : null;
+ const reusable = previous?.schema === 1 && previous.fingerprint === initial
+ && Date.now() - previous.createdAt < MAX_AGE_MS && Date.now() >= previous.createdAt;
+ const state = reusable ? previous : { schema: 1, fingerprint: initial, createdAt: Date.now(), inventory, completed: {} };
+ writeJson(statePath, state);
+ console.log(`[test-checkpoint] discovered=${inventory.length} batches=${batches}; ${reusable ? 'checking saved evidence' : 'fresh execution'}`);
+ const results = [];
+ let changed = false;
+ for (let batch = 1; batch <= batches; batch++) {
+ if (cancelled()) throw Object.assign(new Error('Checkpoint run interrupted'), { signal: cancelled() });
+ let entry = state.completed[batch];
+ let report;
+ if (entry) {
+ try {
+ const safe = file => typeof file === 'string' && !path.isAbsolute(file) && !file.split(/[\\/]/).includes('..');
+ if (![entry.report, entry.stdout, entry.stderr].every(safe)) throw new Error('Invalid artifact location');
+ report = readJson(path.join(directory, entry.report));
+ if (entry.certified !== true || entry.stderrGap || !validReport(report) || await hashFile(path.join(directory, entry.report)) !== entry.reportHash
+ || await hashFile(path.join(directory, entry.stdout)) !== entry.stdoutHash
+ || await hashFile(path.join(directory, entry.stderr)) !== entry.stderrHash) entry = null;
+ if (entry && adapter) {
+ if (!Array.isArray(report.artifacts)) entry = null;
+ else for (const artifact of report.artifacts) {
+ if (!safe(artifact.file) || await hashFile(path.join(directory, artifact.file)) !== artifact.hash) entry = null;
+ }
+ }
+ } catch { entry = null; }
+ }
+ if (entry) {
+ results.push({ batch, reused: true, ...entry, files: report.files, logs: report.logs });
+ console.log(`[test-checkpoint] batch ${batch}/${batches} reused; log analysis ${report.logs.status}`);
+ continue;
+ }
+ delete state.completed[batch];
+ writeJson(statePath, state);
+ console.log(`[test-checkpoint] batch ${batch}/${batches} starting`);
+ const reportFile = path.join(session, `batch-${batch}.json`);
+ // The worker tap still analyzes every case log and surfaces warnings in
+ // the report. Suppress only the duplicate successful-case console replay
+ // so any remaining process stderr has no ambiguous attribution.
+ const attempt = await run(`batch-${batch}`, adapter
+ ? adapter.batchArgs(batch, batches, args, path.join(directory, reportFile))
+ : ['run', '--silent=passed-only', `--shard=${batch}/${batches}`, ...args], path.join(directory, reportFile));
+ if (adapter) await adapter.finish(path.join(directory, reportFile), attempt, directory, false);
+ report = readJson(path.join(directory, reportFile));
+ const stderrGap = attempt.exitCode === 0 && !await stderrMatches(path.join(directory, attempt.stderr), report?.stderr);
+ const certified = attempt.exitCode === 0 && validReport(report) && !stderrGap;
+ if (fingerprint() !== initial) changed = true;
+ entry = { ...attempt, report: reportFile, reportHash: report ? await hashFile(path.join(directory, reportFile)) : '', certified, stderrGap };
+ if (certified && !changed) state.completed[batch] = entry;
+ if (changed) state.completed = {};
+ writeJson(statePath, state);
+ results.push({ batch, reused: false, ...entry, files: report?.files || [], logs: report?.logs });
+ console.log(`[test-checkpoint] batch ${batch}/${batches} ${certified ? 'completed' : 'FAILED'} ${attempt.durationMs}ms; log analysis ${report?.logs?.status || 'MISSING'}`);
+ if (stderrGap) console.log('[test-checkpoint] unclassified stderr: inspect this batch stderr artifact before rerunning');
+ }
+ // A cached or newly executed shard may never hide an omitted/new file.
+ const executed = results.flatMap(result => result.files.map(file => file.file)).sort();
+ const complete = JSON.stringify(executed) === JSON.stringify(inventory);
+ changed ||= fingerprint() !== initial;
+ const exitCode = !complete || changed || results.some(result => !result.certified) ? 1 : 0;
+ // Missing evidence for one batch must not discard independently certified
+ // siblings. A fully certified but inconsistent union invalidates all shards.
+ if (changed || (!complete && results.every(result => result.certified))) {
+ state.completed = {}; writeJson(statePath, state);
+ }
+ const summary = { schema: 1, inventory, complete, inputsChanged: changed, exitCode,
+ wallDurationMs: Math.round(performance.now() - started), inputScanMs: Math.round(inputScanMs),
+ reused: results.filter(result => result.reused).length,
+ savedDurationMs: results.filter(result => result.reused).reduce((sum, result) => sum + result.durationMs, 0), results };
+ writeJson(path.join(sessionDir, 'summary.json'), summary);
+ console.log(`[test-checkpoint] executed=${batches - summary.reused} reused=${summary.reused} saved=${summary.savedDurationMs}ms inventory=${complete ? 'complete' : 'INCOMPLETE'} inputs=${changed ? 'CHANGED' : 'stable'}`);
+ console.log(`[test-checkpoint] reports: ${path.join(sessionDir, 'summary.json')}`);
+ return exitCode;
+ } finally {
+ fs.closeSync(lockFd);
+ fs.unlinkSync(lock);
+ }
+}
diff --git a/scripts/test-runtime-env.mjs b/scripts/test-runtime-env.mjs
index 62b18d09..08519de0 100644
--- a/scripts/test-runtime-env.mjs
+++ b/scripts/test-runtime-env.mjs
@@ -5,7 +5,7 @@ import { fileURLToPath } from 'node:url';
/** Keep deterministic evaluator fixtures on the same project interpreter as
* the test suite, including direct npm test invocations. Explicit evaluator
* overrides remain independent of the ordinary Python suite. */
-export function withTestPython(env, repoRoot = fileURLToPath(new URL('../../', import.meta.url)), platform = process.platform) {
+export function withTestPython(env, repoRoot = fileURLToPath(new URL('../', import.meta.url)), platform = process.platform) {
const result = { ...env };
const localPython = platform === 'win32'
? path.join(repoRoot, 'venv', 'Scripts', 'python.exe')
diff --git a/scripts/test-stderr-receipt.mjs b/scripts/test-stderr-receipt.mjs
new file mode 100644
index 00000000..66ce008e
--- /dev/null
+++ b/scripts/test-stderr-receipt.mjs
@@ -0,0 +1,61 @@
+import { createHash } from 'node:crypto';
+
+const MODULUS = 1n << 256n;
+const MAX_LINE_BYTES = 1024 * 1024;
+
+// Multiset digest of exact lines: concurrent workers may reorder whole lines,
+// but missing, added, modified or interleaved bytes cannot inherit attribution.
+// Only an aggregate digest/count leaves the worker, never raw diagnostic text.
+export function createStderrReceipt() {
+ let lineHash = createHash('sha256');
+ let lineBytes = 0;
+ let sum = 0n;
+ let bytes = 0;
+ let lines = 0;
+ let valid = true;
+ const finishLine = () => {
+ sum = (sum + BigInt(`0x${lineHash.digest('hex')}`)) % MODULUS;
+ lines += 1;
+ lineHash = createHash('sha256');
+ lineBytes = 0;
+ };
+ return {
+ ingest(chunk, encoding) {
+ const data = Buffer.isBuffer(chunk) ? chunk : typeof chunk === 'string' ? Buffer.from(chunk, encoding) : Buffer.from(chunk);
+ bytes += data.length;
+ if (!valid) return;
+ let start = 0;
+ while (start < data.length) {
+ const newline = data.indexOf(10, start);
+ const end = newline === -1 ? data.length : newline + 1;
+ lineBytes += end - start;
+ if (lineBytes > MAX_LINE_BYTES) { valid = false; return; }
+ lineHash.update(data.subarray(start, end));
+ if (newline !== -1) finishLine();
+ start = end;
+ }
+ },
+ finish() {
+ if (valid && lineBytes) finishLine();
+ return { valid, bytes, lines, digest: sum.toString(16).padStart(64, '0') };
+ },
+ };
+}
+
+export function combineStderrReceipts(receipts) {
+ let sum = 0n;
+ let bytes = 0;
+ let lines = 0;
+ let valid = true;
+ for (const receipt of receipts) {
+ if (!receipt?.valid || !Number.isSafeInteger(receipt.bytes) || receipt.bytes < 0
+ || !Number.isSafeInteger(receipt.lines) || receipt.lines < 0 || !/^[a-f0-9]{64}$/.test(receipt.digest)) {
+ valid = false;
+ continue;
+ }
+ sum = (sum + BigInt(`0x${receipt.digest}`)) % MODULUS;
+ bytes += receipt.bytes;
+ lines += receipt.lines;
+ }
+ return { valid, bytes, lines, digest: sum.toString(16).padStart(64, '0') };
+}
diff --git a/test/e2e/fixtures/orkas.ts b/test/e2e/fixtures/orkas.ts
index fd467301..0c3530f7 100644
--- a/test/e2e/fixtures/orkas.ts
+++ b/test/e2e/fixtures/orkas.ts
@@ -1835,6 +1835,10 @@ export class OrkasTestApp {
for (const record of await this.rendererErrorEvidence()) {
if (record.windowId !== null && windows.has(record.windowId)) this.recoveredRendererErrors.add(record.diagnosticIndex);
}
+ return this.recoveredDiagnosticRanges();
+ }
+
+ private recoveredDiagnosticRanges() {
let line = 1;
const ranges: Array<{ start: number; end: number; diagnosticIndex: number }> = [];
this.diagnostics.forEach((entry, diagnosticIndex) => {
@@ -1867,6 +1871,17 @@ export class OrkasTestApp {
failed = true;
}
+ // Export receipts after releasing native/stub ownership. Evidence IO must
+ // fail certification without interrupting resource cleanup.
+ if (process.env.ORKAS_TEST_BATCH_REPORT) {
+ try {
+ await this.testInfo.attach('electron-log-required', { body: Buffer.from(JSON.stringify(this.recoveredDiagnosticRanges())), contentType: 'application/json' });
+ } catch (error) {
+ cleanupError = cleanupError ? new AggregateError([cleanupError, error], 'Fixture cleanup failed') : error;
+ failed = true;
+ }
+ }
+
const unexpectedApiFailures = this.apiResponses.filter(item => {
if (item.status < 400) return false;
const index = this.expectedApiFailures.findIndex(expected => expected.method === item.method
@@ -1876,7 +1891,7 @@ export class OrkasTestApp {
return false;
});
failed ||= unexpectedApiFailures.length > 0 || this.expectedApiFailures.length > 0;
- if (failed || process.env.ORKAS_E2E_KEEP_LOGS === '1') {
+ if (failed || process.env.ORKAS_E2E_KEEP_LOGS === '1' || process.env.ORKAS_TEST_BATCH_REPORT) {
mkdirSync(this.testInfo.outputDir, { recursive: true });
writeFileSync(this.testInfo.outputPath('stub-responses.json'), JSON.stringify(this.apiResponses, null, 2));
}
@@ -1901,7 +1916,8 @@ export class OrkasTestApp {
// reporters truncate an inline attachment to a couple of lines, which is
// exactly the main-process history a failed run needs. `attach({ path })`
// keeps the full file in the artifacts tree.
- const logPath = path.join(this.testInfo.outputDir, 'electron-main.log');
+ const logPath = path.join(this.testInfo.outputDir, process.env.ORKAS_TEST_BATCH_REPORT
+ ? `checkpoint-electron-${randomUUID()}.log` : 'electron-main.log');
try {
mkdirSync(this.testInfo.outputDir, { recursive: true });
writeFileSync(logPath, diagnosticText, 'utf8');
@@ -1921,15 +1937,24 @@ export class OrkasTestApp {
// ORKAS_E2E_KEEP_LOGS=1 preserves the same captured stdout/stderr,
// renderer diagnostics and app logs as a failed run. File logs alone
// omit direct console output, including renderer warnings.
- if (process.env.ORKAS_E2E_KEEP_LOGS === '1') {
+ if (process.env.ORKAS_E2E_KEEP_LOGS === '1' || process.env.ORKAS_TEST_BATCH_REPORT) {
const appLogs = collectLogFiles(this.root);
const diagnosticText = [this.diagnostics.join('\n'), appLogs].filter(Boolean).join('\n');
if (diagnosticText) {
try {
mkdirSync(this.testInfo.outputDir, { recursive: true });
writeFileSync(path.join(this.testInfo.outputDir, 'electron-main.log'), diagnosticText, 'utf8');
- } catch {
- // Diagnostics only — never fail a passing test over log capture.
+ if (process.env.ORKAS_TEST_BATCH_REPORT) {
+ const snapshot = this.testInfo.outputPath(`checkpoint-electron-${randomUUID()}.log`);
+ writeFileSync(snapshot, diagnosticText, 'utf8');
+ await this.testInfo.attach('electron-logs', { path: snapshot, contentType: 'text/plain' });
+ }
+ } catch (error) {
+ // Fail reuse after releasing the temporary root and trace files.
+ if (process.env.ORKAS_TEST_BATCH_REPORT) {
+ cleanupError = new Error('Checkpoint Electron log capture failed', { cause: error });
+ }
+ // Ordinary runs preserve their existing diagnostic behavior.
}
}
}
diff --git a/test/fixtures/test-log-analysis/stateful.fixture.ts b/test/fixtures/test-log-analysis/stateful.fixture.ts
new file mode 100644
index 00000000..008d36a7
--- /dev/null
+++ b/test/fixtures/test-log-analysis/stateful.fixture.ts
@@ -0,0 +1,20 @@
+import { it } from 'vitest';
+import { expectTestLog } from '../../helpers/test-log-analysis';
+
+it.each([
+ ['global', /fixture repeated diagnostic/g],
+ ['sticky', /fixture repeated diagnostic/y],
+ ['frozen-global', Object.freeze(/fixture repeated diagnostic/g)],
+ ['frozen-sticky', Object.freeze(/fixture repeated diagnostic/y)],
+] as const)('matches every repeated %s declaration', (_kind, message) => {
+ expectTestLog({ level: 'error', message });
+ console.error('fixture repeated diagnostic');
+ console.error('fixture repeated diagnostic');
+});
+
+it('still rejects a different error after repeated declared output', () => {
+ expectTestLog({ level: 'error', message: /fixture declared diagnostic/g });
+ console.error('fixture declared diagnostic');
+ console.error('fixture declared diagnostic');
+ console.error('fixture different unexpected diagnostic');
+});
diff --git a/test/fixtures/test-log-analysis/vitest.config.ts b/test/fixtures/test-log-analysis/vitest.config.ts
index 9fb1d8b6..f690b830 100644
--- a/test/fixtures/test-log-analysis/vitest.config.ts
+++ b/test/fixtures/test-log-analysis/vitest.config.ts
@@ -5,6 +5,10 @@ import { fileURLToPath } from 'node:url';
import { defineConfig } from 'vitest/config';
import base from '../../../vitest.config';
+// Exercise checkpoint receipts through the real launcher without inheriting a
+// parent run's report destination.
+process.env.ORKAS_TEST_BATCH_REPORT = process.env.ORKAS_LOG_FIXTURE_BATCH_REPORT || '';
+
const reporters = [...(base.test?.reporters as unknown[])];
if (process.env.ORKAS_LOG_FIXTURE_JSON) reporters.push(['json', { outputFile: process.env.ORKAS_LOG_FIXTURE_JSON }]);
diff --git a/test/helpers/source-map-literals.cjs b/test/helpers/source-map-literals.cjs
new file mode 100644
index 00000000..7f2fe9b0
--- /dev/null
+++ b/test/helpers/source-map-literals.cjs
@@ -0,0 +1,38 @@
+'use strict';
+
+// convert-source-map scans raw lines, including lines inside tsx's template
+// strings. Vitest must retain the original test failure instead of crashing
+// while decoding that template as a source map. Real malformed maps still fail.
+const convert = require('convert-source-map');
+const original = convert.fromSource;
+const ts = require('typescript');
+
+function fromSource(source) {
+ try { return original(source); }
+ catch (error) {
+ if (!(error instanceof SyntaxError)) throw error;
+ const parsed = ts.createSourceFile('source.js', source, ts.ScriptTarget.Latest, true, ts.ScriptKind.JS);
+ if (parsed.parseDiagnostics.length) throw error;
+ const literals = [];
+ function visit(node) {
+ if (ts.isStringLiteralLike(node) || node.kind === ts.SyntaxKind.TemplateHead
+ || node.kind === ts.SyntaxKind.TemplateMiddle || node.kind === ts.SyntaxKind.TemplateTail) {
+ literals.push([node.getStart(parsed), node.end]);
+ }
+ ts.forEachChild(node, visit);
+ }
+ visit(parsed);
+ let removed = false;
+ const comments = [...source.matchAll(convert.commentRegex)].filter(match => {
+ const marker = match.index + match[0].search(/\/[/*]/);
+ const literal = literals.some(([start, end]) => start <= marker && end > marker);
+ removed ||= literal;
+ return !literal;
+ });
+ if (!removed) throw error;
+ return original(comments.map(match => match[0]).join('\n'));
+ }
+}
+
+exports.fromSource = fromSource;
+exports.install = () => { convert.fromSource = fromSource; };
diff --git a/test/helpers/test-log-analysis.ts b/test/helpers/test-log-analysis.ts
index d66d29b1..2eb05c89 100644
--- a/test/helpers/test-log-analysis.ts
+++ b/test/helpers/test-log-analysis.ts
@@ -22,6 +22,7 @@ import { afterAll, afterEach, beforeEach } from 'vitest';
import { createModelEvalLogCollector } from '../../scripts/model-eval-log-policy.mjs';
import { sanitizeLogTextForUpload } from '../../src/main/util/log-sanitize';
import { TEST_LOG_META_KEY } from '../log-analysis-reporter';
+import { createStderrReceipt } from '../../scripts/test-stderr-receipt.mjs';
export type TestLogLevel = 'warn' | 'error';
export type TestLogPhase = 'case' | 'setup' | 'cleanup';
@@ -90,6 +91,7 @@ interface TapState {
fileExpectations: { spec: ExpectedTestLog; hits: number }[];
setup: Scope;
cleanup?: Scope;
+ stderrReceipt?: ReturnType<typeof createStderrReceipt>;
}
const STATE_KEY = Symbol.for('orkas.test.log-analysis');
@@ -106,7 +108,7 @@ function newScope(phase: TestLogPhase, expectations: Scope['expectations']): Sco
const message = String(line ?? '').replace(RUNTIME_PREFIX_RE, '');
const level: TestLogLevel = finding.severity === 'error' ? 'error' : 'warn';
const match = scope.expectations.find(({ spec }) => spec.level === level
- && (spec.source === undefined || spec.source === source) && spec.message.test(message));
+ && (spec.source === undefined || spec.source === source) && matchesExpectedMessage(spec.message, message));
if (match) { match.hits += 1; scope.expectedCount += 1; }
else if (finding.severity === 'error') scope.unexpectedErrorCount += 1;
const signature = testLogSignature(line);
@@ -123,6 +125,12 @@ function newScope(phase: TestLogPhase, expectations: Scope['expectations']): Sco
return scope;
}
+function matchesExpectedMessage(pattern: RegExp, message: string): boolean {
+ // Stateful declarations still match each record independently.
+ if (pattern.global || pattern.sticky) pattern.lastIndex = 0;
+ return pattern.test(message);
+}
+
function summarize(scope: Scope): TestLogScopeSummary {
const analysis = scope.collector.finish();
const unmetExpectations = scope.expectations.filter(item => item.hits === 0).length;
@@ -169,7 +177,11 @@ function tapState(): TapState | undefined {
export function expectTestLog(spec: ExpectedTestLog): void {
const state = tapState();
if (!state) throw new Error('test log analysis is not installed');
- const entry = { spec, hits: 0 };
+ // Own mutable match state without changing (or writing into a frozen)
+ // caller-owned RegExp. Ordinary stateless declarations retain their path.
+ const owned = spec.message.global || spec.message.sticky
+ ? { ...spec, message: new RegExp(spec.message.source, spec.message.flags) } : spec;
+ const entry = { spec: owned, hits: 0 };
if (state.current.phase === 'case') state.current.expectations.push(entry);
else state.fileExpectations.push(entry);
}
@@ -212,6 +224,9 @@ function installStreamTaps(state: TapState): void {
const original = target.write;
target.write = function tappedWrite(this: typeof target, chunk: unknown, ...rest: unknown[]) {
ingest(stream, chunk, rest[0]);
+ if (stream === 'stderr' && state.stderrReceipt) {
+ state.stderrReceipt.ingest(chunk, typeof rest[0] === 'string' ? rest[0] : undefined);
+ }
return (original as (...args: unknown[]) => boolean).call(this, chunk, ...rest);
} as typeof target.write;
}
@@ -241,6 +256,7 @@ export function installTestLogAnalysis(): void {
installStreamTaps(state);
}
const fileState = state;
+ fileState.stderrReceipt = process.env.ORKAS_TEST_BATCH_REPORT ? createStderrReceipt() : undefined;
beforeEach(() => {
fileState.current = newScope('case', []);
@@ -265,6 +281,7 @@ export function installTestLogAnalysis(): void {
for (const phase of phases) phase.unmetExpectations = 0;
phases[phases.length - 1].unmetExpectations = unmet;
(suite.meta as Record<string, unknown>)[TEST_LOG_META_KEY] = phases;
+ if (fileState.stderrReceipt) (suite.meta as Record<string, unknown>).orkasStderrReceipt = fileState.stderrReceipt.finish();
// Output after the last hook (worker teardown) has no reporting channel.
fileState.setup = newScope('setup', []);
fileState.cleanup = undefined;
diff --git a/test/log-analysis-reporter.ts b/test/log-analysis-reporter.ts
index d57791c6..8811797f 100644
--- a/test/log-analysis-reporter.ts
+++ b/test/log-analysis-reporter.ts
@@ -8,8 +8,10 @@
* signatures, repository-relative test files and test names.
*/
import { relative } from 'node:path';
+import { writeFileSync } from 'node:fs';
import type { Reporter, TestModule } from 'vitest/node';
import type { TestLogScopeSummary, TestLogSignature } from './helpers/test-log-analysis';
+import { combineStderrReceipts } from '../scripts/test-stderr-receipt.mjs';
/** Task-meta key shared with the worker tap; this module must not load Vitest's runtime. */
export const TEST_LOG_META_KEY = 'orkasLogAnalysis';
@@ -117,14 +119,45 @@ export function formatTestLogSummary(summary: TestLogRunSummary): string {
export class TestLogAnalysisReporter implements Reporter {
private root = process.cwd();
+ private inventory: string[] = [];
onInit(vitest: { config: { root: string } }): void {
this.root = vitest.config.root;
}
- onTestRunEnd(modules: readonly TestModule[]): void {
+ onTestRunStart(): void {
+ this.inventory = [];
+ }
+
+ onTestModuleQueued(module: TestModule): void {
+ // onTestRunStart receives the pre-shard inventory in Vitest 4. Queue events
+ // identify the selected shard, including files that fail during collection.
+ this.inventory.push(relative(this.root, module.moduleId).replace(/\\/g, '/'));
+ }
+
+ onTestRunEnd(modules: readonly TestModule[], errors: readonly unknown[] = [], reason = 'passed'): void {
const summary = summarizeTestLogs(modules, this.root);
process.stdout.write(formatTestLogSummary(summary));
if (summary.status === 'FAIL') process.exitCode = 1;
+ if (process.env.ORKAS_TEST_BATCH_REPORT) {
+ const files = modules.map(module => {
+ const counts = { passed: 0, failed: 0, skipped: 0 };
+ for (const test of module.children.allTests()) {
+ const state = test.result().state;
+ if (state in counts) counts[state as keyof typeof counts] += 1;
+ }
+ const diagnostic = module.diagnostic();
+ return { file: relative(this.root, module.moduleId).replace(/\\/g, '/'), state: module.state(), ...counts,
+ durationMs: diagnostic.duration, collectMs: diagnostic.collectDuration, setupMs: diagnostic.setupDuration,
+ prepareMs: diagnostic.prepareDuration, environmentMs: diagnostic.environmentSetupDuration };
+ });
+ // No raw logs, environment values or test payloads in structured reports.
+ // The launcher retains stdout/stderr separately with restricted permissions.
+ writeFileSync(process.env.ORKAS_TEST_BATCH_REPORT, JSON.stringify({ schema: 1, reason,
+ unhandledErrors: errors.length, inventory: this.inventory, files, logs: summary,
+ stderr: combineStderrReceipts(modules.filter(module => module.state() !== 'skipped')
+ .map(module => (module.meta() as Record<string, unknown>).orkasStderrReceipt)),
+ }), { mode: 0o600 });
+ }
}
}
diff --git a/test/main/quality/run-non-model.test.ts b/test/main/quality/run-non-model.test.ts
new file mode 100644
index 00000000..904aba5c
--- /dev/null
+++ b/test/main/quality/run-non-model.test.ts
@@ -0,0 +1,145 @@
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { spawnSync } from 'node:child_process';
+import { pathToFileURL } from 'node:url';
+import { afterEach, describe, expect, it } from 'vitest';
+
+const appRoot = path.resolve(__dirname, '../../..');
+const node = process.env.ORKAS_TEST_NODE || process.execPath;
+const roots: string[] = [];
+afterEach(() => roots.splice(0).forEach(root => fs.rmSync(root, { recursive: true, force: true })));
+const laneOrder = ['typecheck', 'test', 'test:e2e:typecheck', 'test:e2e', 'test:platform-native', 'smoke'];
+
+// Execute the actual CLI in a disposable flat checkout. Only child dispatch is
+// replaced: no business test, Electron window or native host process is run.
+function runEntry(options: { platform?: string; args?: string[]; fail?: string; failure?: string } = {}) {
+ // Node resolves the entry module and child cwd through macOS's /var alias.
+ // Use the fixture's real location for both invocation and exact path oracles.
+ const root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-non-model-')));
+ roots.push(root);
+ fs.mkdirSync(path.join(root, 'scripts'));
+ for (const file of ['run-non-model.mjs', 'test-checkpoint.mjs', 'test-stderr-receipt.mjs', 'test-runtime-env.mjs']) {
+ fs.copyFileSync(path.join(appRoot, 'scripts', file), path.join(root, 'scripts', file));
+ }
+ const callsPath = path.join(root, 'dispatch.jsonl');
+ const preload = path.join(root, 'dispatch.mjs');
+ fs.writeFileSync(preload, `
+ import fs from 'node:fs';
+ import cp from 'node:child_process';
+ import { syncBuiltinESMExports } from 'node:module';
+ Object.defineProperty(process, 'platform', { value: ${JSON.stringify(options.platform || process.platform)} });
+ cp.spawnSync = (command, args, options) => {
+ if (command === 'where.exe') return { status: 1, stdout: '' };
+ fs.appendFileSync(${JSON.stringify(callsPath)}, JSON.stringify({ command, args, cwd: options.cwd,
+ stdio: options.stdio, shell: options.shell, sentinel: options.env.NON_MODEL_SENTINEL }) + '\\n');
+ const script = args.includes('run') ? args[args.indexOf('run') + 1] : 'test';
+ // The parent must persist each completed stage before starting the next.
+ const dispatched = fs.readFileSync(${JSON.stringify(callsPath)}, 'utf8').trim().split('\\n').length;
+ if (dispatched > 1) {
+ if (!fs.existsSync('test-results')) return { status: 91 };
+ const report = JSON.parse(fs.readFileSync('test-results/' + fs.readdirSync('test-results')[0], 'utf8'));
+ if (report.stages.length !== dispatched - 1) return { status: 91 };
+ }
+ if (script === ${JSON.stringify(options.fail || '')}) {
+ if (${JSON.stringify(options.failure)} === 'spawn') return { status: null, signal: null, error: new Error('controlled spawn failure') };
+ if (${JSON.stringify(options.failure)} === 'signal') return { status: null, signal: 'SIGTERM' };
+ return { status: 7, signal: null };
+ }
+ return { status: 0, signal: null };
+ };
+ syncBuiltinESMExports();
+ `);
+ const result = spawnSync(node, ['--import', pathToFileURL(preload).href,
+ path.join(root, 'scripts/run-non-model.mjs'), ...(options.args || [])], {
+ cwd: root, encoding: 'utf8', timeout: 10_000,
+ env: { ...process.env, ELECTRON_RUN_AS_NODE: '1', npm_execpath: preload, NON_MODEL_SENTINEL: 'preserved' },
+ });
+ expect(result.error).toBeUndefined();
+ const calls = fs.existsSync(callsPath)
+ ? fs.readFileSync(callsPath, 'utf8').trim().split('\n').map(line => JSON.parse(line)) : [];
+ const reports = fs.existsSync(path.join(root, 'test-results'))
+ ? fs.readdirSync(path.join(root, 'test-results')).map(file => ({ file, value: JSON.parse(fs.readFileSync(path.join(root, 'test-results', file), 'utf8')) })) : [];
+ return { root, preload, result, calls, reports };
+}
+
+describe('flat repository non-model entry', () => {
+ it.each(['darwin', 'linux', 'win32'])('preserves all six serial lanes and stage receipts on %s', platform => {
+ const { root, preload, result, calls, reports } = runEntry({ platform });
+ expect(result.status, result.stderr).toBe(0);
+ expect(result.stderr).toBe('');
+ expect(calls.map(call => call.args.slice(platform === 'win32' ? 1 : 0))).toEqual([
+ ['run', 'typecheck'], ['test'], ['run', 'test:e2e:typecheck'], ['run', 'test:e2e'],
+ ['run', 'test:platform-native'], ['run', 'smoke'],
+ ]);
+ for (const call of calls) {
+ expect(call).toMatchObject({ command: platform === 'win32' ? node : 'npm', cwd: root, stdio: 'inherit', sentinel: 'preserved' });
+ expect(call.shell).toBeUndefined();
+ if (platform === 'win32') expect(call.args[0]).toBe(preload);
+ }
+ expect(reports).toHaveLength(1);
+ expect(result.stdout).toContain(path.join(root, 'test-results', reports[0].file));
+ expect(reports[0].value.stages.map((stage: any) => stage.script)).toEqual(laneOrder);
+ for (const stage of reports[0].value.stages) {
+ expect(stage).toMatchObject({ exitCode: 0, signal: null, execution: 'completed' });
+ expect(Number.isInteger(stage.durationMs)).toBe(true);
+ expect(stage.durationMs).toBeGreaterThanOrEqual(0);
+ }
+ expect(reports[0].value.logReview).toContain('each owning runner report');
+ });
+
+ it.each([
+ ['--checkpoint-dir', 'checkpoint with spaces', '--batch-count', '4', '--resume'],
+ ['--checkpoint-dir=checkpoint with spaces', '--batch-count=4', '--resume'],
+ ])('forwards checkpoint options to the JS/Python suite owner and E2E only (%j)', (...args) => {
+ const { root, result, calls } = runEntry({ platform: 'linux', args });
+ expect(result.status, result.stderr).toBe(0);
+ const forwarded = ['--', `--checkpoint-dir=${path.join(root, 'checkpoint with spaces')}`, '--batch-count=4', '--resume'];
+ expect(calls.map(call => call.args)).toEqual([
+ ['run', 'typecheck'], ['test', ...forwarded], ['run', 'test:e2e:typecheck'],
+ ['run', 'test:e2e', ...forwarded], ['run', 'test:platform-native'], ['run', 'smoke'],
+ ]);
+ // No separate Python invocation: npm test owns builtin Python and its
+ // checkpoint forwarding (covered by test-checkpoint.test.ts's suite case).
+ expect(calls).toHaveLength(6);
+ });
+
+ it('uses the owning checkpoint default without enabling resume implicitly', () => {
+ const { root, result, calls } = runEntry({ platform: 'linux', args: ['--checkpoint-dir=checkpoint'] });
+ expect(result.status, result.stderr).toBe(0);
+ expect(calls[1].args).toEqual(['test', '--', `--checkpoint-dir=${path.join(root, 'checkpoint')}`, '--batch-count=8']);
+ expect(calls[3].args).toEqual(['run', 'test:e2e', '--', `--checkpoint-dir=${path.join(root, 'checkpoint')}`, '--batch-count=8']);
+ });
+
+ it.each(laneOrder)('retains failure and never starts later lanes after %s fails', fail => {
+ const { result, calls, reports } = runEntry({ fail });
+ expect(result.status).toBe(7);
+ expect(result.stderr).toBe('');
+ const expected = laneOrder.slice(0, laneOrder.indexOf(fail) + 1);
+ expect(calls).toHaveLength(expected.length);
+ expect(reports[0].value.stages.map((stage: any) => stage.script)).toEqual(expected);
+ expect(reports[0].value.stages.at(-1)).toMatchObject({ script: fail, exitCode: 7 });
+ });
+
+ it.each(['spawn', 'signal'])('records %s failure without starting the next native lane', failure => {
+ const { result, calls, reports } = runEntry({ fail: 'test:e2e', failure });
+ expect(result.status).toBe(1);
+ expect(calls).toHaveLength(4);
+ expect(reports[0].value.stages.at(-1)).toMatchObject({ script: 'test:e2e', exitCode: null,
+ execution: failure === 'spawn' ? 'unavailable' : 'interrupted', signal: failure === 'signal' ? 'SIGTERM' : null });
+ if (failure === 'spawn') expect(result.stderr).toContain('controlled spawn failure');
+ else expect(result.stderr).toBe('');
+ });
+
+ it.each([
+ ['--resume'], ['--batch-count=4'], ['--checkpoint-dir'], ['--checkpoint-dir='],
+ ['--checkpoint-dir=checkpoint', '--batch-count=0'], ['--checkpoint-dir=checkpoint', '--batch-count=33'],
+ ['--coverage'], ['--include-ios'], ['--checkpoint-dir=checkpoint', '--watch'],
+ ])('rejects invalid or scope-changing arguments before dispatch (%j)', (...args) => {
+ const { result, calls, reports } = runEntry({ args });
+ expect(result.status).toBe(2);
+ expect(result.stderr).toContain('[non-model]');
+ expect(calls).toEqual([]);
+ expect(reports).toEqual([]);
+ });
+});
diff --git a/test/main/quality/source-map-literals.test.ts b/test/main/quality/source-map-literals.test.ts
new file mode 100644
index 00000000..105ef6ab
--- /dev/null
+++ b/test/main/quality/source-map-literals.test.ts
@@ -0,0 +1,40 @@
+import { createRequire } from 'node:module';
+import fs from 'node:fs';
+import path from 'node:path';
+import { describe, expect, it } from 'vitest';
+
+const require = createRequire(import.meta.url);
+const { fromSource, install } = require('../../helpers/source-map-literals.cjs');
+const map = { version: 3, sources: ['original.ts'], names: [], mappings: 'AAAA' };
+const comment = '//# sourceMappingURL=data:application/json;base64,';
+
+describe('dependency error stack source maps', () => {
+ it('preserves real inline maps and rejects malformed real comments', () => {
+ expect(fromSource(comment + Buffer.from(JSON.stringify(map)).toString('base64')).toObject()).toEqual(map);
+ expect(() => fromSource(comment + 'broken')).toThrow(SyntaxError);
+ });
+
+ it('ignores only string-contained false matches and keeps an earlier real map', () => {
+ const template = 'const text = `\n' + comment + '${value}\n`;';
+ expect(fromSource(template)).toBeNull();
+ expect(fromSource(comment + Buffer.from(JSON.stringify(map)).toString('base64') + '\n' + template).toObject()).toEqual(map);
+ expect(() => fromSource(comment + 'broken\n' + template)).toThrow(SyntaxError);
+ });
+
+ it('uses the repaired parser through Vitest while preserving genuine map errors', async () => {
+ install();
+ const { extractSourcemapFromFile } = await import('@vitest/utils/source-map/node');
+ const template = 'const text = `\n' + comment + '${value}\n`;';
+ // Vitest's external-file fallback returns an empty map for a missing filename.
+ expect(extractSourcemapFromFile(template, '/dependency.js')).toEqual({ map: {} });
+ expect(extractSourcemapFromFile(comment + Buffer.from(JSON.stringify(map)).toString('base64'), '/dependency.js')).toEqual({ map });
+ expect(() => extractSourcemapFromFile(comment + 'broken', '/dependency.js')).toThrow(SyntaxError);
+ });
+
+ it('can report errors originating in both installed tsx register modules', () => {
+ const directory = path.dirname(require.resolve('tsx/package.json'));
+ const files = fs.readdirSync(path.join(directory, 'dist')).filter(file => /^register-.*\.(cjs|mjs)$/.test(file));
+ expect(files.length).toBeGreaterThanOrEqual(2);
+ for (const file of files) expect(() => fromSource(fs.readFileSync(path.join(directory, 'dist', file), 'utf8'))).not.toThrow();
+ });
+});
diff --git a/test/main/quality/test-checkpoint.test.ts b/test/main/quality/test-checkpoint.test.ts
new file mode 100644
index 00000000..ace70a28
--- /dev/null
+++ b/test/main/quality/test-checkpoint.test.ts
@@ -0,0 +1,237 @@
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { spawn, spawnSync } from 'node:child_process';
+import { afterEach, expect, it } from 'vitest';
+import { createStderrReceipt, combineStderrReceipts } from '../../../scripts/test-stderr-receipt.mjs';
+
+const pcRoot = path.resolve(__dirname, '../../..');
+const node = process.env.ORKAS_TEST_NODE || process.execPath;
+const roots: string[] = [];
+afterEach(() => roots.splice(0).forEach(root => fs.rmSync(root, { recursive: true, force: true })));
+
+it('attributes the exact stderr bytes across chunk boundaries and concurrent worker line ordering', () => {
+ const first = createStderrReceipt();
+ first.ingest('progress α\n');
+ const second = createStderrReceipt();
+ second.ingest('recoverable\n');
+ const processOutput = createStderrReceipt();
+ const bytes = Buffer.from('recoverable\nprogress α\n');
+ for (const byte of bytes) processOutput.ingest(Buffer.from([byte]));
+ const combined = combineStderrReceipts([first.finish(), second.finish()]);
+ expect(combined).toEqual(processOutput.finish());
+ expect(combined).toMatchObject({ valid: true, bytes: 24, lines: 2 });
+});
+
+it('rejects missing, duplicated, changed or oversized stderr evidence', () => {
+ const expected = createStderrReceipt();
+ expected.ingest('a\nb\n');
+ const reference = expected.finish();
+ for (const text of ['a\n', 'a\nb\nb\n', 'a\nc\n', 'a\nb']) {
+ const changed = createStderrReceipt();
+ changed.ingest(text);
+ expect(changed.finish()).not.toEqual(reference);
+ }
+ const oversized = createStderrReceipt();
+ oversized.ingest(Buffer.alloc(1024 * 1024 + 1));
+ expect(oversized.finish().valid).toBe(false);
+ expect(combineStderrReceipts([undefined]).valid).toBe(false);
+});
+
+function fixture() {
+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-checkpoint-'));
+ roots.push(root);
+ const write = (file: string, content: string) => {
+ fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true });
+ fs.writeFileSync(path.join(root, file), content);
+ };
+ write('.gitignore', 'node_modules/\ntest-results/\n');
+ write('package.json', '{"type":"module"}');
+ for (const script of ['run-tests.mjs', 'test-runtime-env.mjs', 'check-installed-dependencies.mjs', 'test-checkpoint.mjs', 'test-stderr-receipt.mjs']) {
+ const source = path.join(pcRoot, 'scripts', script);
+ if (fs.existsSync(source)) write(`scripts/${script}`, fs.readFileSync(source, 'utf8'));
+ }
+ write('node_modules/electron/index.js', `module.exports = ${JSON.stringify(node)};`);
+ write('scripts/run-python-tests.mjs', "import fs from 'node:fs'; fs.writeFileSync('test-results/resources-ran', '1');");
+ write('test/a.test.ts', 'a');
+ write('test/data/b.test.ts', 'b');
+ write('test/build/c.test.ts', 'c');
+ // Only the test engine is a controlled dependency. Real launcher, discovery
+ // subprocess, source snapshot, report validation and persisted recovery run.
+ write('node_modules/vitest/vitest.mjs', `
+ import fs from 'node:fs';
+ import path from 'node:path';
+ const args = process.argv.slice(2);
+ const files = fs.readdirSync('test', { recursive: true }).filter(f => f.endsWith('.test.ts')).sort().map(f => 'test/' + f.replaceAll('\\\\', '/'));
+ if (args[0] === 'list') {
+ fs.writeFileSync(args.find(a => a.startsWith('--json=')).slice(7), JSON.stringify(files.map(f => ({file:path.resolve(f)}))));
+ } else {
+ const shard = args.find(a => a.startsWith('--shard='))?.slice(8).split('/').map(Number) || [1,1];
+ const selected = files.filter((_, i) => i % shard[1] === shard[0] - 1);
+ const report = process.env.ORKAS_TEST_BATCH_REPORT;
+ const logFailure = process.env.FIXTURE_LOG_GAP === '1' && selected.includes('test/data/b.test.ts');
+ const failure = (selected.includes('test/data/b.test.ts') && !fs.existsSync('test-results/failed-once')) || selected.some(file => fs.readFileSync(file, 'utf8') === 'FAIL');
+ fs.mkdirSync('test-results', {recursive:true});
+ if (failure) fs.writeFileSync('test-results/failed-once', '1');
+ fs.appendFileSync('test-results/executed.jsonl', JSON.stringify(selected) + '\\n');
+ if (report && !(process.env.FIXTURE_NO_REPORT === '1' && selected.includes('test/data/b.test.ts'))) fs.writeFileSync(report, JSON.stringify({schema:1, reason:failure?'failed':'passed', unhandledErrors:0,
+ stderr:{valid:true,bytes:0,lines:0,digest:'0'.repeat(64)},
+ inventory:selected, files:selected.map(file => ({file,state:failure?'failed':'passed',passed:failure?0:1,failed:failure?1:0,skipped:0,durationMs:20})),
+ logs:{status:logFailure?'FAIL':'WARN',files:selected.length,cases:selected.length,lines:1,unexpected:0,unmetExpectations:0,coverageGaps:logFailure?['missing tap']:[],signatures:[]}
+ }));
+ console.log('engine stdout');
+ if (failure || process.env.FIXTURE_STDERR === '1') console.error('engine stderr retained for inspection');
+ if (process.env.FIXTURE_HOLD === '1' && selected.includes('test/data/b.test.ts') && !fs.existsSync('test-results/hold-ready')) {
+ fs.writeFileSync('test-results/hold-ready', '1');
+ setTimeout(() => {}, 30000);
+ }
+ process.exitCode = failure ? 1 : 0;
+ }
+ `);
+ const git = spawnSync('git', ['init', '-q'], { cwd: root, encoding: 'utf8' });
+ expect(git.status, git.stderr).toBe(0);
+ const run = (resume = false, extraEnv: Record<string, string> = {}, suite = false) => spawnSync(node, [
+ 'scripts/run-tests.mjs', suite ? 'suite' : 'run', '--checkpoint-dir=test-results/checkpoint', '--batch-count=3', ...(resume ? ['--resume'] : []),
+ ], { cwd: root, encoding: 'utf8', timeout: 30_000,
+ env: { ...process.env, ...extraEnv, ORKAS_TEST_BATCH_REPORT: '' } });
+ const executed = () => fs.readFileSync(path.join(root, 'test-results/executed.jsonl'), 'utf8').trim().split('\n').map(line => JSON.parse(line));
+ return { root, write, run, executed };
+}
+
+it('resumes only completed, log-certified batches and keeps the original failure and both log streams', () => {
+ const f = fixture();
+ const first = f.run();
+ expect(first.status, first.stderr).toBe(1);
+ expect(f.executed()).toEqual([['test/a.test.ts'], ['test/data/b.test.ts'], ['test/build/c.test.ts']].sort((a,b) => a[0].localeCompare(b[0])));
+ const second = f.run(true);
+ expect(second.status, second.stderr).toBe(0);
+ expect(f.executed()).toHaveLength(4);
+ expect(f.executed().at(-1)).toEqual(['test/data/b.test.ts']);
+ expect(second.stdout).toContain('reused=2');
+ expect(second.stdout).toContain('WARN');
+ const dir = path.join(f.root, 'test-results/checkpoint');
+ const artifacts = fs.readdirSync(dir, { recursive: true }).map(String);
+ const stderrLogs = artifacts.filter(file => file.endsWith('stderr.log'));
+ expect(stderrLogs).toHaveLength(6); // Two discoveries and four batches.
+ expect(stderrLogs.some(file => fs.readFileSync(path.join(dir,file), 'utf8').includes('engine stderr retained'))).toBe(true);
+ const attempts = artifacts.filter(file => file.endsWith('attempt.json')).map(file => JSON.parse(fs.readFileSync(path.join(dir,file), 'utf8')));
+ expect(attempts.some(item => item.exitCode === 1)).toBe(true);
+});
+
+it('retains the builtin Python suite after a successful checkpoint resume through npm test', () => {
+ const f = fixture();
+ const marker = path.join(f.root, 'test-results/resources-ran');
+ expect(f.run(false, {}, true).status).toBe(1);
+ expect(fs.existsSync(marker)).toBe(false);
+ expect(f.run(true, {}, true).status).toBe(0);
+ expect(fs.existsSync(marker)).toBe(true);
+});
+
+it.each(['--coverage', '--watch', '--reporter=json', '--shard=1/2', '--batch-count=0'])(
+ 'rejects unsupported checkpoint selection %s before dispatch', flag => {
+ const f = fixture();
+ const result = spawnSync(node, ['scripts/run-tests.mjs', 'run', '--checkpoint-dir=test-results/checkpoint', flag], {
+ cwd: f.root, env: process.env, encoding: 'utf8', timeout: 10_000,
+ });
+ expect(result.status).toBe(2);
+ expect(fs.existsSync(path.join(f.root, 'test-results/executed.jsonl'))).toBe(false);
+ },
+);
+
+it.each(['--resume', '--batch-count=8'])('rejects %s without a checkpoint directory before dispatch', flag => {
+ const f = fixture();
+ const result = spawnSync(node, ['scripts/run-tests.mjs', 'run', flag], {
+ cwd: f.root, env: process.env, encoding: 'utf8', timeout: 10_000,
+ });
+ expect(result.status).toBe(2);
+ expect(fs.existsSync(path.join(f.root, 'test-results/executed.jsonl'))).toBe(false);
+});
+
+it('invalidates recovery for new generic directories, source, installed dependency and environment changes', () => {
+ const f = fixture();
+ f.run();
+ expect(f.run(true).status).toBe(0);
+ for (const [file, value] of [
+ ['test/new/data/new.test.ts', 'new'],
+ ['test/a.test.ts', 'changed'],
+ ['node_modules/extra/index.js', 'changed dependency'],
+ ]) {
+ const before = f.executed().length;
+ f.write(file, value);
+ const result = f.run(true);
+ expect(result.status, result.stderr).toBe(0);
+ expect(f.executed().length - before).toBe(3);
+ }
+ const before = f.executed().length;
+ expect(f.run(true, { FIXTURE_CHANGED_ENV: '1' }).status).toBe(0);
+ expect(f.executed().length - before).toBe(3);
+ expect(f.executed().flat()).toContain('test/new/data/new.test.ts');
+});
+
+it('reruns a batch whose log artifact was lost instead of inheriting its passing exit code', () => {
+ const f = fixture();
+ f.run();
+ expect(f.run(true).status).toBe(0);
+ const dir = path.join(f.root, 'test-results/checkpoint');
+ const state = JSON.parse(fs.readFileSync(path.join(dir, 'checkpoint.json'), 'utf8'));
+ fs.unlinkSync(path.join(dir, state.completed['1'].stdout));
+ const before = f.executed().length;
+ expect(f.run(true).status).toBe(0);
+ expect(f.executed().length - before).toBe(1);
+});
+
+it.each(['FIXTURE_LOG_GAP', 'FIXTURE_NO_REPORT'])('never certifies missing log evidence (%s) even when the engine exits zero', key => {
+ const f = fixture();
+ const env = { [key]: '1' };
+ f.run(false, env);
+ const result = f.run(true, env);
+ expect(result.status).toBe(1);
+ expect(result.stdout).toContain('reused=2');
+ expect(f.executed()).toHaveLength(4);
+ expect(result.stdout).toContain('log analysis');
+});
+
+it('fails an otherwise green engine for unattributed process stderr', () => {
+ const f = fixture();
+ f.run(false, { FIXTURE_STDERR: '1' });
+ const result = f.run(true, { FIXTURE_STDERR: '1' });
+ expect(result.status).toBe(1);
+ expect(result.stdout).toContain('reused=0');
+ expect(result.stdout).toContain('unclassified stderr');
+});
+
+it('discovers a newly added failing file after a previously complete run', () => {
+ const f = fixture();
+ f.run();
+ expect(f.run(true).status).toBe(0);
+ f.write('test/new/data/new.test.ts', 'FAIL');
+ expect(f.run(true).status).toBe(1);
+ expect(f.executed().flat()).toContain('test/new/data/new.test.ts');
+});
+
+it.skipIf(process.platform === 'win32')('cancels the active batch, releases the lock and resumes only terminal evidence', async () => {
+ const f = fixture();
+ const child = spawn(node, ['scripts/run-tests.mjs', 'run', '--checkpoint-dir=test-results/checkpoint', '--batch-count=3'], {
+ cwd: f.root, stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, FIXTURE_HOLD: '1', ORKAS_TEST_BATCH_REPORT: '' },
+ });
+ let stderr = '';
+ child.stdout.resume();
+ child.stderr.on('data', data => { stderr += data.toString(); });
+ const closed = new Promise(resolve => child.once('close', (code, signal) => resolve({ code, signal })));
+ try {
+ const ready = path.join(f.root, 'test-results/hold-ready');
+ const deadline = Date.now() + 10_000;
+ while (!fs.existsSync(ready) && Date.now() < deadline) await new Promise(resolve => setTimeout(resolve, 20));
+ expect(fs.existsSync(ready), stderr).toBe(true);
+ const conflicting = f.run(true, { FIXTURE_HOLD: '1' });
+ expect(conflicting.status).toBe(1);
+ expect(conflicting.stderr).toContain('Checkpoint is locked');
+ child.kill('SIGTERM');
+ expect(await closed).toEqual({ code: null, signal: 'SIGTERM' });
+ expect(stderr).toBe('');
+ const resumed = f.run(true, { FIXTURE_HOLD: '1' });
+ expect(resumed.status, resumed.stderr).toBe(0);
+ expect(resumed.stdout).toContain('reused=2');
+ expect(f.executed()).toHaveLength(4);
+ } finally { if (child.exitCode === null && child.signalCode === null) child.kill('SIGTERM'); await closed; }
+});
diff --git a/test/main/quality/test-log-analysis.test.ts b/test/main/quality/test-log-analysis.test.ts
index d7f7cefb..96b8b9d9 100644
--- a/test/main/quality/test-log-analysis.test.ts
+++ b/test/main/quality/test-log-analysis.test.ts
@@ -3,6 +3,7 @@ import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { spawnSync } from 'node:child_process';
+import { createStderrReceipt } from '../../../scripts/test-stderr-receipt.mjs';
// Scenario: a maintainer runs any PC Vitest entry (`npm test`, `npm run
// test:apps`, `run-tests.mjs suite <files>`). Emitted logs must be attributed
@@ -18,14 +19,14 @@ const outDir = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-log-analysis-'));
afterAll(() => fs.rmSync(outDir, { recursive: true, force: true }));
-function runFixtures(files: string[], env: Record<string, string> = {}) {
+function runFixtures(files: string[], env: Record<string, string> = {}, args: string[] = []) {
const json = path.join(outDir, `${files.join('-').replace(/\W+/g, '_')}-${Object.keys(env).join('_')}.json`);
const childEnv: NodeJS.ProcessEnv = Object.fromEntries(Object.entries(process.env)
.filter(([key]) => !key.startsWith('VITEST')));
const result = spawnSync(node, ['scripts/run-tests.mjs', 'suite', '--config', `${fixtureDir}/vitest.config.ts`,
- ...files.map(file => `${fixtureDir}/${file}`)], {
+ ...files.map(file => `${fixtureDir}/${file}`), ...args], {
cwd: pcRoot, encoding: 'utf8', timeout: 90_000,
- env: { ...childEnv, ORKAS_LOG_FIXTURE_JSON: json, NO_COLOR: '1', ...env },
+ env: { ...childEnv, ORKAS_LOG_FIXTURE_JSON: json, ORKAS_LOG_FIXTURE_BATCH_REPORT: `${json}.batch.json`, NO_COLOR: '1', ...env },
});
expect(result.error).toBeUndefined();
// The reporter block starts at its status line and is the last stdout output.
@@ -37,10 +38,32 @@ function runFixtures(files: string[], env: Record<string, string> = {}) {
for (const file of report.testResults) {
for (const item of file.assertionResults) cases.set(item.title, { status: item.status, failure: (item.failureMessages || []).join('\n') });
}
- return { status: result.status, summary: summary.join('\n'), head: summary[0] || '', cases, stderr: result.stderr };
+ const outcomes = [...cases].map(([name, item]) => ({ name, status: item.status }));
+ console.info('Nested test log analysis result', {
+ fixtures: files, exit_code: result.status, analysis: summary[0] || 'missing',
+ cases: outcomes.slice(0, 20), omitted_cases: Math.max(0, outcomes.length - 20),
+ });
+ const batchPath = `${json}.batch.json`;
+ const batch = fs.existsSync(batchPath) ? JSON.parse(fs.readFileSync(batchPath, 'utf8')) : null;
+ return { status: result.status, summary: summary.join('\n'), head: summary[0] || '', cases, stderr: result.stderr, batch };
}
describe('default Vitest log analysis', () => {
+ it('keeps warning and error analysis when checkpoint mode suppresses successful console replay', () => {
+ const expected = runFixtures(['expected.fixture.ts'], {}, ['--silent=passed-only']);
+ expect(expected.status).toBe(0);
+ expect(JSON.parse(expected.stderr.trim())).toEqual({ type: 'progress', source: 'video_edit', op: 'mix', status: 'running', elapsed_sec: 0.5 });
+ const receipt = createStderrReceipt();
+ receipt.ingest(expected.stderr);
+ expect(expected.batch.stderr).toEqual(receipt.finish());
+ expect(expected.batch.inventory).toEqual([`${fixtureDir}/expected.fixture.ts`]);
+ expect(expected.batch.files[0]).toMatchObject({ passed: 4, failed: 0, skipped: 0 });
+ expect(expected.batch.files[0].durationMs).toBeGreaterThanOrEqual(0);
+ expect(expected.head).toMatch(/^\[test-log-analysis\] WARN files=1 cases=4 .* unexpected=0 expected=2 unmet=0 gaps=0$/);
+ const unexpected = runFixtures(['unexpected.fixture.ts'], {}, ['--silent=passed-only']);
+ expect(unexpected.status).toBe(1);
+ expect(unexpected.head).toMatch(/^\[test-log-analysis\] FAIL .* unexpected=4 expected=0 unmet=1 gaps=0$/);
+ }, 120_000);
it('passes declared output but surfaces it, and surfaces an undeclared warning with its owning case', () => {
const run = runFixtures(['expected.fixture.ts']);
expect(run.status).toBe(0);
@@ -86,4 +109,16 @@ describe('default Vitest log analysis', () => {
expect(run.head).toMatch(/^\[test-log-analysis\] FAIL .* gaps=5$/);
expect(run.summary).toContain(`coverage gap: ${fixtureDir}/expected.fixture.ts [setup/cleanup] (no analysis)`);
}, 120_000);
+
+ it('matches global and sticky declarations independently while still rejecting a different error', () => {
+ const run = runFixtures(['stateful.fixture.ts']);
+ expect(run.status).toBe(1); // The independent unexpected-error control must fail.
+ expect(run.cases.get('matches every repeated global declaration')?.status).toBe('passed');
+ expect(run.cases.get('matches every repeated sticky declaration')?.status).toBe('passed');
+ expect(run.cases.get('matches every repeated frozen-global declaration')?.status).toBe('passed');
+ expect(run.cases.get('matches every repeated frozen-sticky declaration')?.status).toBe('passed');
+ expect(run.cases.get('still rejects a different error after repeated declared output')?.status).toBe('failed');
+ expect(run.head).toMatch(/^\[test-log-analysis\] FAIL files=1 cases=5 .* warn=0 error=11 unexpected=1 expected=10 unmet=0 gaps=0$/);
+ expect(run.summary).toContain('fixture different unexpected diagnostic');
+ }, 120_000);
});
diff --git a/test/main/quality/test-runner-checkpoint.test.ts b/test/main/quality/test-runner-checkpoint.test.ts
new file mode 100644
index 00000000..e7da53e9
--- /dev/null
+++ b/test/main/quality/test-runner-checkpoint.test.ts
@@ -0,0 +1,264 @@
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import ts from 'typescript';
+import { spawn, spawnSync } from 'node:child_process';
+import { afterEach, expect, it } from 'vitest';
+
+const pcRoot = path.resolve(__dirname, '../../..');
+const node = process.env.ORKAS_TEST_NODE || process.execPath;
+const python = process.env.ORKAS_TEST_PYTHON!;
+const roots: string[] = [];
+afterEach(() => roots.splice(0).forEach(root => fs.rmSync(root, { recursive: true, force: true })));
+
+function fixture(kind: 'python' | 'e2e') {
+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-runner-resume-'));
+ roots.push(root);
+ const pc = root;
+ const write = (file: string, value: string) => {
+ fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true });
+ fs.writeFileSync(path.join(root, file), value);
+ };
+ write('.gitignore', 'test-results/\nnode_modules/\n__pycache__/\n*.pyc\n.pytest_cache/\n');
+ write('package.json', '{}');
+ for (const name of fs.readdirSync(path.join(pcRoot, 'scripts'))) {
+ if (['test-checkpoint.mjs', 'test-stderr-receipt.mjs', 'test-runtime-env.mjs', 'model-eval-log-policy.mjs',
+ 'run-python-tests.mjs', 'run-e2e.cjs', 'runner-checkpoint.mjs', 'pytest_checkpoint.py', 'checkpoint-reporter.ts'].includes(name)) {
+ write(`scripts/${name}`, fs.readFileSync(path.join(pcRoot, 'scripts', name), 'utf8'));
+ }
+ }
+ write('test/e2e/fixtures/web-app-log-analysis.ts', fs.readFileSync(path.join(pcRoot, 'test/e2e/fixtures/web-app-log-analysis.ts'), 'utf8'));
+ write('node_modules/playwright/package.json', '{}');
+ write('node_modules/playwright/cli.js', `require(${JSON.stringify(path.join(path.dirname(require.resolve('playwright/package.json')), 'cli.js'))});`);
+ write('node_modules/playwright/lib/program.js', `module.exports = require(${JSON.stringify(require.resolve('playwright/lib/program'))});`);
+ write('playwright.config.ts', `export default { testDir: './test/e2e', workers: 1, fullyParallel: false, retries: 0, reporter: 'list' };`);
+ const add = (name: string, fail = false, logs = '') => {
+ if (kind === 'python') write(`tests/test_${name}.py`, `
+from pathlib import Path
+import pytest, sys, logging, warnings
+
+def test_${name}():
+ marker = Path('test-results/${name}')
+ marker.parent.mkdir(exist_ok=True)
+ previous = marker.exists()
+ marker.write_text(str(int(marker.read_text()) + 1) if previous else '1')
+ ${logs || 'pass'}
+ assert ${fail ? 'previous' : 'True'}
+`);
+ else write(`test/e2e/${name}_e2e_fixture.spec.ts`, `
+import { test, expect } from ${JSON.stringify(require.resolve('@playwright/test'))};
+import fs from 'node:fs';
+test('${name}', async () => {
+ const marker = 'test-results/${name}';
+ fs.mkdirSync('test-results', { recursive: true });
+ const previous = fs.existsSync(marker);
+ fs.writeFileSync(marker, String(previous ? Number(fs.readFileSync(marker)) + 1 : 1));
+ ${logs}
+ expect(${fail ? 'previous' : 'true'}).toBe(true);
+});
+`);
+ };
+ add('a'); add('b', true); add('c');
+ expect(spawnSync('git', ['init', '-q'], { cwd: root }).status).toBe(0);
+ const command = (resume = false, extra: string[] = []) => [
+ kind === 'python' ? 'scripts/run-python-tests.mjs' : 'scripts/run-e2e.cjs',
+ ...(kind === 'python' ? ['tests', '-q'] : ['test', '--config', 'playwright.config.ts']),
+ '--checkpoint-dir=test-results/checkpoints', '--batch-count=3', ...(resume ? ['--resume'] : []), ...extra,
+ ];
+ const env = { ...process.env, ORKAS_TEST_PYTHON: python, ORKAS_TEST_BATCH_REPORT: '', NO_COLOR: '1', FORCE_COLOR: '0' };
+ const run = (resume = false, extra: string[] = [], extraEnv = {}) => spawnSync(node, command(resume, extra),
+ { cwd: pc, encoding: 'utf8', timeout: 60_000, env: { ...env, ...extraEnv } });
+ const start = () => spawn(node, command(), { cwd: pc, env, stdio: ['ignore', 'pipe', 'pipe'] });
+ const count = (name: string) => Number(fs.readFileSync(path.join(pc, 'test-results', name), 'utf8'));
+ const stateFile = () => fs.readdirSync(path.join(pc, 'test-results/checkpoints'), { recursive: true })
+ .map(String).find(file => file.endsWith('checkpoint.json'))!;
+ return { root, pc, write, add, run, start, count, stateFile };
+}
+
+it.each(['python', 'e2e'] as const)('%s resumes failed work, preserves evidence and discovers new cases', kind => {
+ const f = fixture(kind);
+ const first = f.run();
+ expect(first.status, first.stdout + first.stderr).toBe(1);
+ expect([f.count('a'), f.count('b'), f.count('c')]).toEqual([1, 1, 1]);
+ const second = f.run(true);
+ expect(second.status, second.stdout + second.stderr).toBe(0);
+ expect(second.stdout).toContain('reused=2');
+ expect([f.count('a'), f.count('b'), f.count('c')]).toEqual([1, 2, 1]);
+ const base = path.join(f.pc, 'test-results/checkpoints', path.dirname(f.stateFile()));
+ const state = JSON.parse(fs.readFileSync(path.join(base, 'checkpoint.json'), 'utf8'));
+ fs.unlinkSync(path.join(base, state.completed['1'].stdout));
+ expect(f.run(true).status).toBe(0);
+ expect(f.count('a') + f.count('b') + f.count('c')).toBe(5);
+ const refreshed = JSON.parse(fs.readFileSync(path.join(base, 'checkpoint.json'), 'utf8'));
+ const entry = refreshed.completed['1'];
+ const report = JSON.parse(fs.readFileSync(path.join(base, entry.report), 'utf8'));
+ fs.appendFileSync(path.join(base, report.artifacts[0].file), 'corrupted');
+ const corrupt = f.run(true);
+ expect(corrupt.status, corrupt.stdout + corrupt.stderr).toBe(0);
+ expect(corrupt.stdout).toContain('reused=2');
+ f.add('new', true);
+ const changed = f.run(true);
+ expect(changed.status, changed.stdout + changed.stderr).toBe(1);
+ expect(f.count('new')).toBe(1);
+ expect(changed.stdout).toContain('reused=0');
+ const attempts = fs.readdirSync(base, { recursive: true }).map(String).filter(file => file.endsWith('attempt.json'));
+ expect(attempts.some(file => JSON.parse(fs.readFileSync(path.join(base, file), 'utf8')).exitCode === 1)).toBe(true);
+}, 120_000);
+
+it.each(['python', 'e2e'] as const)('%s rejects unclassified stderr and surfaces structured warnings', kind => {
+ const f = fixture(kind);
+ f.add('b', false, kind === 'python' ? "sys.stderr.write('unclassified probe\\n')" : "process.stderr.write('unclassified probe\\n');");
+ const invalid = f.run();
+ expect(invalid.status, invalid.stdout + invalid.stderr).toBe(1);
+ expect(invalid.stdout).toContain('log analysis FAIL');
+ f.add('b', false, kind === 'python' ? "logging.error('structured failure')" : "process.stdout.write('[00:00:00.000] [error] [(probe)] structured failure\\n');");
+ expect(f.run(true).status).toBe(1);
+ f.add('b', false, kind === 'python' ? "warnings.warn('probe warning')" : "process.stdout.write('warning: probe warning\\n');");
+ const warned = f.run(true);
+ expect(warned.status, warned.stdout + warned.stderr).toBe(0);
+ expect(warned.stdout).toContain('log analysis WARN');
+}, 120_000);
+
+
+it.each(['python', 'e2e'] as const)('%s preserves skipped cases and whole files with uneven case counts', kind => {
+ const f = fixture(kind);
+ f.add('b');
+ if (kind === 'python') f.write('tests/test_a.py', `
+import pytest
[email protected]('number', range(12))
+def test_many(number): assert number >= 0
[email protected](reason='platform fixture')
+def test_skip(): assert False
+`);
+ else f.write('test/e2e/a_e2e_fixture.spec.ts', `
+import {test, expect} from ${JSON.stringify(require.resolve('@playwright/test'))};
+for (let i = 0; i < 12; i++) test('case ' + i, () => expect(i).toBeGreaterThanOrEqual(0));
+test.skip('platform fixture', () => expect(false).toBe(true));
+`);
+ const initial = f.run();
+ expect(initial.status, initial.stdout + initial.stderr).toBe(0);
+ const resumed = f.run(true);
+ expect(resumed.status, resumed.stdout + resumed.stderr).toBe(0);
+ expect(resumed.stdout).toContain('reused=3');
+ const base = path.join(f.pc, 'test-results/checkpoints', path.dirname(f.stateFile()));
+ const state = JSON.parse(fs.readFileSync(path.join(base, 'checkpoint.json'), 'utf8'));
+ const results = Object.values(state.completed).flatMap((entry: any) => JSON.parse(fs.readFileSync(path.join(base, entry.report), 'utf8')).files);
+ expect(results).toHaveLength(15);
+ expect(results.filter((file: any) => file.skipped === 1)).toHaveLength(1);
+}, 120_000);
+
+it('invalidates Python results when an external import dependency changes', () => {
+ const f = fixture('python');
+ const library = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-external-python-'));
+ roots.push(library);
+ const source = path.join(library, 'checkpoint_dependency.py');
+ fs.writeFileSync(source, 'value = 1\n');
+ f.add('b', false, 'import checkpoint_dependency; assert checkpoint_dependency.value == 1');
+ const env = { PYTHONPATH: library };
+ const first = f.run(false, [], env);
+ expect(first.status, first.stdout + first.stderr).toBe(0);
+ expect(f.run(true, [], env).stdout).toContain('reused=3');
+ fs.writeFileSync(source, 'value = 2\n');
+ const changed = f.run(true, [], env);
+ expect(changed.status, changed.stdout + changed.stderr).toBe(1);
+ expect(changed.stdout).toContain('reused=0');
+}, 120_000);
+
+it.skipIf(process.platform === 'win32').each(['python', 'e2e'] as const)(
+ '%s cancellation releases its lock and only resumes terminal batches', async kind => {
+ const f = fixture(kind);
+ f.add('b', false, kind === 'python'
+ ? "if not Path('test-results/hold-ready').exists(): Path('test-results/hold-ready').write_text('1'); __import__('time').sleep(30)"
+ : "if (!fs.existsSync('test-results/hold-ready')) { fs.writeFileSync('test-results/hold-ready', '1'); await new Promise(resolve => setTimeout(resolve, 30000)); }");
+ const child = f.start();
+ let stderr = '';
+ child.stdout.resume();
+ child.stderr.on('data', data => { stderr += data.toString(); });
+ const closed = new Promise(resolve => child.once('close', (code, signal) => resolve({ code, signal })));
+ try {
+ const deadline = Date.now() + 30_000;
+ while (!fs.existsSync(path.join(f.pc, 'test-results/hold-ready')) && Date.now() < deadline) {
+ await new Promise(resolve => setTimeout(resolve, 50));
+ }
+ expect(fs.existsSync(path.join(f.pc, 'test-results/hold-ready')), stderr).toBe(true);
+ const conflict = f.run(true);
+ expect(conflict.status).toBe(1);
+ expect(conflict.stderr).toContain('Checkpoint is locked');
+ child.kill('SIGTERM');
+ expect(await closed).toEqual({ code: null, signal: 'SIGTERM' });
+ const resumed = f.run(true);
+ expect(resumed.status, resumed.stdout + resumed.stderr).toBe(0);
+ expect(resumed.stdout).toContain('reused=1');
+ expect([f.count('a'), f.count('b'), f.count('c')]).toEqual([1, 2, 1]);
+ } finally { if (child.exitCode === null && child.signalCode === null) child.kill('SIGTERM'); await closed; }
+ }, 120_000,
+);
+
+
+it('requires every Electron log attachment and retains independently verified recovery evidence', () => {
+ const f = fixture('e2e');
+ f.add('b', false, `await test.info().attach('electron-log-required', { body: Buffer.from('[]') });`);
+ const missing = f.run();
+ expect(missing.status, missing.stdout + missing.stderr).toBe(1);
+ f.add('b', false, `
+ await test.info().attach('electron-log-required', { body: Buffer.from(JSON.stringify([{ start: 1, end: 1, diagnosticIndex: 0 }])) });
+ const log = test.info().outputPath('electron.log');
+ fs.writeFileSync(log, '[renderer:pageerror] injected preview crash\\n');
+ await test.info().attach('electron-logs', { path: log });
+ `);
+ const recovered = f.run(true);
+ expect(recovered.status, recovered.stdout + recovered.stderr).toBe(0);
+ expect(recovered.stdout).toContain('log analysis WARN');
+ const resumed = f.run(true);
+ expect(resumed.stdout).toContain('reused=3');
+}, 120_000);
+
+
+// Run the actual dispose owner without importing Playwright or starting an app.
+// A missing receipt must reject certification and still release every resource.
+it.each(['electron-log-required', 'electron-logs'])(
+ 'checkpoint %s attachment failures preserve cleanup and remain failures', async failedAttachment => {
+ const file = ts.createSourceFile('orkas.ts', fs.readFileSync(
+ path.join(pcRoot, 'test/e2e/fixtures/orkas.ts'), 'utf8'), ts.ScriptTarget.Latest, true);
+ const owner = file.statements.find((statement): statement is ts.ClassDeclaration =>
+ ts.isClassDeclaration(statement) && statement.name?.text === 'OrkasTestApp');
+ if (!owner) throw new Error('Missing Electron fixture owner');
+ const methods = owner.members.filter(member => member.name
+ && ['dispose', 'recoveredDiagnosticRanges'].includes(member.name.getText(file)));
+ const code = ts.transpileModule('class DisposeOwner {' + methods.map(method => method.getText(file)).join('\n') + '}', {
+ compilerOptions: { target: ts.ScriptTarget.ES2022 },
+ }).outputText;
+ const Owner = new Function('process', 'expect', 'path', 'mkdirSync', 'writeFileSync', 'rmSync',
+ 'unlinkSync', 'collectLogFiles', 'randomUUID', code + '; return DisposeOwner;')(
+ { env: { ORKAS_TEST_BATCH_REPORT: 'fixture-report' } }, expect, path, fs.mkdirSync, fs.writeFileSync,
+ fs.rmSync, fs.unlinkSync, () => '', () => 'snapshot');
+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-checkpoint-dispose-'));
+ roots.push(root);
+ const appRoot = path.join(root, 'app');
+ fs.mkdirSync(appRoot);
+ fs.writeFileSync(path.join(appRoot, 'owned-file'), 'temporary fixture data');
+ const released: string[] = [];
+ const failure = new Error('injected checkpoint attachment failure');
+ const fixture = Object.assign(new Owner(), {
+ root: appRoot, page: null, apiResponses: [], expectedApiFailures: [], cliStub: false,
+ rendererPageErrors: [], recoveredRendererErrors: new Set(), diagnostics: ['[fixture-cleanup] completed'], tracePaths: [],
+ closeCurrentApp: async () => { released.push('app'); },
+ closeStubServer: async () => { released.push('stub'); },
+ testInfo: {
+ status: 'passed', expectedStatus: 'passed', outputDir: path.join(root, 'output'),
+ outputPath: (name: string) => path.join(root, 'output', name),
+ attach: async (name: string) => {
+ expect(released).toEqual(['app', 'stub']);
+ if (name === failedAttachment) throw failure;
+ },
+ },
+ });
+ if (failedAttachment === 'electron-log-required') await expect(fixture.dispose()).rejects.toBe(failure);
+ else await expect(fixture.dispose()).rejects.toMatchObject({
+ message: 'Checkpoint Electron log capture failed', cause: failure,
+ });
+ expect(released).toEqual(['app', 'stub']);
+ expect(fs.existsSync(appRoot)).toBe(false);
+ expect(fs.existsSync(path.join(root, 'output', 'checkpoint-electron-snapshot.log'))).toBe(true);
+ },
+);
diff --git a/test/main/util/test-runtime-env.test.ts b/test/main/util/test-runtime-env.test.ts
index ec50f84d..d4a6d2b9 100644
--- a/test/main/util/test-runtime-env.test.ts
+++ b/test/main/util/test-runtime-env.test.ts
@@ -1,6 +1,8 @@
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
+import { spawnSync } from 'node:child_process';
+import { pathToFileURL } from 'node:url';
import { afterEach, describe, expect, it } from 'vitest';
@@ -21,6 +23,25 @@ function temporaryRoot(): string {
}
describe('test runtime environment', () => {
+ it.each(['win32', 'darwin', 'linux'])('resolves the flat checkout default Python instead of its parent for %s', platform => {
+ const parent = fs.realpathSync(temporaryRoot());
+ const root = path.join(parent, 'app');
+ const suffix = platform === 'win32' ? ['venv', 'Scripts', 'python.exe'] : ['venv', 'bin', 'python'];
+ for (const directory of [parent, root]) {
+ const python = path.join(directory, ...suffix);
+ fs.mkdirSync(path.dirname(python), { recursive: true });
+ fs.writeFileSync(python, '');
+ }
+ const helper = path.join(root, 'scripts', 'test-runtime-env.mjs');
+ fs.mkdirSync(path.dirname(helper), { recursive: true });
+ fs.copyFileSync(path.resolve(__dirname, '../../../scripts/test-runtime-env.mjs'), helper);
+ const result = spawnSync(process.env.ORKAS_TEST_NODE || process.execPath,
+ ['--input-type=module', '-e', 'const { withTestPython } = await import(process.argv[1]); console.log(JSON.stringify(withTestPython({}, undefined, process.argv[2])));', pathToFileURL(helper).href, platform],
+ { encoding: 'utf8', timeout: 10_000 });
+ expect(result.status, result.stderr).toBe(0);
+ expect(JSON.parse(result.stdout)).toEqual({ ORKAS_TEST_PYTHON: path.join(root, ...suffix), ORKAS_EVAL_PYTHON: path.join(root, ...suffix) });
+ });
+
it.each(['win32', 'darwin', 'linux'])('uses the repository virtual environment for %s fixture subprocesses', platform => {
const root = temporaryRoot();
const python = platform === 'win32'
diff --git a/vitest.config.ts b/vitest.config.ts
index 33546a5c..a194630b 100644
--- a/vitest.config.ts
+++ b/vitest.config.ts
@@ -1,7 +1,10 @@
import { defineConfig } from 'vitest/config';
import { cpus } from 'node:os';
+import { createRequire } from 'node:module';
import { TestLogAnalysisReporter } from './test/log-analysis-reporter';
+createRequire(import.meta.url)('./test/helpers/source-map-literals.cjs').install();
+
const logicalCpus = cpus().length || 1;
// Test files exercise real Electron workers, shell helpers, Git, SQLite,
// FFmpeg, and whisper.cpp. Four concurrent forks can exhaust the desktop