Orkas is an open-source (MIT), local-first AI team for your desktop: a Commander turns your goal into to-dos and coordinates 16 built-in specialist agents plus Claude Code, Codex, OpenCode, OpenClaw and Hermes, with projects, scheduled automations and connectors. Free app for macOS & Windows; built-in models or your own keys.
JavaScript agent-collaborationagent-harnessagent-orchestrationagent-teamagent-workflowai-assistantai-automationai-employeeai-productivityai-teamai-workflowautonomous-agentsclaude-code-guiclaude-coworkdesktop-agentlocal-firstmulti-agentmulti-agent-systemself-evolving-agentstask-automation
Resume deterministic test lanes with validated evidence
wytfafb721703a7
diff --git a/docs/test/test-infrastructure-regression.md b/docs/test/test-infrastructure-regression.mdindex 37f9a0fe..3297ed46 100644--- a/docs/test/test-infrastructure-regression.md+++ b/docs/test/test-infrastructure-regression.md@@ -19,3 +19,80 @@ not evidence of coverage.| Staged language completeness | Locale-resource or language-registry edits run the existing Main/Renderer i18n cases for the full supported registry; failed cases block the commit. Unstaged validation inputs and inputs changed during checks are rejected, preserving unrelated work | `test/main/util/staged-i18n-gate.test.ts` exercises real Git/hook execution, incomplete translations, placeholder mismatch, partial staging, unavailable runtime and changed inputs; `test/main/i18n.test.ts` and `test/renderer/i18n.test.ts` own language semantics | Not applicable; local commit workflow | Not applicable || Windows CLI fixture runtime | A desktop test process that can see `git.exe` but resolves `bash` to the System32 WSL placeholder must still execute Git for Windows shell contracts; command fixtures with `.cmd` launchers must be invoked through their real Windows entry point | `test/main/util/test-runtime-env.test.ts`, the separately authorized model canary, and the separately authorized model canary | Not applicable | Not applicable |++## Full non-model runs and recovery++From the application root, run the existing typecheck, unit/resource, E2E,+platform-native and smoke lanes in sequence:++```bash+npm run test:non-model -- --checkpoint-dir=/tmp/orkas-non-model+npm run test:non-model -- --checkpoint-dir=/tmp/orkas-non-model --resume+```++Native, Office and Electron tests share desktop processes. Do not run another+native or GUI suite concurrently on the same host. The unified entry preserves+its existing lanes and stops at a failed lane. No live model evaluation is added.+Checkpoints cover JS, builtin Python resources and desktop E2E; typechecks,+platform-native and smoke lanes execute afresh. Each family uses a separate+checkpoint namespace so its evidence cannot overwrite another family's progress.+The JS runner uses native Vitest discovery and eight sequential shards by default,+with the existing per-file isolation and worker limit. Use `--batch-count=N`+(1–32) to change granularity; it does not increase concurrency. A direct JS run+can also use `npm run test:js -- --checkpoint-dir=/tmp/orkas-unit-run`.+For JS, file filters and `--maxWorkers=N` are supported; coverage, watch mode,+custom configs/reporters and test-name filters use the ordinary runner.++Python and E2E also use sequential file batches, preserving parameterized cases,+module fixtures and serial tests within a file. Each batch creates a fresh process;+a partial file restarts from its beginning. Every invocation rediscovers actual+case IDs, and their final union must exactly match discovery. Python fingerprints+include its interpreter, standard library and external import dependencies.+Python accepts test paths and `-q`/`-v`; E2E accepts file filters with the default+Playwright config. Debugging, custom configs and other flags use the ordinary entry.+For example, from the application root:++```bash+node scripts/run-python-tests.mjs resources/test -q --checkpoint-dir=test-results/checkpoint --resume+npm run test:e2e -- app_e2e_smoke.spec.ts --checkpoint-dir=test-results/checkpoint --resume+```++Python captures per-case setup/call/teardown output, logging and warnings. E2E+retains worker output, Electron log snapshots and attachments in unique batch+output directories, with structured JSON reports instead of the ordinary HTML+report. The existing log policy gates unexpected errors and unclassified stderr;+warnings remain reviewable. Retry attempts do not qualify an E2E batch for reuse.+Missing or modified reports, captured events, logs or attachments force a rerun.+Tests that consume their own captured output, or dependencies that never send logs+to these captured streams, still need their owning assertions/log review; a+checkpoint does not create evidence for an unobserved stream.++`--resume` reuses only completed batches with intact stdout/stderr and structured+log-analysis artifacts. Failed, interrupted or uncertified batches run again;+warnings remain visible, and skips remain distinct from passes. Each invocation+rediscovers files and verifies that the final batch union exactly covers that+inventory. Source metadata, installed dependency metadata, runtime, environment,+selection or inventory changes invalidate reuse. Private configuration contents+are never read for the fingerprint. Checkpoints expire after 12 hours and are+local to the same host. External service state, native application updates or+OS permission changes are not certified by this fingerprint: start a fresh run+without `--resume` after such changes.++Each invocation keeps a separate report directory, including original failed+attempts, file setup/collection/test durations and log summaries. Successful-case+console replay is suppressed in checkpoint mode, but every record still passes+through the existing worker log analyzer. Exact-line aggregate digests reconcile+raw process stderr with worker-analyzed output; missing or additional bytes fail+the batch without storing raw text in the structured summary. Inspect the reported+warning signatures and raw failure logs; a timing+report alone is not verification evidence. `summary.json` separates newly executed+and reused results, wall time, input-scan overhead and avoided batch execution+time. Input scanning adds overhead, so use checkpoints for long/full runs rather+than tiny local selections. The root entry also records per-lane durations under+`test-results/non-model-*.json`.++Only one invocation may write a checkpoint directory. Cancellation releases its+lock; after a forced kill, confirm no associated run remains before removing+`run.lock`. Old attempt directories are deliberately retained for diagnosis;+remove the whole checkpoint directory when that evidence is no longer needed.+diff --git a/package.json b/package.jsonindex e0445c77..5c58baa4 100644--- a/package.json+++ b/package.json@@ -22,7 +22,7 @@"typecheck:main": "tsc --noEmit","typecheck:core-agent": "tsc --noEmit -p src/core-agent/tsconfig.json","test": "node scripts/run-tests.mjs suite",- "test:non-model": "npm run typecheck && npm test && npm run test:e2e:typecheck && npm run test:e2e && npm run test:platform-native && npm run smoke",+ "test:non-model": "node scripts/run-non-model.mjs","test:js": "node scripts/run-tests.mjs run","test:e2e": "node scripts/run-e2e.cjs test --config playwright.config.ts","test:e2e:debug": "node scripts/run-e2e.cjs test --config playwright.config.ts --debug",diff --git a/scripts/checkpoint-reporter.ts b/scripts/checkpoint-reporter.tsnew file mode 100644index 00000000..e2137267--- /dev/null+++ b/scripts/checkpoint-reporter.ts@@ -0,0 +1,69 @@+import fs from 'node:fs';+import path from 'node:path';+import { createHash } from 'node:crypto';+import type { Reporter, FullConfig, Suite, TestCase, TestResult, FullResult } from '@playwright/test/reporter';+import { analyzeWebAppLogs } from '../test/e2e/fixtures/web-app-log-analysis';++/** Structured outcomes, not console text, determine completion. */+export default class CheckpointReporter implements Reporter {+ private report = process.env.ORKAS_TEST_BATCH_REPORT!;+ private events = this.report.replace(/\.json$/, '.events.jsonl');+ private inventory: string[] = [];+ private files: Array<{ file: string; state: string; passed: number; failed: number; skipped: number; durationMs: number }> = [];+ private testFiles: string[] = [];+ private errors = 0;+ private appAnalyses: Array<ReturnType<typeof analyzeWebAppLogs> & { case: string }> = [];+ private artifacts: string[] = [];+ private outputDir = '';++ onBegin(config: FullConfig, suite: Suite) {+ this.outputDir = config.projects[0].outputDir;+ this.inventory = suite.allTests().map(test => test.id);+ this.testFiles = [...new Set(suite.allTests().map(test => test.location.file))].sort();+ fs.writeFileSync(this.events, '', { mode: 0o600 });+ }+ onStdOut(chunk: string | Buffer, test?: TestCase) { this.event('stdout', chunk, test); }+ onStdErr(chunk: string | Buffer, test?: TestCase) { this.event('stderr', chunk, test); }+ private event(stream: string, chunk: string | Buffer, test?: TestCase) {+ fs.appendFileSync(this.events, JSON.stringify({ case: test?.id || 'setup', phase: 'worker', stream, text: chunk.toString() }) + '\n');+ }+ onError() { this.errors++; }+ onTestEnd(test: TestCase, result: TestResult) {+ const failed = result.status !== test.expectedStatus && result.status !== 'skipped';+ // A retry is preserved as a finding and never certified for reuse.+ const state = failed || result.retry > 0 ? 'failed' : result.status === 'skipped' ? 'skipped' : 'passed';+ this.files = this.files.filter(file => file.file !== test.id);+ this.files.push({ file: test.id, state, passed: Number(state === 'passed'), failed: Number(state === 'failed'),+ skipped: Number(state === 'skipped'), durationMs: result.duration });+ const requiredLogs = result.attachments.filter(item => item.name === 'electron-log-required').length;+ if (requiredLogs !== result.attachments.filter(item => item.name === 'electron-logs').length) this.errors++;+ let recovered: Parameters<typeof analyzeWebAppLogs>[1] = [];+ for (const [index, original] of result.attachments.entries()) {+ const attachment = { ...original };+ if (!attachment.path && attachment.body) {+ const id = createHash('sha256').update(`${test.id}/${result.retry}/${index}`).digest('hex');+ attachment.path = path.join(this.outputDir, 'checkpoint-attachments', id);+ fs.mkdirSync(path.dirname(attachment.path), { recursive: true });+ fs.writeFileSync(attachment.path, attachment.body, { mode: 0o600 });+ }+ if (!attachment.path) continue;+ this.artifacts.push(attachment.path);+ if (attachment.name === 'electron-log-required') {+ try { recovered = JSON.parse(fs.readFileSync(attachment.path, 'utf8')); }+ catch { this.errors++; }+ }+ if (attachment.name === 'electron-logs') {+ try {+ const analysis = analyzeWebAppLogs(fs.readFileSync(attachment.path, 'utf8'), recovered);+ this.appAnalyses.push({ ...analysis, case: test.id });+ recovered = [];+ } catch { this.errors++; }+ }+ }+ }+ onEnd(result: FullResult) {+ fs.writeFileSync(this.report, JSON.stringify({ schema: 1, reason: result.status === 'passed' ? 'passed' : 'failed',+ batchUnits: this.testFiles.length, testFiles: this.testFiles, unhandledErrors: this.errors, inventory: this.inventory, files: this.files, appAnalyses: this.appAnalyses,+ artifactPaths: this.artifacts, outputDir: this.outputDir }), { mode: 0o600 });+ }+}diff --git a/scripts/pytest_checkpoint.py b/scripts/pytest_checkpoint.pynew file mode 100644index 00000000..cc64d69c--- /dev/null+++ b/scripts/pytest_checkpoint.py@@ -0,0 +1,121 @@+"""Checkpoint protocol for pytest; keeps module/session fixtures within a shard."""+import json+import logging+import os+import signal+from pathlib import Path++import pytest+++def pytest_addoption(parser):+ parser.addoption('--orkas-checkpoint-shard', default='1/1')+++class Checkpoint:+ def __init__(self, config):+ self.config = config+ self.path = Path(os.environ['ORKAS_TEST_BATCH_REPORT'])+ self.events = self.path.with_suffix('.events.jsonl')+ self.events.write_text('', encoding='utf-8')+ self.inventory = []+ self.results = {}+ self.errors = 0+ self.current = 'collection'+ self.phase = 'collection'+ self.signal_handlers = {}+ for name in ('SIGTERM', 'SIGHUP'):+ value = getattr(signal, name, None)+ if value is not None:+ self.signal_handlers[value] = signal.getsignal(value)+ signal.signal(value, self.interrupt)+ owner = self++ class Handler(logging.Handler):+ def emit(self, record):+ level = 'error' if record.levelno >= 40 else 'warn' if record.levelno >= 30 else 'info'+ owner.event('stdout', f'[00:00:00.000] [{level}] [(python)] {record.getMessage()}\n')+ self.handler = Handler()+ logging.getLogger().addHandler(self.handler)++ @staticmethod+ def interrupt(signum, frame):+ # Let pytest unwind fixtures and stop its children before the parent exits.+ raise KeyboardInterrupt++ def event(self, stream, text):+ with self.events.open('a', encoding='utf-8') as output:+ output.write(json.dumps({'case': self.current, 'phase': self.phase, 'stream': stream, 'text': text}) + '\n')++ @pytest.hookimpl(trylast=True)+ def pytest_collection_modifyitems(self, session, config, items):+ part, total = map(int, config.getoption('--orkas-checkpoint-shard').split('/'))+ files = sorted({str(item.path) for item in items})+ selected_files = set(files[part - 1::total])+ kept = [item for item in items if str(item.path) in selected_files]+ removed = [item for item in items if str(item.path) not in selected_files]+ if removed:+ config.hook.pytest_deselected(items=removed)+ items[:] = kept+ self.inventory = [item.nodeid for item in items]+ self.batch_units = len(files)++ def pytest_collectreport(self, report):+ if report.failed:+ self.errors += 1+ for name, text in report.sections:+ self.event('stderr' if name == 'Captured stderr' else 'stdout', text)++ def pytest_runtest_logstart(self, nodeid, location):+ self.current = nodeid++ def pytest_runtest_setup(self, item):+ self.phase = 'setup'++ def pytest_runtest_call(self, item):+ self.phase = 'call'++ def pytest_runtest_teardown(self, item):+ self.phase = 'teardown'++ def pytest_runtest_logreport(self, report):+ result = self.results.setdefault(report.nodeid, {'phases': {}, 'durationMs': 0})+ result['phases'][report.when] = report.outcome+ result['durationMs'] += round(report.duration * 1000)+ for name, text in report.sections:+ # pytest includes previous phases in later reports. Read each once.+ if name == f'Captured stdout {report.when}':+ self.event('stdout', text)+ elif name == f'Captured stderr {report.when}':+ self.event('stderr', text)++ def pytest_warning_recorded(self, warning_message, when, nodeid, location):+ previous = self.current+ self.current = nodeid or 'collection'+ self.event('stdout', f'[00:00:00.000] [warn] [(python-warning)] {warning_message.message}\n')+ self.current = previous++ @pytest.hookimpl(trylast=True)+ def pytest_sessionfinish(self, session, exitstatus):+ files = []+ for nodeid, result in self.results.items():+ phases = result['phases']+ skipped = 'skipped' in phases.values()+ passed = phases.get('setup') == 'passed' and phases.get('call') == 'passed' and phases.get('teardown') == 'passed'+ failed = 'failed' in phases.values() or phases.get('teardown') != 'passed'+ state = 'failed' if failed else 'passed' if passed else 'skipped' if skipped else 'failed'+ files.append({'file': nodeid, 'state': state, 'passed': int(state == 'passed'),+ 'failed': int(state == 'failed'), 'skipped': int(state == 'skipped'),+ 'durationMs': result['durationMs']})+ self.path.write_text(json.dumps({'schema': 1, 'reason': 'passed' if exitstatus == 0 else 'failed',+ 'unhandledErrors': self.errors, 'inventory': self.inventory,+ 'files': files, 'batchUnits': getattr(self, 'batch_units', 0)}), encoding='utf-8')++ def pytest_unconfigure(self, config):+ logging.getLogger().removeHandler(self.handler)+ for value, handler in self.signal_handlers.items():+ signal.signal(value, handler)+++def pytest_configure(config):+ config.pluginmanager.register(Checkpoint(config), 'orkas-checkpoint-protocol')diff --git a/scripts/run-e2e.cjs b/scripts/run-e2e.cjsindex 675cda8e..5795eb5c 100644--- a/scripts/run-e2e.cjs+++ b/scripts/run-e2e.cjs@@ -20,4 +20,11 @@ if (process.env.NO_COLOR !== undefined) {delete process.env.NO_COLOR;}-require('playwright/lib/program').program.parse(process.argv);+if (args.some(arg => arg === '--resume' || arg.startsWith('--checkpoint-dir') || arg.startsWith('--batch-count'))) {+ import('./runner-checkpoint.mjs').then(({ runRunnerCheckpoint }) =>+ runRunnerCheckpoint('e2e', args, { cwd: process.cwd(), env: process.env }))+ .then(code => { process.exitCode = code; })+ .catch(error => { console.error(`[run-e2e] ${error.message}`); process.exitCode = 1; });+} else {+ require('playwright/lib/program').program.parse(process.argv);+}diff --git a/scripts/run-non-model.mjs b/scripts/run-non-model.mjsnew file mode 100644index 00000000..d00f54ef--- /dev/null+++ b/scripts/run-non-model.mjs@@ -0,0 +1,72 @@+#!/usr/bin/env node++import { existsSync, mkdirSync, writeFileSync } from 'node:fs';+import { dirname, resolve } from 'node:path';+import { fileURLToPath } from 'node:url';+import { spawnSync } from 'node:child_process';+import { randomUUID } from 'node:crypto';+import { checkpointOptions } from './test-checkpoint.mjs';+import { withWindowsGitOnPath } from './test-runtime-env.mjs';++const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..');+let checkpointArgs;+try {+ const checkpoint = checkpointOptions(process.argv.slice(2));+ if (checkpoint.args.length) throw new Error('Only --checkpoint-dir, --batch-count and --resume are supported; run coverage independently');+ checkpointArgs = checkpoint.directory ? [+ `--checkpoint-dir=${resolve(checkpoint.directory)}`,+ `--batch-count=${checkpoint.batches}`,+ ...(checkpoint.resume ? ['--resume'] : []),+ ] : [];+} catch (error) {+ console.error(`[non-model] ${error.message}`);+ process.exit(2);+}++const testEnvironment = withWindowsGitOnPath(process.env);+function npmInvocation() {+ if (process.platform !== 'win32') return { command: 'npm', prefix: [] };+ const candidates = [testEnvironment.npm_execpath];+ const where = spawnSync('where.exe', ['npm.cmd'], { encoding: 'utf8', env: testEnvironment });+ if (where.status === 0) {+ for (const launcher of where.stdout.split(/\r?\n/).map(line => line.trim()).filter(Boolean)) {+ candidates.push(resolve(dirname(launcher), 'node_modules', 'npm', 'bin', 'npm-cli.js'));+ }+ }+ const cli = candidates.find(candidate => candidate && existsSync(candidate));+ if (!cli) throw new Error('npm CLI not found; install npm or invoke this runner from npm');+ // Node cannot directly spawn a .cmd launcher on Windows. Keep arguments out+ // of a command shell and execute the same installed npm JavaScript entry.+ return { command: process.execPath, prefix: [cli] };+}++let npm;+try { npm = npmInvocation(); }+catch (error) {+ console.error(`[non-model] ${error.message}`);+ process.exit(2);+}++const reportPath = resolve(repoRoot, 'test-results', `non-model-${randomUUID()}.json`);+const stages = [];+console.log(`[non-model] stage timing report: ${reportPath}`);++// Preserve the flat repository's existing six-stage shell chain. In particular,+// npm test already owns both the JS and builtin Python suites. Their launchers+// and the E2E launcher own checkpoint discovery, reuse and log certification.+// Native Office/Electron lanes share host resources and must remain serial.+for (const script of ['typecheck', 'test', 'test:e2e:typecheck', 'test:e2e', 'test:platform-native', 'smoke']) {+ const args = [...npm.prefix, ...(script === 'test' ? ['test'] : ['run', script]),+ ...(['test', 'test:e2e'].includes(script) && checkpointArgs.length ? ['--', ...checkpointArgs] : [])];+ const started = performance.now();+ const result = spawnSync(npm.command, args, { cwd: repoRoot, stdio: 'inherit', env: testEnvironment });+ const durationMs = Math.round(performance.now() - started);+ const execution = result.error ? 'unavailable' : result.signal ? 'interrupted' : 'completed';+ stages.push({ script, command: npm.command, args, durationMs, exitCode: result.status, signal: result.signal, execution });+ mkdirSync(dirname(reportPath), { recursive: true });+ writeFileSync(reportPath, `${JSON.stringify({ schema: 1, stages,+ logReview: 'Inspect each owning runner report and all emitted logs' }, null, 2)}\n`);+ console.log(`[non-model] ${script} duration=${durationMs}ms exit=${result.status ?? 'unavailable'} execution=${execution}`);+ if (result.error) console.error(`[non-model] failed to start ${script}: ${result.error.message}`);+ if (result.error || result.signal || result.status !== 0) process.exit(result.status || 1);+}diff --git a/scripts/run-python-tests.mjs b/scripts/run-python-tests.mjsindex 9a6313b7..c2d9081a 100644--- a/scripts/run-python-tests.mjs+++ b/scripts/run-python-tests.mjs@@ -7,13 +7,11 @@ import { spawnSync } from 'node:child_process';import { withWindowsGitOnPath } from './test-runtime-env.mjs';const here = dirname(fileURLToPath(import.meta.url));-// The open-source app is a flat repository (unlike the private monorepo, which-// nests the desktop app under PC/), so resolve both the test venv and pytest-// working directory from this repository root.-const appRoot = resolve(here, '..');+// The open-source app and its test virtual environment share this flat root.+const repoRoot = resolve(here, '..');const localPython = process.platform === 'win32'- ? resolve(appRoot, 'venv', 'Scripts', 'python.exe')- : resolve(appRoot, 'venv', 'bin', 'python');+ ? resolve(repoRoot, 'venv', 'Scripts', 'python.exe')+ : resolve(repoRoot, 'venv', 'bin', 'python');const testEnvironment = withWindowsGitOnPath(process.env);const candidates = [testEnvironment.ORKAS_TEST_PYTHON, localPython, 'python3', 'python'].filter((value, index, all) => value && all.indexOf(value) === index);@@ -32,15 +30,27 @@ if (!python) {testEnvironment.ORKAS_TEST_PYTHON = python;const args = process.argv.slice(2);+const cwdFlag = args.findIndex(arg => arg.startsWith('--cwd='));+const cwd = cwdFlag < 0 ? repoRoot : resolve(args.splice(cwdFlag, 1)[0].slice(6));if (args.length === 0) args.push('resources/builtin');-const result = spawnSync(python, ['-m', 'pytest', ...args], {- cwd: appRoot,- stdio: 'inherit',- env: testEnvironment,-});+if (args.some(arg => arg === '--resume' || arg.startsWith('--checkpoint-dir') || arg.startsWith('--batch-count'))) {+ try {+ const { runRunnerCheckpoint } = await import('./runner-checkpoint.mjs');+ process.exitCode = await runRunnerCheckpoint('python', args, { cwd, env: testEnvironment, python });+ } catch (error) {+ console.error(`[run-python-tests] ${error.message}`);+ process.exitCode = 1;+ }+} else {+ const result = spawnSync(python, ['-m', 'pytest', ...args], {+ cwd,+ stdio: 'inherit',+ env: testEnvironment,+ });-if (result.error) {- console.error(`[run-python-tests] failed to start pytest: ${result.error.message}`);- process.exit(1);+ if (result.error) {+ console.error(`[run-python-tests] failed to start pytest: ${result.error.message}`);+ process.exit(1);+ }+ process.exit(result.status ?? 1);}-process.exit(result.status ?? 1);diff --git a/scripts/run-tests.mjs b/scripts/run-tests.mjsindex 1ff88cea..1bb70db5 100644--- a/scripts/run-tests.mjs+++ b/scripts/run-tests.mjs@@ -26,12 +26,28 @@ const require_ = createRequire(import.meta.url);const electronBin = require_('electron');const vitestBin = resolve(here, '..', 'node_modules', 'vitest', 'vitest.mjs');const testEnvironment = withTestPython(withWindowsGitOnPath(process.env));+// Nested launcher calls must not overwrite their parent's checkpoint report.+delete testEnvironment.ORKAS_TEST_BATCH_REPORT;// npm appends arguments to the final shell command of a compound script.// Own the suite here so filters reach Vitest without first running every JS// test or being interpreted by npm/pytest. No arguments retains the full suite.const suiteMode = process.argv[2] === 'suite';-const args = process.argv.slice(suiteMode ? 3 : 2);+let args = process.argv.slice(suiteMode ? 3 : 2);+let checkpoint;+let runCheckpoint;+if (args.some(arg => arg === '--resume' || arg.startsWith('--checkpoint-dir') || arg.startsWith('--batch-count'))) {+ try {+ const helper = await import('./test-checkpoint.mjs');+ checkpoint = helper.checkpointOptions(!suiteMode && args[0] === 'run' ? args.slice(1) : args);+ if (!suiteMode && args[0] !== 'run') throw new Error('Checkpoint mode requires run or suite');+ args = suiteMode ? checkpoint.args : ['run', ...checkpoint.args];+ runCheckpoint = helper.runCheckpoint;+ } catch (error) {+ console.error(`[run-tests] ${error.message}`);+ process.exit(2);+ }+}const commands = [{executable: electronBin,args: [vitestBin, ...(suiteMode ? ['run', ...args] : args)],@@ -44,11 +60,14 @@ const commands = [{// outer npm/node executable as the explicit test-helper runtime.ORKAS_TEST_NODE: process.execPath,},+ checkpoint,}];if (suiteMode && args.length === 0) {commands.push({executable: process.execPath,- args: [resolve(here, 'run-python-tests.mjs'), 'resources/builtin', 'resources/test', '-q'],+ args: [resolve(here, 'run-python-tests.mjs'), 'resources/builtin', 'resources/test', '-q',+ ...(checkpoint?.directory ? [`--checkpoint-dir=${resolve(checkpoint.directory)}`, `--batch-count=${checkpoint.batches}`,+ ...(checkpoint.resume ? ['--resume'] : [])] : [])],env: testEnvironment,});}@@ -74,12 +93,33 @@ for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP', 'SIGQUIT']) {process.on(signal, handler);}-function runNext() {+async function runNext() {const command = commands.shift();if (!command) {removeSignalHandlers();return;}+ if (command.checkpoint?.directory) {+ let code;+ try {+ code = await runCheckpoint({ ...command.checkpoint, executable: electronBin, vitestBin,+ env: command.env, onChild: active => { child = active; }, cancelled: () => forwardedSignal });+ } catch (error) {+ if (error.signal) forwardedSignal ||= error.signal;+ else console.error(`[run-tests] ${error.message}`);+ code = 1;+ }+ if (forwardedSignal) {+ removeSignalHandlers();+ process.kill(process.pid, forwardedSignal);+ return;+ }+ if (code !== 0) {+ removeSignalHandlers();+ process.exit(code);+ }+ return runNext();+ }child = spawn(command.executable, command.args, { stdio: 'inherit', env: command.env });child.once('error', (error) => {removeSignalHandlers();diff --git a/scripts/runner-checkpoint.mjs b/scripts/runner-checkpoint.mjsnew file mode 100644index 00000000..1496f29a--- /dev/null+++ b/scripts/runner-checkpoint.mjs@@ -0,0 +1,152 @@+/** pytest and Playwright adapters for the shared checkpoint lifecycle. */+import fs from 'node:fs';+import path from 'node:path';+import { createHash } from 'node:crypto';+import { createRequire } from 'node:module';+import { fileURLToPath } from 'node:url';+import { spawnSync } from 'node:child_process';+import { createInterface } from 'node:readline';+import { checkpointOptions, runCheckpoint } from './test-checkpoint.mjs';+import { createStderrReceipt } from './test-stderr-receipt.mjs';+import { createModelEvalLogCollector, mergeModelEvalLogAnalyses } from './model-eval-log-policy.mjs';++const here = path.dirname(fileURLToPath(import.meta.url));+const require = createRequire(import.meta.url);+const maxBytes = 16 * 1024 * 1024;+const readJson = file => {+ if (fs.statSync(file).size > maxBytes) throw new Error('Checkpoint report exceeds its size limit');+ return JSON.parse(fs.readFileSync(file, 'utf8'));+};+const digest = async file => {+ const hash = createHash('sha256');+ for await (const chunk of fs.createReadStream(file)) hash.update(chunk);+ return hash.digest('hex');+};++async function finish(reportPath, attempt, directory, discovery) {+ if (!fs.existsSync(reportPath)) return;+ const report = readJson(reportPath);+ const analyses = [];+ const receipt = createStderrReceipt();+ for (const stream of ['stdout', 'stderr']) {+ const collector = createModelEvalLogCollector(`process-${stream}`);+ for await (const chunk of fs.createReadStream(path.join(directory, attempt[stream]), { encoding: 'utf8' })) {+ collector.ingest(stream, chunk);+ if (stream === 'stderr') receipt.ingest(chunk);+ }+ analyses.push(collector.finish());+ }+ const eventsPath = reportPath.replace(/\.json$/, '.events.jsonl');+ const scopes = new Map();+ const input = fs.createReadStream(eventsPath, { encoding: 'utf8' });+ try {+ for await (const line of createInterface({ input, crlfDelay: Infinity })) {+ if (line.length > maxBytes) throw new Error('Checkpoint event exceeds its size limit');+ const event = JSON.parse(line);+ if (!['stdout', 'stderr'].includes(event.stream) || typeof event.text !== 'string') throw new Error('Invalid checkpoint event');+ const key = JSON.stringify([event.case, event.phase]);+ if (!scopes.has(key)) {+ if (scopes.size >= 100_000) throw new Error('Checkpoint event inventory exceeds its size limit');+ scopes.set(key, createModelEvalLogCollector(key));+ }+ scopes.get(key).ingest(event.stream, event.text);+ }+ } finally { input.destroy(); }+ analyses.push(...[...scopes.values()].map(scope => scope.finish()), ...(report.appAnalyses || []));+ const analysis = mergeModelEvalLogAnalyses(analyses);+ report.logs = { status: !analysis.passed ? 'FAIL' : analysis.requiresReview ? 'WARN' : 'PASS',+ cases: report.files.reduce((sum, file) => sum + file.passed + file.failed, 0),+ files: report.files.filter(file => file.passed + file.failed > 0).length,+ unexpected: analysis.findings.filter(item => item.severity === 'error').reduce((sum, item) => sum + item.count, 0),+ unmetExpectations: 0, coverageGaps: analysis.coveragePassed ? [] : ['capture'], analysis };+ report.stderr = receipt.finish();+ report.discoveryPassed = discovery && report.reason === 'passed' && report.unhandledErrors === 0 && analysis.passed;+ const artifactPaths = [eventsPath, ...(report.artifactPaths || [])];+ // Playwright owns attachments, traces and its last-run metadata. Each batch+ // has a unique output directory so a later batch cannot erase failed evidence.+ if (!discovery && report.outputDir && fs.existsSync(report.outputDir)) {+ for (const file of fs.readdirSync(report.outputDir, { recursive: true, withFileTypes: true })) {+ if (file.isFile()) artifactPaths.push(path.join(file.parentPath || file.path, file.name));+ }+ }+ report.artifacts = [];+ for (const file of new Set(artifactPaths)) {+ const relative = path.relative(directory, file);+ if (path.isAbsolute(relative) || relative.split(path.sep).includes('..')) throw new Error('Checkpoint artifact escaped its output directory');+ report.artifacts.push({ file: relative, hash: await digest(file) });+ }+ delete report.artifactPaths;+ delete report.outputDir;+ fs.writeFileSync(reportPath, JSON.stringify(report, null, 2), { mode: 0o600 });+}++export async function runRunnerCheckpoint(lane, argv, { cwd, env, python } = {}) {+ const options = checkpointOptions(argv, lane);+ let args = options.args;+ if (lane === 'python' && args.length === 0) args = ['resources/builtin'];+ env = { ...env };+ delete env.ORKAS_TEST_BATCH_REPORT;+ let executable = process.execPath;+ let prefix;+ let extraRoots = [];+ let discoveryArgs;+ let batchArgs;+ let testFiles;+ const inventory = report => { testFiles = report?.testFiles; return report?.inventory?.map(file => ({ file })); };+ if (lane === 'python') {+ if (args.some(arg => arg.startsWith('-') && !['-q', '-v'].includes(arg))) throw new Error('Python checkpoints accept paths and -q/-v only');+ env.PYTHONDONTWRITEBYTECODE = '1';+ env.PYTHONPATH = [here, env.PYTHONPATH].filter(Boolean).join(path.delimiter);+ const probe = spawnSync(python, ['-c', 'import sys, json, sysconfig, pytest; print(json.dumps({"executable":sys.executable,"prefix":sys.prefix,"stdlib":sysconfig.get_path("stdlib"),"version":sys.version,"pytest":pytest.__version__,"paths":sys.path}))'],+ { cwd, env, encoding: 'utf8', timeout: 15_000 });+ if (probe.status !== 0) throw new Error('Cannot verify the project Python interpreter and pytest');+ const runtime = JSON.parse(probe.stdout);+ executable = runtime.executable;+ const repository = path.resolve(here, '..');+ extraRoots = [...new Set([runtime.prefix, runtime.stdlib, ...runtime.paths.filter(Boolean)])]+ .filter(root => { const relative = path.relative(repository, path.resolve(root));+ return relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative); });+ env.ORKAS_CHECKPOINT_PYTHON_RUNTIME = JSON.stringify(runtime);+ prefix = ['-m', 'pytest', '-p', 'pytest_checkpoint'];+ discoveryArgs = selection => [...selection, '--collect-only'];+ batchArgs = (batch, batches, selection) => [...selection, `--orkas-checkpoint-shard=${batch}/${batches}`];+ } else {+ if (args[0] === 'test') args = args.slice(1);+ // Keep the normal product config, reporter and output ownership explicit.+ if (args[0] === '--config' && args[1] === 'playwright.config.ts') args = args.slice(2);+ if (args.some(arg => arg.startsWith('-'))) throw new Error('E2E checkpoints accept file filters only with the default config');+ env.ORKAS_E2E_KEEP_LOGS = '1';+ prefix = [path.join(path.dirname(require.resolve('playwright/package.json')), 'cli.js')];+ const reporter = path.join(here, 'checkpoint-reporter.ts');+ discoveryArgs = (selection, report) => ['test', '--config', 'playwright.config.ts', '--list', `--reporter=${reporter}`, ...selection];+ batchArgs = (batch, batches, selection, report) => ['test', '--config', 'playwright.config.ts',+ `--reporter=line,${reporter}`, `--output=${report}.artifacts`,+ ...testFiles.filter((_, index) => index % batches === batch - 1)+ .map(file => `^${file.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`)];+ }+ // A Python lane is keyed by cwd and selected suites, not by a caller label.+ // Thus root suites and builtin resources cannot overwrite each other's state.+ const key = createHash('sha256').update(JSON.stringify([lane, cwd, args])).digest('hex').slice(0, 16);+ const directory = path.join(path.resolve(options.directory), `${lane}-${key}`);+ let child;+ let signal;+ const handlers = new Map();+ for (const name of ['SIGINT', 'SIGTERM', 'SIGHUP', 'SIGQUIT']) {+ const handler = () => { signal ||= name; if (child && !child.killed) child.kill(name); };+ handlers.set(name, handler); process.on(name, handler);+ }+ let code;+ try {+ code = await runCheckpoint({ ...options, args, directory, executable, env,+ onChild: value => { child = value; }, cancelled: () => signal,+ adapter: { lane, cwd, prefix, extraRoots, discoveryArgs, batchArgs, finish,+ inventory } });+ } catch (error) {+ if (error.signal) signal ||= error.signal;+ else throw error;+ } finally {+ for (const [name, handler] of handlers) process.off(name, handler);+ }+ if (signal) { process.kill(process.pid, signal); return 1; }+ return code;+}diff --git a/scripts/test-checkpoint.mjs b/scripts/test-checkpoint.mjsnew file mode 100644index 00000000..917417ab--- /dev/null+++ b/scripts/test-checkpoint.mjs@@ -0,0 +1,288 @@+/** Resumable Vitest batches. No application/test module state crosses batches. */+import fs from 'node:fs';+import path from 'node:path';+import os from 'node:os';+import { createHash, randomUUID } from 'node:crypto';+import { spawn, spawnSync } from 'node:child_process';+import { createStderrReceipt } from './test-stderr-receipt.mjs';++const MAX_REPORT_BYTES = 16 * 1024 * 1024;+const MAX_AGE_MS = 12 * 60 * 60 * 1000;+const slash = value => value.replaceAll('\\', '/');++export function checkpointOptions(argv, lane = 'vitest') {+ const args = [];+ const options = { directory: '', resume: false, batches: 8 };+ for (let i = 0; i < argv.length; i++) {+ const arg = argv[i];+ if (arg === '--resume') options.resume = true;+ else if (arg === '--checkpoint-dir' || arg.startsWith('--checkpoint-dir=')) {+ options.directory = arg === '--checkpoint-dir' ? argv[++i] : arg.slice('--checkpoint-dir='.length);+ if (!options.directory || options.directory.startsWith('--')) throw new Error('--checkpoint-dir needs a directory');+ } else if (arg === '--batch-count' || arg.startsWith('--batch-count=')) {+ options.batches = Number(arg === '--batch-count' ? argv[++i] : arg.slice('--batch-count='.length));+ } else args.push(arg);+ }+ if (!Number.isInteger(options.batches) || options.batches < 1 || options.batches > 32) throw new Error('--batch-count must be between 1 and 32');+ if (!options.directory && (options.resume || argv.some(arg => arg === '--batch-count' || arg.startsWith('--batch-count=')))) {+ throw new Error('Checkpoint options require --checkpoint-dir');+ }+ // Keep collection and execution equivalent. Watch, coverage, custom reporters,+ // alternate projects/configs and test-name filters need their own evidence.+ if (options.directory && lane === 'vitest' && args.some(arg => arg.startsWith('-') && !/^--maxWorkers=[1-9]\d*$/.test(arg))) {+ throw new Error('Checkpoint mode accepts file filters and --maxWorkers=N only; use the ordinary runner for other options');+ }+ return { args, ...options };+}++function readJson(file) {+ try {+ if (fs.statSync(file).size > MAX_REPORT_BYTES) return null;+ return JSON.parse(fs.readFileSync(file, 'utf8'));+ } catch { return null; }+}++function writeJson(file, value) {+ const temporary = `${file}.tmp`;+ fs.writeFileSync(temporary, `${JSON.stringify(value, null, 2)}\n`, { mode: 0o600 });+ fs.renameSync(temporary, file);+}++async function hashFile(file) {+ const hash = createHash('sha256');+ for await (const chunk of fs.createReadStream(file)) hash.update(chunk);+ return hash.digest('hex');+}++async function stderrMatches(file, expected) {+ if (!expected?.valid) return false;+ const receipt = createStderrReceipt();+ for await (const chunk of fs.createReadStream(file)) receipt.ingest(chunk);+ const actual = receipt.finish();+ return actual.valid && actual.bytes === expected.bytes && actual.lines === expected.lines && actual.digest === expected.digest;+}++// Metadata, not file bodies: tracked private configuration must never be read.+// ctime also invalidates restored timestamps. Scan once at each batch boundary,+// never per case; output directories are outside the source inventory.+export function inputFingerprint(repoRoot, env, selection, inventory, executable, extraRoots = []) {+ const git = spawnSync('git', ['ls-files', '-z', '--cached', '--others', '--exclude-standard', '--deduplicate'], {+ cwd: repoRoot, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024,+ });+ if (git.status !== 0) throw new Error('Cannot establish the repository input inventory');+ const hash = createHash('sha256');+ const add = value => hash.update(JSON.stringify(value) + '\n');+ add({ schema: 1, host: os.hostname(), release: os.release(), platform: process.platform, arch: process.arch,+ versions: process.versions, executable, cwd: repoRoot, selection, inventory,+ env: Object.entries(env).sort(([a], [b]) => a.localeCompare(b)) });+ let entries = 0;+ const record = (file, knownStat) => {+ if (++entries > 500_000) throw new Error('Input inventory exceeds the checkpoint safety limit');+ try {+ const stat = knownStat || fs.lstatSync(file, { bigint: true });+ add([file, stat.mode.toString(), stat.size.toString(), stat.mtimeNs.toString(), stat.ctimeNs.toString(), stat.ino.toString()]);+ if (stat.isSymbolicLink()) {+ const target = fs.realpathSync(file);+ const resolved = fs.statSync(target, { bigint: true });+ add([target, resolved.size.toString(), resolved.mtimeNs.toString(), resolved.ctimeNs.toString()]);+ }+ } catch (error) {+ if (error.code !== 'ENOENT') throw new Error('Cannot inspect a required checkpoint input');+ add([file, 'missing']);+ }+ };+ for (const file of git.stdout.split('\0').filter(Boolean).sort()) record(path.join(repoRoot, file));+ const visited = new Set();+ const walk = file => {+ let stat;+ try { stat = fs.lstatSync(file, { bigint: true }); }+ catch (error) { if (error.code === 'ENOENT') { record(file); return; } throw error; }+ if (stat.isSymbolicLink()) {+ record(file, stat);+ // Follow dependency links as well as recording the link itself.+ if (fs.existsSync(file)) walk(fs.realpathSync(file));+ return;+ }+ if (!stat.isDirectory()) { record(file, stat); return; }+ if (visited.has(file)) return;+ visited.add(file);+ for (const entry of fs.readdirSync(file).sort()) {+ // Vitest writes these caches while collecting/executing unchanged inputs.+ if (path.basename(file) === 'node_modules' && ['.vite', '.vite-temp', '.cache'].includes(entry)) continue;+ walk(path.join(file, entry));+ }+ };+ for (const root of ['node_modules', 'src/core-agent/node_modules', 'venv',+ 'resources/runtime', 'resources/embedding-model', 'resources/officecli', 'src/main/native/build', 'out', ...extraRoots]) {+ walk(path.resolve(repoRoot, root));+ }+ record(process.execPath);+ record(executable);+ if (env.ORKAS_TEST_PYTHON && path.isAbsolute(env.ORKAS_TEST_PYTHON)) record(env.ORKAS_TEST_PYTHON);+ return hash.digest('hex');+}++function validReport(report) {+ return report?.schema === 1 && report.reason === 'passed' && report.unhandledErrors === 0+ && report.stderr?.valid === true+ && Array.isArray(report.inventory) && Array.isArray(report.files)+ && new Set(report.inventory).size === report.inventory.length+ && JSON.stringify([...report.inventory].sort()) === JSON.stringify(report.files.map(file => file.file).sort())+ && report.files.every(file => ['passed', 'skipped'].includes(file.state) && file.failed === 0+ && [file.passed, file.failed, file.skipped].every(count => Number.isInteger(count) && count >= 0))+ && report.logs?.cases === report.files.reduce((count, file) => count + file.passed + file.failed, 0)+ && report.logs?.files === report.files.filter(file => file.passed + file.failed > 0).length+ && ['PASS', 'WARN'].includes(report.logs?.status) && report.logs.unexpected === 0+ && report.logs.unmetExpectations === 0 && report.logs.coverageGaps?.length === 0;+}++export async function runCheckpoint({ directory, resume, batches, args, executable, vitestBin, env, onChild, cancelled = () => null, adapter }) {+ const started = performance.now();+ let inputScanMs = 0;+ const pcRoot = adapter?.cwd || path.resolve(path.dirname(vitestBin), '../..');+ const git = spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: pcRoot, encoding: 'utf8' });+ if (git.status !== 0) throw new Error('Checkpoint mode requires a Git checkout');+ const repoRoot = git.stdout.trim();+ directory = path.resolve(directory);+ const relative = path.relative(repoRoot, directory);+ if (!relative.startsWith(`..${path.sep}`) && !path.isAbsolute(relative)) {+ const ignored = spawnSync('git', ['check-ignore', '-q', path.join(directory, 'checkpoint.json')], { cwd: repoRoot });+ if (ignored.status !== 0) throw new Error('Checkpoint directory must be Git-ignored or outside the checkout');+ }+ fs.mkdirSync(directory, { recursive: true, mode: 0o700 });+ const lock = path.join(directory, 'run.lock');+ let lockFd;+ try { lockFd = fs.openSync(lock, 'wx', 0o600); }+ catch { throw new Error('Checkpoint is locked; stop the other run before removing a stale run.lock'); }+ fs.writeSync(lockFd, String(process.pid));+ const session = randomUUID();+ const sessionDir = path.join(directory, session);+ fs.mkdirSync(sessionDir, { mode: 0o700 });+ const run = async (label, commandArgs, report) => {+ const started = performance.now();+ const stdout = path.join(session, `${label}.stdout.log`);+ const stderr = path.join(session, `${label}.stderr.log`);+ const out = fs.openSync(path.join(directory, stdout), 'wx', 0o600);+ const err = fs.openSync(path.join(directory, stderr), 'wx', 0o600);+ let result;+ try {+ result = await new Promise((resolve, reject) => {+ const child = spawn(executable, [...(adapter?.prefix || [vitestBin]), ...commandArgs], {+ cwd: pcRoot, env: { ...env, ORKAS_TEST_BATCH_REPORT: report || '' }, stdio: ['inherit', out, err],+ });+ onChild?.(child);+ child.once('error', () => reject(new Error('Cannot start the checkpoint test runtime')));+ child.once('close', (exitCode, signal) => resolve({ exitCode, signal }));+ });+ } finally { fs.closeSync(out); fs.closeSync(err); }+ const attempt = { ...result, durationMs: Math.round(performance.now() - started), stdout, stderr,+ stdoutHash: await hashFile(path.join(directory, stdout)), stderrHash: await hashFile(path.join(directory, stderr)) };+ writeJson(path.join(sessionDir, `${label}.attempt.json`), attempt);+ if (result.signal) throw Object.assign(new Error('Checkpoint run interrupted'), { signal: result.signal });+ return attempt;+ };+ try {+ const discoveryPath = path.join(sessionDir, 'inventory.json');+ const discovery = await run('discovery', adapter+ ? adapter.discoveryArgs(args, discoveryPath)+ : ['list', '--filesOnly', `--json=${discoveryPath}`, ...args], adapter ? discoveryPath : undefined);+ if (adapter) await adapter.finish(discoveryPath, discovery, directory, true);+ const discoveryReport = readJson(discoveryPath);+ const discovered = adapter ? adapter.inventory(discoveryReport) : discoveryReport;+ if (discovery.exitCode !== 0 || (adapter ? !discoveryReport?.discoveryPassed : fs.statSync(path.join(directory, discovery.stderr)).size !== 0)+ || !Array.isArray(discovered) || !discovered.length+ || discovered.some(item => typeof item.file !== 'string' || item.projectName)) {+ throw new Error('Test discovery failed or returned an unsupported inventory; inspect discovery logs');+ }+ if (adapter) console.log(`[test-checkpoint] discovery log analysis ${discoveryReport.logs.status}`);+ const inventory = discovered.map(item => adapter ? item.file : slash(path.relative(pcRoot, item.file))).sort();+ if (new Set(inventory).size !== inventory.length) throw new Error('Duplicate test discovery entries');+ batches = Math.min(batches, adapter ? discoveryReport.batchUnits : inventory.length);+ if (!Number.isInteger(batches) || batches < 1) throw new Error('Invalid checkpoint batch inventory');+ const fingerprint = () => {+ const scanStarted = performance.now();+ const result = inputFingerprint(repoRoot, env, { args, batches, ...(adapter ? { lane: adapter.lane } : {}) }, inventory, executable, adapter?.extraRoots);+ inputScanMs += performance.now() - scanStarted;+ return result;+ };+ const initial = fingerprint();+ const statePath = path.join(directory, 'checkpoint.json');+ const previous = resume ? readJson(statePath) : null;+ const reusable = previous?.schema === 1 && previous.fingerprint === initial+ && Date.now() - previous.createdAt < MAX_AGE_MS && Date.now() >= previous.createdAt;+ const state = reusable ? previous : { schema: 1, fingerprint: initial, createdAt: Date.now(), inventory, completed: {} };+ writeJson(statePath, state);+ console.log(`[test-checkpoint] discovered=${inventory.length} batches=${batches}; ${reusable ? 'checking saved evidence' : 'fresh execution'}`);+ const results = [];+ let changed = false;+ for (let batch = 1; batch <= batches; batch++) {+ if (cancelled()) throw Object.assign(new Error('Checkpoint run interrupted'), { signal: cancelled() });+ let entry = state.completed[batch];+ let report;+ if (entry) {+ try {+ const safe = file => typeof file === 'string' && !path.isAbsolute(file) && !file.split(/[\\/]/).includes('..');+ if (![entry.report, entry.stdout, entry.stderr].every(safe)) throw new Error('Invalid artifact location');+ report = readJson(path.join(directory, entry.report));+ if (entry.certified !== true || entry.stderrGap || !validReport(report) || await hashFile(path.join(directory, entry.report)) !== entry.reportHash+ || await hashFile(path.join(directory, entry.stdout)) !== entry.stdoutHash+ || await hashFile(path.join(directory, entry.stderr)) !== entry.stderrHash) entry = null;+ if (entry && adapter) {+ if (!Array.isArray(report.artifacts)) entry = null;+ else for (const artifact of report.artifacts) {+ if (!safe(artifact.file) || await hashFile(path.join(directory, artifact.file)) !== artifact.hash) entry = null;+ }+ }+ } catch { entry = null; }+ }+ if (entry) {+ results.push({ batch, reused: true, ...entry, files: report.files, logs: report.logs });+ console.log(`[test-checkpoint] batch ${batch}/${batches} reused; log analysis ${report.logs.status}`);+ continue;+ }+ delete state.completed[batch];+ writeJson(statePath, state);+ console.log(`[test-checkpoint] batch ${batch}/${batches} starting`);+ const reportFile = path.join(session, `batch-${batch}.json`);+ // The worker tap still analyzes every case log and surfaces warnings in+ // the report. Suppress only the duplicate successful-case console replay+ // so any remaining process stderr has no ambiguous attribution.+ const attempt = await run(`batch-${batch}`, adapter+ ? adapter.batchArgs(batch, batches, args, path.join(directory, reportFile))+ : ['run', '--silent=passed-only', `--shard=${batch}/${batches}`, ...args], path.join(directory, reportFile));+ if (adapter) await adapter.finish(path.join(directory, reportFile), attempt, directory, false);+ report = readJson(path.join(directory, reportFile));+ const stderrGap = attempt.exitCode === 0 && !await stderrMatches(path.join(directory, attempt.stderr), report?.stderr);+ const certified = attempt.exitCode === 0 && validReport(report) && !stderrGap;+ if (fingerprint() !== initial) changed = true;+ entry = { ...attempt, report: reportFile, reportHash: report ? await hashFile(path.join(directory, reportFile)) : '', certified, stderrGap };+ if (certified && !changed) state.completed[batch] = entry;+ if (changed) state.completed = {};+ writeJson(statePath, state);+ results.push({ batch, reused: false, ...entry, files: report?.files || [], logs: report?.logs });+ console.log(`[test-checkpoint] batch ${batch}/${batches} ${certified ? 'completed' : 'FAILED'} ${attempt.durationMs}ms; log analysis ${report?.logs?.status || 'MISSING'}`);+ if (stderrGap) console.log('[test-checkpoint] unclassified stderr: inspect this batch stderr artifact before rerunning');+ }+ // A cached or newly executed shard may never hide an omitted/new file.+ const executed = results.flatMap(result => result.files.map(file => file.file)).sort();+ const complete = JSON.stringify(executed) === JSON.stringify(inventory);+ changed ||= fingerprint() !== initial;+ const exitCode = !complete || changed || results.some(result => !result.certified) ? 1 : 0;+ // Missing evidence for one batch must not discard independently certified+ // siblings. A fully certified but inconsistent union invalidates all shards.+ if (changed || (!complete && results.every(result => result.certified))) {+ state.completed = {}; writeJson(statePath, state);+ }+ const summary = { schema: 1, inventory, complete, inputsChanged: changed, exitCode,+ wallDurationMs: Math.round(performance.now() - started), inputScanMs: Math.round(inputScanMs),+ reused: results.filter(result => result.reused).length,+ savedDurationMs: results.filter(result => result.reused).reduce((sum, result) => sum + result.durationMs, 0), results };+ writeJson(path.join(sessionDir, 'summary.json'), summary);+ console.log(`[test-checkpoint] executed=${batches - summary.reused} reused=${summary.reused} saved=${summary.savedDurationMs}ms inventory=${complete ? 'complete' : 'INCOMPLETE'} inputs=${changed ? 'CHANGED' : 'stable'}`);+ console.log(`[test-checkpoint] reports: ${path.join(sessionDir, 'summary.json')}`);+ return exitCode;+ } finally {+ fs.closeSync(lockFd);+ fs.unlinkSync(lock);+ }+}diff --git a/scripts/test-runtime-env.mjs b/scripts/test-runtime-env.mjsindex 62b18d09..08519de0 100644--- a/scripts/test-runtime-env.mjs+++ b/scripts/test-runtime-env.mjs@@ -5,7 +5,7 @@ import { fileURLToPath } from 'node:url';/** Keep deterministic evaluator fixtures on the same project interpreter as* the test suite, including direct npm test invocations. Explicit evaluator* overrides remain independent of the ordinary Python suite. */-export function withTestPython(env, repoRoot = fileURLToPath(new URL('../../', import.meta.url)), platform = process.platform) {+export function withTestPython(env, repoRoot = fileURLToPath(new URL('../', import.meta.url)), platform = process.platform) {const result = { ...env };const localPython = platform === 'win32'? path.join(repoRoot, 'venv', 'Scripts', 'python.exe')diff --git a/scripts/test-stderr-receipt.mjs b/scripts/test-stderr-receipt.mjsnew file mode 100644index 00000000..66ce008e--- /dev/null+++ b/scripts/test-stderr-receipt.mjs@@ -0,0 +1,61 @@+import { createHash } from 'node:crypto';++const MODULUS = 1n << 256n;+const MAX_LINE_BYTES = 1024 * 1024;++// Multiset digest of exact lines: concurrent workers may reorder whole lines,+// but missing, added, modified or interleaved bytes cannot inherit attribution.+// Only an aggregate digest/count leaves the worker, never raw diagnostic text.+export function createStderrReceipt() {+ let lineHash = createHash('sha256');+ let lineBytes = 0;+ let sum = 0n;+ let bytes = 0;+ let lines = 0;+ let valid = true;+ const finishLine = () => {+ sum = (sum + BigInt(`0x${lineHash.digest('hex')}`)) % MODULUS;+ lines += 1;+ lineHash = createHash('sha256');+ lineBytes = 0;+ };+ return {+ ingest(chunk, encoding) {+ const data = Buffer.isBuffer(chunk) ? chunk : typeof chunk === 'string' ? Buffer.from(chunk, encoding) : Buffer.from(chunk);+ bytes += data.length;+ if (!valid) return;+ let start = 0;+ while (start < data.length) {+ const newline = data.indexOf(10, start);+ const end = newline === -1 ? data.length : newline + 1;+ lineBytes += end - start;+ if (lineBytes > MAX_LINE_BYTES) { valid = false; return; }+ lineHash.update(data.subarray(start, end));+ if (newline !== -1) finishLine();+ start = end;+ }+ },+ finish() {+ if (valid && lineBytes) finishLine();+ return { valid, bytes, lines, digest: sum.toString(16).padStart(64, '0') };+ },+ };+}++export function combineStderrReceipts(receipts) {+ let sum = 0n;+ let bytes = 0;+ let lines = 0;+ let valid = true;+ for (const receipt of receipts) {+ if (!receipt?.valid || !Number.isSafeInteger(receipt.bytes) || receipt.bytes < 0+ || !Number.isSafeInteger(receipt.lines) || receipt.lines < 0 || !/^[a-f0-9]{64}$/.test(receipt.digest)) {+ valid = false;+ continue;+ }+ sum = (sum + BigInt(`0x${receipt.digest}`)) % MODULUS;+ bytes += receipt.bytes;+ lines += receipt.lines;+ }+ return { valid, bytes, lines, digest: sum.toString(16).padStart(64, '0') };+}diff --git a/test/e2e/fixtures/orkas.ts b/test/e2e/fixtures/orkas.tsindex fd467301..0c3530f7 100644--- a/test/e2e/fixtures/orkas.ts+++ b/test/e2e/fixtures/orkas.ts@@ -1835,6 +1835,10 @@ export class OrkasTestApp {for (const record of await this.rendererErrorEvidence()) {if (record.windowId !== null && windows.has(record.windowId)) this.recoveredRendererErrors.add(record.diagnosticIndex);}+ return this.recoveredDiagnosticRanges();+ }++ private recoveredDiagnosticRanges() {let line = 1;const ranges: Array<{ start: number; end: number; diagnosticIndex: number }> = [];this.diagnostics.forEach((entry, diagnosticIndex) => {@@ -1867,6 +1871,17 @@ export class OrkasTestApp {failed = true;}+ // Export receipts after releasing native/stub ownership. Evidence IO must+ // fail certification without interrupting resource cleanup.+ if (process.env.ORKAS_TEST_BATCH_REPORT) {+ try {+ await this.testInfo.attach('electron-log-required', { body: Buffer.from(JSON.stringify(this.recoveredDiagnosticRanges())), contentType: 'application/json' });+ } catch (error) {+ cleanupError = cleanupError ? new AggregateError([cleanupError, error], 'Fixture cleanup failed') : error;+ failed = true;+ }+ }+const unexpectedApiFailures = this.apiResponses.filter(item => {if (item.status < 400) return false;const index = this.expectedApiFailures.findIndex(expected => expected.method === item.method@@ -1876,7 +1891,7 @@ export class OrkasTestApp {return false;});failed ||= unexpectedApiFailures.length > 0 || this.expectedApiFailures.length > 0;- if (failed || process.env.ORKAS_E2E_KEEP_LOGS === '1') {+ if (failed || process.env.ORKAS_E2E_KEEP_LOGS === '1' || process.env.ORKAS_TEST_BATCH_REPORT) {mkdirSync(this.testInfo.outputDir, { recursive: true });writeFileSync(this.testInfo.outputPath('stub-responses.json'), JSON.stringify(this.apiResponses, null, 2));}@@ -1901,7 +1916,8 @@ export class OrkasTestApp {// reporters truncate an inline attachment to a couple of lines, which is// exactly the main-process history a failed run needs. `attach({ path })`// keeps the full file in the artifacts tree.- const logPath = path.join(this.testInfo.outputDir, 'electron-main.log');+ const logPath = path.join(this.testInfo.outputDir, process.env.ORKAS_TEST_BATCH_REPORT+ ? `checkpoint-electron-${randomUUID()}.log` : 'electron-main.log');try {mkdirSync(this.testInfo.outputDir, { recursive: true });writeFileSync(logPath, diagnosticText, 'utf8');@@ -1921,15 +1937,24 @@ export class OrkasTestApp {// ORKAS_E2E_KEEP_LOGS=1 preserves the same captured stdout/stderr,// renderer diagnostics and app logs as a failed run. File logs alone// omit direct console output, including renderer warnings.- if (process.env.ORKAS_E2E_KEEP_LOGS === '1') {+ if (process.env.ORKAS_E2E_KEEP_LOGS === '1' || process.env.ORKAS_TEST_BATCH_REPORT) {const appLogs = collectLogFiles(this.root);const diagnosticText = [this.diagnostics.join('\n'), appLogs].filter(Boolean).join('\n');if (diagnosticText) {try {mkdirSync(this.testInfo.outputDir, { recursive: true });writeFileSync(path.join(this.testInfo.outputDir, 'electron-main.log'), diagnosticText, 'utf8');- } catch {- // Diagnostics only — never fail a passing test over log capture.+ if (process.env.ORKAS_TEST_BATCH_REPORT) {+ const snapshot = this.testInfo.outputPath(`checkpoint-electron-${randomUUID()}.log`);+ writeFileSync(snapshot, diagnosticText, 'utf8');+ await this.testInfo.attach('electron-logs', { path: snapshot, contentType: 'text/plain' });+ }+ } catch (error) {+ // Fail reuse after releasing the temporary root and trace files.+ if (process.env.ORKAS_TEST_BATCH_REPORT) {+ cleanupError = new Error('Checkpoint Electron log capture failed', { cause: error });+ }+ // Ordinary runs preserve their existing diagnostic behavior.}}}diff --git a/test/fixtures/test-log-analysis/stateful.fixture.ts b/test/fixtures/test-log-analysis/stateful.fixture.tsnew file mode 100644index 00000000..008d36a7--- /dev/null+++ b/test/fixtures/test-log-analysis/stateful.fixture.ts@@ -0,0 +1,20 @@+import { it } from 'vitest';+import { expectTestLog } from '../../helpers/test-log-analysis';++it.each([+ ['global', /fixture repeated diagnostic/g],+ ['sticky', /fixture repeated diagnostic/y],+ ['frozen-global', Object.freeze(/fixture repeated diagnostic/g)],+ ['frozen-sticky', Object.freeze(/fixture repeated diagnostic/y)],+] as const)('matches every repeated %s declaration', (_kind, message) => {+ expectTestLog({ level: 'error', message });+ console.error('fixture repeated diagnostic');+ console.error('fixture repeated diagnostic');+});++it('still rejects a different error after repeated declared output', () => {+ expectTestLog({ level: 'error', message: /fixture declared diagnostic/g });+ console.error('fixture declared diagnostic');+ console.error('fixture declared diagnostic');+ console.error('fixture different unexpected diagnostic');+});diff --git a/test/fixtures/test-log-analysis/vitest.config.ts b/test/fixtures/test-log-analysis/vitest.config.tsindex 9fb1d8b6..f690b830 100644--- a/test/fixtures/test-log-analysis/vitest.config.ts+++ b/test/fixtures/test-log-analysis/vitest.config.ts@@ -5,6 +5,10 @@ import { fileURLToPath } from 'node:url';import { defineConfig } from 'vitest/config';import base from '../../../vitest.config';+// Exercise checkpoint receipts through the real launcher without inheriting a+// parent run's report destination.+process.env.ORKAS_TEST_BATCH_REPORT = process.env.ORKAS_LOG_FIXTURE_BATCH_REPORT || '';+const reporters = [...(base.test?.reporters as unknown[])];if (process.env.ORKAS_LOG_FIXTURE_JSON) reporters.push(['json', { outputFile: process.env.ORKAS_LOG_FIXTURE_JSON }]);diff --git a/test/helpers/source-map-literals.cjs b/test/helpers/source-map-literals.cjsnew file mode 100644index 00000000..7f2fe9b0--- /dev/null+++ b/test/helpers/source-map-literals.cjs@@ -0,0 +1,38 @@+'use strict';++// convert-source-map scans raw lines, including lines inside tsx's template+// strings. Vitest must retain the original test failure instead of crashing+// while decoding that template as a source map. Real malformed maps still fail.+const convert = require('convert-source-map');+const original = convert.fromSource;+const ts = require('typescript');++function fromSource(source) {+ try { return original(source); }+ catch (error) {+ if (!(error instanceof SyntaxError)) throw error;+ const parsed = ts.createSourceFile('source.js', source, ts.ScriptTarget.Latest, true, ts.ScriptKind.JS);+ if (parsed.parseDiagnostics.length) throw error;+ const literals = [];+ function visit(node) {+ if (ts.isStringLiteralLike(node) || node.kind === ts.SyntaxKind.TemplateHead+ || node.kind === ts.SyntaxKind.TemplateMiddle || node.kind === ts.SyntaxKind.TemplateTail) {+ literals.push([node.getStart(parsed), node.end]);+ }+ ts.forEachChild(node, visit);+ }+ visit(parsed);+ let removed = false;+ const comments = [...source.matchAll(convert.commentRegex)].filter(match => {+ const marker = match.index + match[0].search(/\/[/*]/);+ const literal = literals.some(([start, end]) => start <= marker && end > marker);+ removed ||= literal;+ return !literal;+ });+ if (!removed) throw error;+ return original(comments.map(match => match[0]).join('\n'));+ }+}++exports.fromSource = fromSource;+exports.install = () => { convert.fromSource = fromSource; };diff --git a/test/helpers/test-log-analysis.ts b/test/helpers/test-log-analysis.tsindex d66d29b1..2eb05c89 100644--- a/test/helpers/test-log-analysis.ts+++ b/test/helpers/test-log-analysis.ts@@ -22,6 +22,7 @@ import { afterAll, afterEach, beforeEach } from 'vitest';import { createModelEvalLogCollector } from '../../scripts/model-eval-log-policy.mjs';import { sanitizeLogTextForUpload } from '../../src/main/util/log-sanitize';import { TEST_LOG_META_KEY } from '../log-analysis-reporter';+import { createStderrReceipt } from '../../scripts/test-stderr-receipt.mjs';export type TestLogLevel = 'warn' | 'error';export type TestLogPhase = 'case' | 'setup' | 'cleanup';@@ -90,6 +91,7 @@ interface TapState {fileExpectations: { spec: ExpectedTestLog; hits: number }[];setup: Scope;cleanup?: Scope;+ stderrReceipt?: ReturnType<typeof createStderrReceipt>;}const STATE_KEY = Symbol.for('orkas.test.log-analysis');@@ -106,7 +108,7 @@ function newScope(phase: TestLogPhase, expectations: Scope['expectations']): Scoconst message = String(line ?? '').replace(RUNTIME_PREFIX_RE, '');const level: TestLogLevel = finding.severity === 'error' ? 'error' : 'warn';const match = scope.expectations.find(({ spec }) => spec.level === level- && (spec.source === undefined || spec.source === source) && spec.message.test(message));+ && (spec.source === undefined || spec.source === source) && matchesExpectedMessage(spec.message, message));if (match) { match.hits += 1; scope.expectedCount += 1; }else if (finding.severity === 'error') scope.unexpectedErrorCount += 1;const signature = testLogSignature(line);@@ -123,6 +125,12 @@ function newScope(phase: TestLogPhase, expectations: Scope['expectations']): Scoreturn scope;}+function matchesExpectedMessage(pattern: RegExp, message: string): boolean {+ // Stateful declarations still match each record independently.+ if (pattern.global || pattern.sticky) pattern.lastIndex = 0;+ return pattern.test(message);+}+function summarize(scope: Scope): TestLogScopeSummary {const analysis = scope.collector.finish();const unmetExpectations = scope.expectations.filter(item => item.hits === 0).length;@@ -169,7 +177,11 @@ function tapState(): TapState | undefined {export function expectTestLog(spec: ExpectedTestLog): void {const state = tapState();if (!state) throw new Error('test log analysis is not installed');- const entry = { spec, hits: 0 };+ // Own mutable match state without changing (or writing into a frozen)+ // caller-owned RegExp. Ordinary stateless declarations retain their path.+ const owned = spec.message.global || spec.message.sticky+ ? { ...spec, message: new RegExp(spec.message.source, spec.message.flags) } : spec;+ const entry = { spec: owned, hits: 0 };if (state.current.phase === 'case') state.current.expectations.push(entry);else state.fileExpectations.push(entry);}@@ -212,6 +224,9 @@ function installStreamTaps(state: TapState): void {const original = target.write;target.write = function tappedWrite(this: typeof target, chunk: unknown, ...rest: unknown[]) {ingest(stream, chunk, rest[0]);+ if (stream === 'stderr' && state.stderrReceipt) {+ state.stderrReceipt.ingest(chunk, typeof rest[0] === 'string' ? rest[0] : undefined);+ }return (original as (...args: unknown[]) => boolean).call(this, chunk, ...rest);} as typeof target.write;}@@ -241,6 +256,7 @@ export function installTestLogAnalysis(): void {installStreamTaps(state);}const fileState = state;+ fileState.stderrReceipt = process.env.ORKAS_TEST_BATCH_REPORT ? createStderrReceipt() : undefined;beforeEach(() => {fileState.current = newScope('case', []);@@ -265,6 +281,7 @@ export function installTestLogAnalysis(): void {for (const phase of phases) phase.unmetExpectations = 0;phases[phases.length - 1].unmetExpectations = unmet;(suite.meta as Record<string, unknown>)[TEST_LOG_META_KEY] = phases;+ if (fileState.stderrReceipt) (suite.meta as Record<string, unknown>).orkasStderrReceipt = fileState.stderrReceipt.finish();// Output after the last hook (worker teardown) has no reporting channel.fileState.setup = newScope('setup', []);fileState.cleanup = undefined;diff --git a/test/log-analysis-reporter.ts b/test/log-analysis-reporter.tsindex d57791c6..8811797f 100644--- a/test/log-analysis-reporter.ts+++ b/test/log-analysis-reporter.ts@@ -8,8 +8,10 @@* signatures, repository-relative test files and test names.*/import { relative } from 'node:path';+import { writeFileSync } from 'node:fs';import type { Reporter, TestModule } from 'vitest/node';import type { TestLogScopeSummary, TestLogSignature } from './helpers/test-log-analysis';+import { combineStderrReceipts } from '../scripts/test-stderr-receipt.mjs';/** Task-meta key shared with the worker tap; this module must not load Vitest's runtime. */export const TEST_LOG_META_KEY = 'orkasLogAnalysis';@@ -117,14 +119,45 @@ export function formatTestLogSummary(summary: TestLogRunSummary): string {export class TestLogAnalysisReporter implements Reporter {private root = process.cwd();+ private inventory: string[] = [];onInit(vitest: { config: { root: string } }): void {this.root = vitest.config.root;}- onTestRunEnd(modules: readonly TestModule[]): void {+ onTestRunStart(): void {+ this.inventory = [];+ }++ onTestModuleQueued(module: TestModule): void {+ // onTestRunStart receives the pre-shard inventory in Vitest 4. Queue events+ // identify the selected shard, including files that fail during collection.+ this.inventory.push(relative(this.root, module.moduleId).replace(/\\/g, '/'));+ }++ onTestRunEnd(modules: readonly TestModule[], errors: readonly unknown[] = [], reason = 'passed'): void {const summary = summarizeTestLogs(modules, this.root);process.stdout.write(formatTestLogSummary(summary));if (summary.status === 'FAIL') process.exitCode = 1;+ if (process.env.ORKAS_TEST_BATCH_REPORT) {+ const files = modules.map(module => {+ const counts = { passed: 0, failed: 0, skipped: 0 };+ for (const test of module.children.allTests()) {+ const state = test.result().state;+ if (state in counts) counts[state as keyof typeof counts] += 1;+ }+ const diagnostic = module.diagnostic();+ return { file: relative(this.root, module.moduleId).replace(/\\/g, '/'), state: module.state(), ...counts,+ durationMs: diagnostic.duration, collectMs: diagnostic.collectDuration, setupMs: diagnostic.setupDuration,+ prepareMs: diagnostic.prepareDuration, environmentMs: diagnostic.environmentSetupDuration };+ });+ // No raw logs, environment values or test payloads in structured reports.+ // The launcher retains stdout/stderr separately with restricted permissions.+ writeFileSync(process.env.ORKAS_TEST_BATCH_REPORT, JSON.stringify({ schema: 1, reason,+ unhandledErrors: errors.length, inventory: this.inventory, files, logs: summary,+ stderr: combineStderrReceipts(modules.filter(module => module.state() !== 'skipped')+ .map(module => (module.meta() as Record<string, unknown>).orkasStderrReceipt)),+ }), { mode: 0o600 });+ }}}diff --git a/test/main/quality/run-non-model.test.ts b/test/main/quality/run-non-model.test.tsnew file mode 100644index 00000000..904aba5c--- /dev/null+++ b/test/main/quality/run-non-model.test.ts@@ -0,0 +1,145 @@+import fs from 'node:fs';+import os from 'node:os';+import path from 'node:path';+import { spawnSync } from 'node:child_process';+import { pathToFileURL } from 'node:url';+import { afterEach, describe, expect, it } from 'vitest';++const appRoot = path.resolve(__dirname, '../../..');+const node = process.env.ORKAS_TEST_NODE || process.execPath;+const roots: string[] = [];+afterEach(() => roots.splice(0).forEach(root => fs.rmSync(root, { recursive: true, force: true })));+const laneOrder = ['typecheck', 'test', 'test:e2e:typecheck', 'test:e2e', 'test:platform-native', 'smoke'];++// Execute the actual CLI in a disposable flat checkout. Only child dispatch is+// replaced: no business test, Electron window or native host process is run.+function runEntry(options: { platform?: string; args?: string[]; fail?: string; failure?: string } = {}) {+ // Node resolves the entry module and child cwd through macOS's /var alias.+ // Use the fixture's real location for both invocation and exact path oracles.+ const root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-non-model-')));+ roots.push(root);+ fs.mkdirSync(path.join(root, 'scripts'));+ for (const file of ['run-non-model.mjs', 'test-checkpoint.mjs', 'test-stderr-receipt.mjs', 'test-runtime-env.mjs']) {+ fs.copyFileSync(path.join(appRoot, 'scripts', file), path.join(root, 'scripts', file));+ }+ const callsPath = path.join(root, 'dispatch.jsonl');+ const preload = path.join(root, 'dispatch.mjs');+ fs.writeFileSync(preload, `+ import fs from 'node:fs';+ import cp from 'node:child_process';+ import { syncBuiltinESMExports } from 'node:module';+ Object.defineProperty(process, 'platform', { value: ${JSON.stringify(options.platform || process.platform)} });+ cp.spawnSync = (command, args, options) => {+ if (command === 'where.exe') return { status: 1, stdout: '' };+ fs.appendFileSync(${JSON.stringify(callsPath)}, JSON.stringify({ command, args, cwd: options.cwd,+ stdio: options.stdio, shell: options.shell, sentinel: options.env.NON_MODEL_SENTINEL }) + '\\n');+ const script = args.includes('run') ? args[args.indexOf('run') + 1] : 'test';+ // The parent must persist each completed stage before starting the next.+ const dispatched = fs.readFileSync(${JSON.stringify(callsPath)}, 'utf8').trim().split('\\n').length;+ if (dispatched > 1) {+ if (!fs.existsSync('test-results')) return { status: 91 };+ const report = JSON.parse(fs.readFileSync('test-results/' + fs.readdirSync('test-results')[0], 'utf8'));+ if (report.stages.length !== dispatched - 1) return { status: 91 };+ }+ if (script === ${JSON.stringify(options.fail || '')}) {+ if (${JSON.stringify(options.failure)} === 'spawn') return { status: null, signal: null, error: new Error('controlled spawn failure') };+ if (${JSON.stringify(options.failure)} === 'signal') return { status: null, signal: 'SIGTERM' };+ return { status: 7, signal: null };+ }+ return { status: 0, signal: null };+ };+ syncBuiltinESMExports();+ `);+ const result = spawnSync(node, ['--import', pathToFileURL(preload).href,+ path.join(root, 'scripts/run-non-model.mjs'), ...(options.args || [])], {+ cwd: root, encoding: 'utf8', timeout: 10_000,+ env: { ...process.env, ELECTRON_RUN_AS_NODE: '1', npm_execpath: preload, NON_MODEL_SENTINEL: 'preserved' },+ });+ expect(result.error).toBeUndefined();+ const calls = fs.existsSync(callsPath)+ ? fs.readFileSync(callsPath, 'utf8').trim().split('\n').map(line => JSON.parse(line)) : [];+ const reports = fs.existsSync(path.join(root, 'test-results'))+ ? fs.readdirSync(path.join(root, 'test-results')).map(file => ({ file, value: JSON.parse(fs.readFileSync(path.join(root, 'test-results', file), 'utf8')) })) : [];+ return { root, preload, result, calls, reports };+}++describe('flat repository non-model entry', () => {+ it.each(['darwin', 'linux', 'win32'])('preserves all six serial lanes and stage receipts on %s', platform => {+ const { root, preload, result, calls, reports } = runEntry({ platform });+ expect(result.status, result.stderr).toBe(0);+ expect(result.stderr).toBe('');+ expect(calls.map(call => call.args.slice(platform === 'win32' ? 1 : 0))).toEqual([+ ['run', 'typecheck'], ['test'], ['run', 'test:e2e:typecheck'], ['run', 'test:e2e'],+ ['run', 'test:platform-native'], ['run', 'smoke'],+ ]);+ for (const call of calls) {+ expect(call).toMatchObject({ command: platform === 'win32' ? node : 'npm', cwd: root, stdio: 'inherit', sentinel: 'preserved' });+ expect(call.shell).toBeUndefined();+ if (platform === 'win32') expect(call.args[0]).toBe(preload);+ }+ expect(reports).toHaveLength(1);+ expect(result.stdout).toContain(path.join(root, 'test-results', reports[0].file));+ expect(reports[0].value.stages.map((stage: any) => stage.script)).toEqual(laneOrder);+ for (const stage of reports[0].value.stages) {+ expect(stage).toMatchObject({ exitCode: 0, signal: null, execution: 'completed' });+ expect(Number.isInteger(stage.durationMs)).toBe(true);+ expect(stage.durationMs).toBeGreaterThanOrEqual(0);+ }+ expect(reports[0].value.logReview).toContain('each owning runner report');+ });++ it.each([+ ['--checkpoint-dir', 'checkpoint with spaces', '--batch-count', '4', '--resume'],+ ['--checkpoint-dir=checkpoint with spaces', '--batch-count=4', '--resume'],+ ])('forwards checkpoint options to the JS/Python suite owner and E2E only (%j)', (...args) => {+ const { root, result, calls } = runEntry({ platform: 'linux', args });+ expect(result.status, result.stderr).toBe(0);+ const forwarded = ['--', `--checkpoint-dir=${path.join(root, 'checkpoint with spaces')}`, '--batch-count=4', '--resume'];+ expect(calls.map(call => call.args)).toEqual([+ ['run', 'typecheck'], ['test', ...forwarded], ['run', 'test:e2e:typecheck'],+ ['run', 'test:e2e', ...forwarded], ['run', 'test:platform-native'], ['run', 'smoke'],+ ]);+ // No separate Python invocation: npm test owns builtin Python and its+ // checkpoint forwarding (covered by test-checkpoint.test.ts's suite case).+ expect(calls).toHaveLength(6);+ });++ it('uses the owning checkpoint default without enabling resume implicitly', () => {+ const { root, result, calls } = runEntry({ platform: 'linux', args: ['--checkpoint-dir=checkpoint'] });+ expect(result.status, result.stderr).toBe(0);+ expect(calls[1].args).toEqual(['test', '--', `--checkpoint-dir=${path.join(root, 'checkpoint')}`, '--batch-count=8']);+ expect(calls[3].args).toEqual(['run', 'test:e2e', '--', `--checkpoint-dir=${path.join(root, 'checkpoint')}`, '--batch-count=8']);+ });++ it.each(laneOrder)('retains failure and never starts later lanes after %s fails', fail => {+ const { result, calls, reports } = runEntry({ fail });+ expect(result.status).toBe(7);+ expect(result.stderr).toBe('');+ const expected = laneOrder.slice(0, laneOrder.indexOf(fail) + 1);+ expect(calls).toHaveLength(expected.length);+ expect(reports[0].value.stages.map((stage: any) => stage.script)).toEqual(expected);+ expect(reports[0].value.stages.at(-1)).toMatchObject({ script: fail, exitCode: 7 });+ });++ it.each(['spawn', 'signal'])('records %s failure without starting the next native lane', failure => {+ const { result, calls, reports } = runEntry({ fail: 'test:e2e', failure });+ expect(result.status).toBe(1);+ expect(calls).toHaveLength(4);+ expect(reports[0].value.stages.at(-1)).toMatchObject({ script: 'test:e2e', exitCode: null,+ execution: failure === 'spawn' ? 'unavailable' : 'interrupted', signal: failure === 'signal' ? 'SIGTERM' : null });+ if (failure === 'spawn') expect(result.stderr).toContain('controlled spawn failure');+ else expect(result.stderr).toBe('');+ });++ it.each([+ ['--resume'], ['--batch-count=4'], ['--checkpoint-dir'], ['--checkpoint-dir='],+ ['--checkpoint-dir=checkpoint', '--batch-count=0'], ['--checkpoint-dir=checkpoint', '--batch-count=33'],+ ['--coverage'], ['--include-ios'], ['--checkpoint-dir=checkpoint', '--watch'],+ ])('rejects invalid or scope-changing arguments before dispatch (%j)', (...args) => {+ const { result, calls, reports } = runEntry({ args });+ expect(result.status).toBe(2);+ expect(result.stderr).toContain('[non-model]');+ expect(calls).toEqual([]);+ expect(reports).toEqual([]);+ });+});diff --git a/test/main/quality/source-map-literals.test.ts b/test/main/quality/source-map-literals.test.tsnew file mode 100644index 00000000..105ef6ab--- /dev/null+++ b/test/main/quality/source-map-literals.test.ts@@ -0,0 +1,40 @@+import { createRequire } from 'node:module';+import fs from 'node:fs';+import path from 'node:path';+import { describe, expect, it } from 'vitest';++const require = createRequire(import.meta.url);+const { fromSource, install } = require('../../helpers/source-map-literals.cjs');+const map = { version: 3, sources: ['original.ts'], names: [], mappings: 'AAAA' };+const comment = '//# sourceMappingURL=data:application/json;base64,';++describe('dependency error stack source maps', () => {+ it('preserves real inline maps and rejects malformed real comments', () => {+ expect(fromSource(comment + Buffer.from(JSON.stringify(map)).toString('base64')).toObject()).toEqual(map);+ expect(() => fromSource(comment + 'broken')).toThrow(SyntaxError);+ });++ it('ignores only string-contained false matches and keeps an earlier real map', () => {+ const template = 'const text = `\n' + comment + '${value}\n`;';+ expect(fromSource(template)).toBeNull();+ expect(fromSource(comment + Buffer.from(JSON.stringify(map)).toString('base64') + '\n' + template).toObject()).toEqual(map);+ expect(() => fromSource(comment + 'broken\n' + template)).toThrow(SyntaxError);+ });++ it('uses the repaired parser through Vitest while preserving genuine map errors', async () => {+ install();+ const { extractSourcemapFromFile } = await import('@vitest/utils/source-map/node');+ const template = 'const text = `\n' + comment + '${value}\n`;';+ // Vitest's external-file fallback returns an empty map for a missing filename.+ expect(extractSourcemapFromFile(template, '/dependency.js')).toEqual({ map: {} });+ expect(extractSourcemapFromFile(comment + Buffer.from(JSON.stringify(map)).toString('base64'), '/dependency.js')).toEqual({ map });+ expect(() => extractSourcemapFromFile(comment + 'broken', '/dependency.js')).toThrow(SyntaxError);+ });++ it('can report errors originating in both installed tsx register modules', () => {+ const directory = path.dirname(require.resolve('tsx/package.json'));+ const files = fs.readdirSync(path.join(directory, 'dist')).filter(file => /^register-.*\.(cjs|mjs)$/.test(file));+ expect(files.length).toBeGreaterThanOrEqual(2);+ for (const file of files) expect(() => fromSource(fs.readFileSync(path.join(directory, 'dist', file), 'utf8'))).not.toThrow();+ });+});diff --git a/test/main/quality/test-checkpoint.test.ts b/test/main/quality/test-checkpoint.test.tsnew file mode 100644index 00000000..ace70a28--- /dev/null+++ b/test/main/quality/test-checkpoint.test.ts@@ -0,0 +1,237 @@+import fs from 'node:fs';+import os from 'node:os';+import path from 'node:path';+import { spawn, spawnSync } from 'node:child_process';+import { afterEach, expect, it } from 'vitest';+import { createStderrReceipt, combineStderrReceipts } from '../../../scripts/test-stderr-receipt.mjs';++const pcRoot = path.resolve(__dirname, '../../..');+const node = process.env.ORKAS_TEST_NODE || process.execPath;+const roots: string[] = [];+afterEach(() => roots.splice(0).forEach(root => fs.rmSync(root, { recursive: true, force: true })));++it('attributes the exact stderr bytes across chunk boundaries and concurrent worker line ordering', () => {+ const first = createStderrReceipt();+ first.ingest('progress α\n');+ const second = createStderrReceipt();+ second.ingest('recoverable\n');+ const processOutput = createStderrReceipt();+ const bytes = Buffer.from('recoverable\nprogress α\n');+ for (const byte of bytes) processOutput.ingest(Buffer.from([byte]));+ const combined = combineStderrReceipts([first.finish(), second.finish()]);+ expect(combined).toEqual(processOutput.finish());+ expect(combined).toMatchObject({ valid: true, bytes: 24, lines: 2 });+});++it('rejects missing, duplicated, changed or oversized stderr evidence', () => {+ const expected = createStderrReceipt();+ expected.ingest('a\nb\n');+ const reference = expected.finish();+ for (const text of ['a\n', 'a\nb\nb\n', 'a\nc\n', 'a\nb']) {+ const changed = createStderrReceipt();+ changed.ingest(text);+ expect(changed.finish()).not.toEqual(reference);+ }+ const oversized = createStderrReceipt();+ oversized.ingest(Buffer.alloc(1024 * 1024 + 1));+ expect(oversized.finish().valid).toBe(false);+ expect(combineStderrReceipts([undefined]).valid).toBe(false);+});++function fixture() {+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-checkpoint-'));+ roots.push(root);+ const write = (file: string, content: string) => {+ fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true });+ fs.writeFileSync(path.join(root, file), content);+ };+ write('.gitignore', 'node_modules/\ntest-results/\n');+ write('package.json', '{"type":"module"}');+ for (const script of ['run-tests.mjs', 'test-runtime-env.mjs', 'check-installed-dependencies.mjs', 'test-checkpoint.mjs', 'test-stderr-receipt.mjs']) {+ const source = path.join(pcRoot, 'scripts', script);+ if (fs.existsSync(source)) write(`scripts/${script}`, fs.readFileSync(source, 'utf8'));+ }+ write('node_modules/electron/index.js', `module.exports = ${JSON.stringify(node)};`);+ write('scripts/run-python-tests.mjs', "import fs from 'node:fs'; fs.writeFileSync('test-results/resources-ran', '1');");+ write('test/a.test.ts', 'a');+ write('test/data/b.test.ts', 'b');+ write('test/build/c.test.ts', 'c');+ // Only the test engine is a controlled dependency. Real launcher, discovery+ // subprocess, source snapshot, report validation and persisted recovery run.+ write('node_modules/vitest/vitest.mjs', `+ import fs from 'node:fs';+ import path from 'node:path';+ const args = process.argv.slice(2);+ const files = fs.readdirSync('test', { recursive: true }).filter(f => f.endsWith('.test.ts')).sort().map(f => 'test/' + f.replaceAll('\\\\', '/'));+ if (args[0] === 'list') {+ fs.writeFileSync(args.find(a => a.startsWith('--json=')).slice(7), JSON.stringify(files.map(f => ({file:path.resolve(f)}))));+ } else {+ const shard = args.find(a => a.startsWith('--shard='))?.slice(8).split('/').map(Number) || [1,1];+ const selected = files.filter((_, i) => i % shard[1] === shard[0] - 1);+ const report = process.env.ORKAS_TEST_BATCH_REPORT;+ const logFailure = process.env.FIXTURE_LOG_GAP === '1' && selected.includes('test/data/b.test.ts');+ const failure = (selected.includes('test/data/b.test.ts') && !fs.existsSync('test-results/failed-once')) || selected.some(file => fs.readFileSync(file, 'utf8') === 'FAIL');+ fs.mkdirSync('test-results', {recursive:true});+ if (failure) fs.writeFileSync('test-results/failed-once', '1');+ fs.appendFileSync('test-results/executed.jsonl', JSON.stringify(selected) + '\\n');+ if (report && !(process.env.FIXTURE_NO_REPORT === '1' && selected.includes('test/data/b.test.ts'))) fs.writeFileSync(report, JSON.stringify({schema:1, reason:failure?'failed':'passed', unhandledErrors:0,+ stderr:{valid:true,bytes:0,lines:0,digest:'0'.repeat(64)},+ inventory:selected, files:selected.map(file => ({file,state:failure?'failed':'passed',passed:failure?0:1,failed:failure?1:0,skipped:0,durationMs:20})),+ logs:{status:logFailure?'FAIL':'WARN',files:selected.length,cases:selected.length,lines:1,unexpected:0,unmetExpectations:0,coverageGaps:logFailure?['missing tap']:[],signatures:[]}+ }));+ console.log('engine stdout');+ if (failure || process.env.FIXTURE_STDERR === '1') console.error('engine stderr retained for inspection');+ if (process.env.FIXTURE_HOLD === '1' && selected.includes('test/data/b.test.ts') && !fs.existsSync('test-results/hold-ready')) {+ fs.writeFileSync('test-results/hold-ready', '1');+ setTimeout(() => {}, 30000);+ }+ process.exitCode = failure ? 1 : 0;+ }+ `);+ const git = spawnSync('git', ['init', '-q'], { cwd: root, encoding: 'utf8' });+ expect(git.status, git.stderr).toBe(0);+ const run = (resume = false, extraEnv: Record<string, string> = {}, suite = false) => spawnSync(node, [+ 'scripts/run-tests.mjs', suite ? 'suite' : 'run', '--checkpoint-dir=test-results/checkpoint', '--batch-count=3', ...(resume ? ['--resume'] : []),+ ], { cwd: root, encoding: 'utf8', timeout: 30_000,+ env: { ...process.env, ...extraEnv, ORKAS_TEST_BATCH_REPORT: '' } });+ const executed = () => fs.readFileSync(path.join(root, 'test-results/executed.jsonl'), 'utf8').trim().split('\n').map(line => JSON.parse(line));+ return { root, write, run, executed };+}++it('resumes only completed, log-certified batches and keeps the original failure and both log streams', () => {+ const f = fixture();+ const first = f.run();+ expect(first.status, first.stderr).toBe(1);+ expect(f.executed()).toEqual([['test/a.test.ts'], ['test/data/b.test.ts'], ['test/build/c.test.ts']].sort((a,b) => a[0].localeCompare(b[0])));+ const second = f.run(true);+ expect(second.status, second.stderr).toBe(0);+ expect(f.executed()).toHaveLength(4);+ expect(f.executed().at(-1)).toEqual(['test/data/b.test.ts']);+ expect(second.stdout).toContain('reused=2');+ expect(second.stdout).toContain('WARN');+ const dir = path.join(f.root, 'test-results/checkpoint');+ const artifacts = fs.readdirSync(dir, { recursive: true }).map(String);+ const stderrLogs = artifacts.filter(file => file.endsWith('stderr.log'));+ expect(stderrLogs).toHaveLength(6); // Two discoveries and four batches.+ expect(stderrLogs.some(file => fs.readFileSync(path.join(dir,file), 'utf8').includes('engine stderr retained'))).toBe(true);+ const attempts = artifacts.filter(file => file.endsWith('attempt.json')).map(file => JSON.parse(fs.readFileSync(path.join(dir,file), 'utf8')));+ expect(attempts.some(item => item.exitCode === 1)).toBe(true);+});++it('retains the builtin Python suite after a successful checkpoint resume through npm test', () => {+ const f = fixture();+ const marker = path.join(f.root, 'test-results/resources-ran');+ expect(f.run(false, {}, true).status).toBe(1);+ expect(fs.existsSync(marker)).toBe(false);+ expect(f.run(true, {}, true).status).toBe(0);+ expect(fs.existsSync(marker)).toBe(true);+});++it.each(['--coverage', '--watch', '--reporter=json', '--shard=1/2', '--batch-count=0'])(+ 'rejects unsupported checkpoint selection %s before dispatch', flag => {+ const f = fixture();+ const result = spawnSync(node, ['scripts/run-tests.mjs', 'run', '--checkpoint-dir=test-results/checkpoint', flag], {+ cwd: f.root, env: process.env, encoding: 'utf8', timeout: 10_000,+ });+ expect(result.status).toBe(2);+ expect(fs.existsSync(path.join(f.root, 'test-results/executed.jsonl'))).toBe(false);+ },+);++it.each(['--resume', '--batch-count=8'])('rejects %s without a checkpoint directory before dispatch', flag => {+ const f = fixture();+ const result = spawnSync(node, ['scripts/run-tests.mjs', 'run', flag], {+ cwd: f.root, env: process.env, encoding: 'utf8', timeout: 10_000,+ });+ expect(result.status).toBe(2);+ expect(fs.existsSync(path.join(f.root, 'test-results/executed.jsonl'))).toBe(false);+});++it('invalidates recovery for new generic directories, source, installed dependency and environment changes', () => {+ const f = fixture();+ f.run();+ expect(f.run(true).status).toBe(0);+ for (const [file, value] of [+ ['test/new/data/new.test.ts', 'new'],+ ['test/a.test.ts', 'changed'],+ ['node_modules/extra/index.js', 'changed dependency'],+ ]) {+ const before = f.executed().length;+ f.write(file, value);+ const result = f.run(true);+ expect(result.status, result.stderr).toBe(0);+ expect(f.executed().length - before).toBe(3);+ }+ const before = f.executed().length;+ expect(f.run(true, { FIXTURE_CHANGED_ENV: '1' }).status).toBe(0);+ expect(f.executed().length - before).toBe(3);+ expect(f.executed().flat()).toContain('test/new/data/new.test.ts');+});++it('reruns a batch whose log artifact was lost instead of inheriting its passing exit code', () => {+ const f = fixture();+ f.run();+ expect(f.run(true).status).toBe(0);+ const dir = path.join(f.root, 'test-results/checkpoint');+ const state = JSON.parse(fs.readFileSync(path.join(dir, 'checkpoint.json'), 'utf8'));+ fs.unlinkSync(path.join(dir, state.completed['1'].stdout));+ const before = f.executed().length;+ expect(f.run(true).status).toBe(0);+ expect(f.executed().length - before).toBe(1);+});++it.each(['FIXTURE_LOG_GAP', 'FIXTURE_NO_REPORT'])('never certifies missing log evidence (%s) even when the engine exits zero', key => {+ const f = fixture();+ const env = { [key]: '1' };+ f.run(false, env);+ const result = f.run(true, env);+ expect(result.status).toBe(1);+ expect(result.stdout).toContain('reused=2');+ expect(f.executed()).toHaveLength(4);+ expect(result.stdout).toContain('log analysis');+});++it('fails an otherwise green engine for unattributed process stderr', () => {+ const f = fixture();+ f.run(false, { FIXTURE_STDERR: '1' });+ const result = f.run(true, { FIXTURE_STDERR: '1' });+ expect(result.status).toBe(1);+ expect(result.stdout).toContain('reused=0');+ expect(result.stdout).toContain('unclassified stderr');+});++it('discovers a newly added failing file after a previously complete run', () => {+ const f = fixture();+ f.run();+ expect(f.run(true).status).toBe(0);+ f.write('test/new/data/new.test.ts', 'FAIL');+ expect(f.run(true).status).toBe(1);+ expect(f.executed().flat()).toContain('test/new/data/new.test.ts');+});++it.skipIf(process.platform === 'win32')('cancels the active batch, releases the lock and resumes only terminal evidence', async () => {+ const f = fixture();+ const child = spawn(node, ['scripts/run-tests.mjs', 'run', '--checkpoint-dir=test-results/checkpoint', '--batch-count=3'], {+ cwd: f.root, stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, FIXTURE_HOLD: '1', ORKAS_TEST_BATCH_REPORT: '' },+ });+ let stderr = '';+ child.stdout.resume();+ child.stderr.on('data', data => { stderr += data.toString(); });+ const closed = new Promise(resolve => child.once('close', (code, signal) => resolve({ code, signal })));+ try {+ const ready = path.join(f.root, 'test-results/hold-ready');+ const deadline = Date.now() + 10_000;+ while (!fs.existsSync(ready) && Date.now() < deadline) await new Promise(resolve => setTimeout(resolve, 20));+ expect(fs.existsSync(ready), stderr).toBe(true);+ const conflicting = f.run(true, { FIXTURE_HOLD: '1' });+ expect(conflicting.status).toBe(1);+ expect(conflicting.stderr).toContain('Checkpoint is locked');+ child.kill('SIGTERM');+ expect(await closed).toEqual({ code: null, signal: 'SIGTERM' });+ expect(stderr).toBe('');+ const resumed = f.run(true, { FIXTURE_HOLD: '1' });+ expect(resumed.status, resumed.stderr).toBe(0);+ expect(resumed.stdout).toContain('reused=2');+ expect(f.executed()).toHaveLength(4);+ } finally { if (child.exitCode === null && child.signalCode === null) child.kill('SIGTERM'); await closed; }+});diff --git a/test/main/quality/test-log-analysis.test.ts b/test/main/quality/test-log-analysis.test.tsindex d7f7cefb..96b8b9d9 100644--- a/test/main/quality/test-log-analysis.test.ts+++ b/test/main/quality/test-log-analysis.test.ts@@ -3,6 +3,7 @@ import fs from 'node:fs';import os from 'node:os';import path from 'node:path';import { spawnSync } from 'node:child_process';+import { createStderrReceipt } from '../../../scripts/test-stderr-receipt.mjs';// Scenario: a maintainer runs any PC Vitest entry (`npm test`, `npm run// test:apps`, `run-tests.mjs suite <files>`). Emitted logs must be attributed@@ -18,14 +19,14 @@ const outDir = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-log-analysis-'));afterAll(() => fs.rmSync(outDir, { recursive: true, force: true }));-function runFixtures(files: string[], env: Record<string, string> = {}) {+function runFixtures(files: string[], env: Record<string, string> = {}, args: string[] = []) {const json = path.join(outDir, `${files.join('-').replace(/\W+/g, '_')}-${Object.keys(env).join('_')}.json`);const childEnv: NodeJS.ProcessEnv = Object.fromEntries(Object.entries(process.env).filter(([key]) => !key.startsWith('VITEST')));const result = spawnSync(node, ['scripts/run-tests.mjs', 'suite', '--config', `${fixtureDir}/vitest.config.ts`,- ...files.map(file => `${fixtureDir}/${file}`)], {+ ...files.map(file => `${fixtureDir}/${file}`), ...args], {cwd: pcRoot, encoding: 'utf8', timeout: 90_000,- env: { ...childEnv, ORKAS_LOG_FIXTURE_JSON: json, NO_COLOR: '1', ...env },+ env: { ...childEnv, ORKAS_LOG_FIXTURE_JSON: json, ORKAS_LOG_FIXTURE_BATCH_REPORT: `${json}.batch.json`, NO_COLOR: '1', ...env },});expect(result.error).toBeUndefined();// The reporter block starts at its status line and is the last stdout output.@@ -37,10 +38,32 @@ function runFixtures(files: string[], env: Record<string, string> = {}) {for (const file of report.testResults) {for (const item of file.assertionResults) cases.set(item.title, { status: item.status, failure: (item.failureMessages || []).join('\n') });}- return { status: result.status, summary: summary.join('\n'), head: summary[0] || '', cases, stderr: result.stderr };+ const outcomes = [...cases].map(([name, item]) => ({ name, status: item.status }));+ console.info('Nested test log analysis result', {+ fixtures: files, exit_code: result.status, analysis: summary[0] || 'missing',+ cases: outcomes.slice(0, 20), omitted_cases: Math.max(0, outcomes.length - 20),+ });+ const batchPath = `${json}.batch.json`;+ const batch = fs.existsSync(batchPath) ? JSON.parse(fs.readFileSync(batchPath, 'utf8')) : null;+ return { status: result.status, summary: summary.join('\n'), head: summary[0] || '', cases, stderr: result.stderr, batch };}describe('default Vitest log analysis', () => {+ it('keeps warning and error analysis when checkpoint mode suppresses successful console replay', () => {+ const expected = runFixtures(['expected.fixture.ts'], {}, ['--silent=passed-only']);+ expect(expected.status).toBe(0);+ expect(JSON.parse(expected.stderr.trim())).toEqual({ type: 'progress', source: 'video_edit', op: 'mix', status: 'running', elapsed_sec: 0.5 });+ const receipt = createStderrReceipt();+ receipt.ingest(expected.stderr);+ expect(expected.batch.stderr).toEqual(receipt.finish());+ expect(expected.batch.inventory).toEqual([`${fixtureDir}/expected.fixture.ts`]);+ expect(expected.batch.files[0]).toMatchObject({ passed: 4, failed: 0, skipped: 0 });+ expect(expected.batch.files[0].durationMs).toBeGreaterThanOrEqual(0);+ expect(expected.head).toMatch(/^\[test-log-analysis\] WARN files=1 cases=4 .* unexpected=0 expected=2 unmet=0 gaps=0$/);+ const unexpected = runFixtures(['unexpected.fixture.ts'], {}, ['--silent=passed-only']);+ expect(unexpected.status).toBe(1);+ expect(unexpected.head).toMatch(/^\[test-log-analysis\] FAIL .* unexpected=4 expected=0 unmet=1 gaps=0$/);+ }, 120_000);it('passes declared output but surfaces it, and surfaces an undeclared warning with its owning case', () => {const run = runFixtures(['expected.fixture.ts']);expect(run.status).toBe(0);@@ -86,4 +109,16 @@ describe('default Vitest log analysis', () => {expect(run.head).toMatch(/^\[test-log-analysis\] FAIL .* gaps=5$/);expect(run.summary).toContain(`coverage gap: ${fixtureDir}/expected.fixture.ts [setup/cleanup] (no analysis)`);}, 120_000);++ it('matches global and sticky declarations independently while still rejecting a different error', () => {+ const run = runFixtures(['stateful.fixture.ts']);+ expect(run.status).toBe(1); // The independent unexpected-error control must fail.+ expect(run.cases.get('matches every repeated global declaration')?.status).toBe('passed');+ expect(run.cases.get('matches every repeated sticky declaration')?.status).toBe('passed');+ expect(run.cases.get('matches every repeated frozen-global declaration')?.status).toBe('passed');+ expect(run.cases.get('matches every repeated frozen-sticky declaration')?.status).toBe('passed');+ expect(run.cases.get('still rejects a different error after repeated declared output')?.status).toBe('failed');+ expect(run.head).toMatch(/^\[test-log-analysis\] FAIL files=1 cases=5 .* warn=0 error=11 unexpected=1 expected=10 unmet=0 gaps=0$/);+ expect(run.summary).toContain('fixture different unexpected diagnostic');+ }, 120_000);});diff --git a/test/main/quality/test-runner-checkpoint.test.ts b/test/main/quality/test-runner-checkpoint.test.tsnew file mode 100644index 00000000..e7da53e9--- /dev/null+++ b/test/main/quality/test-runner-checkpoint.test.ts@@ -0,0 +1,264 @@+import fs from 'node:fs';+import os from 'node:os';+import path from 'node:path';+import ts from 'typescript';+import { spawn, spawnSync } from 'node:child_process';+import { afterEach, expect, it } from 'vitest';++const pcRoot = path.resolve(__dirname, '../../..');+const node = process.env.ORKAS_TEST_NODE || process.execPath;+const python = process.env.ORKAS_TEST_PYTHON!;+const roots: string[] = [];+afterEach(() => roots.splice(0).forEach(root => fs.rmSync(root, { recursive: true, force: true })));++function fixture(kind: 'python' | 'e2e') {+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-runner-resume-'));+ roots.push(root);+ const pc = root;+ const write = (file: string, value: string) => {+ fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true });+ fs.writeFileSync(path.join(root, file), value);+ };+ write('.gitignore', 'test-results/\nnode_modules/\n__pycache__/\n*.pyc\n.pytest_cache/\n');+ write('package.json', '{}');+ for (const name of fs.readdirSync(path.join(pcRoot, 'scripts'))) {+ if (['test-checkpoint.mjs', 'test-stderr-receipt.mjs', 'test-runtime-env.mjs', 'model-eval-log-policy.mjs',+ 'run-python-tests.mjs', 'run-e2e.cjs', 'runner-checkpoint.mjs', 'pytest_checkpoint.py', 'checkpoint-reporter.ts'].includes(name)) {+ write(`scripts/${name}`, fs.readFileSync(path.join(pcRoot, 'scripts', name), 'utf8'));+ }+ }+ write('test/e2e/fixtures/web-app-log-analysis.ts', fs.readFileSync(path.join(pcRoot, 'test/e2e/fixtures/web-app-log-analysis.ts'), 'utf8'));+ write('node_modules/playwright/package.json', '{}');+ write('node_modules/playwright/cli.js', `require(${JSON.stringify(path.join(path.dirname(require.resolve('playwright/package.json')), 'cli.js'))});`);+ write('node_modules/playwright/lib/program.js', `module.exports = require(${JSON.stringify(require.resolve('playwright/lib/program'))});`);+ write('playwright.config.ts', `export default { testDir: './test/e2e', workers: 1, fullyParallel: false, retries: 0, reporter: 'list' };`);+ const add = (name: string, fail = false, logs = '') => {+ if (kind === 'python') write(`tests/test_${name}.py`, `+from pathlib import Path+import pytest, sys, logging, warnings++def test_${name}():+ marker = Path('test-results/${name}')+ marker.parent.mkdir(exist_ok=True)+ previous = marker.exists()+ marker.write_text(str(int(marker.read_text()) + 1) if previous else '1')+ ${logs || 'pass'}+ assert ${fail ? 'previous' : 'True'}+`);+ else write(`test/e2e/${name}_e2e_fixture.spec.ts`, `+import { test, expect } from ${JSON.stringify(require.resolve('@playwright/test'))};+import fs from 'node:fs';+test('${name}', async () => {+ const marker = 'test-results/${name}';+ fs.mkdirSync('test-results', { recursive: true });+ const previous = fs.existsSync(marker);+ fs.writeFileSync(marker, String(previous ? Number(fs.readFileSync(marker)) + 1 : 1));+ ${logs}+ expect(${fail ? 'previous' : 'true'}).toBe(true);+});+`);+ };+ add('a'); add('b', true); add('c');+ expect(spawnSync('git', ['init', '-q'], { cwd: root }).status).toBe(0);+ const command = (resume = false, extra: string[] = []) => [+ kind === 'python' ? 'scripts/run-python-tests.mjs' : 'scripts/run-e2e.cjs',+ ...(kind === 'python' ? ['tests', '-q'] : ['test', '--config', 'playwright.config.ts']),+ '--checkpoint-dir=test-results/checkpoints', '--batch-count=3', ...(resume ? ['--resume'] : []), ...extra,+ ];+ const env = { ...process.env, ORKAS_TEST_PYTHON: python, ORKAS_TEST_BATCH_REPORT: '', NO_COLOR: '1', FORCE_COLOR: '0' };+ const run = (resume = false, extra: string[] = [], extraEnv = {}) => spawnSync(node, command(resume, extra),+ { cwd: pc, encoding: 'utf8', timeout: 60_000, env: { ...env, ...extraEnv } });+ const start = () => spawn(node, command(), { cwd: pc, env, stdio: ['ignore', 'pipe', 'pipe'] });+ const count = (name: string) => Number(fs.readFileSync(path.join(pc, 'test-results', name), 'utf8'));+ const stateFile = () => fs.readdirSync(path.join(pc, 'test-results/checkpoints'), { recursive: true })+ .map(String).find(file => file.endsWith('checkpoint.json'))!;+ return { root, pc, write, add, run, start, count, stateFile };+}++it.each(['python', 'e2e'] as const)('%s resumes failed work, preserves evidence and discovers new cases', kind => {+ const f = fixture(kind);+ const first = f.run();+ expect(first.status, first.stdout + first.stderr).toBe(1);+ expect([f.count('a'), f.count('b'), f.count('c')]).toEqual([1, 1, 1]);+ const second = f.run(true);+ expect(second.status, second.stdout + second.stderr).toBe(0);+ expect(second.stdout).toContain('reused=2');+ expect([f.count('a'), f.count('b'), f.count('c')]).toEqual([1, 2, 1]);+ const base = path.join(f.pc, 'test-results/checkpoints', path.dirname(f.stateFile()));+ const state = JSON.parse(fs.readFileSync(path.join(base, 'checkpoint.json'), 'utf8'));+ fs.unlinkSync(path.join(base, state.completed['1'].stdout));+ expect(f.run(true).status).toBe(0);+ expect(f.count('a') + f.count('b') + f.count('c')).toBe(5);+ const refreshed = JSON.parse(fs.readFileSync(path.join(base, 'checkpoint.json'), 'utf8'));+ const entry = refreshed.completed['1'];+ const report = JSON.parse(fs.readFileSync(path.join(base, entry.report), 'utf8'));+ fs.appendFileSync(path.join(base, report.artifacts[0].file), 'corrupted');+ const corrupt = f.run(true);+ expect(corrupt.status, corrupt.stdout + corrupt.stderr).toBe(0);+ expect(corrupt.stdout).toContain('reused=2');+ f.add('new', true);+ const changed = f.run(true);+ expect(changed.status, changed.stdout + changed.stderr).toBe(1);+ expect(f.count('new')).toBe(1);+ expect(changed.stdout).toContain('reused=0');+ const attempts = fs.readdirSync(base, { recursive: true }).map(String).filter(file => file.endsWith('attempt.json'));+ expect(attempts.some(file => JSON.parse(fs.readFileSync(path.join(base, file), 'utf8')).exitCode === 1)).toBe(true);+}, 120_000);++it.each(['python', 'e2e'] as const)('%s rejects unclassified stderr and surfaces structured warnings', kind => {+ const f = fixture(kind);+ f.add('b', false, kind === 'python' ? "sys.stderr.write('unclassified probe\\n')" : "process.stderr.write('unclassified probe\\n');");+ const invalid = f.run();+ expect(invalid.status, invalid.stdout + invalid.stderr).toBe(1);+ expect(invalid.stdout).toContain('log analysis FAIL');+ f.add('b', false, kind === 'python' ? "logging.error('structured failure')" : "process.stdout.write('[00:00:00.000] [error] [(probe)] structured failure\\n');");+ expect(f.run(true).status).toBe(1);+ f.add('b', false, kind === 'python' ? "warnings.warn('probe warning')" : "process.stdout.write('warning: probe warning\\n');");+ const warned = f.run(true);+ expect(warned.status, warned.stdout + warned.stderr).toBe(0);+ expect(warned.stdout).toContain('log analysis WARN');+}, 120_000);+++it.each(['python', 'e2e'] as const)('%s preserves skipped cases and whole files with uneven case counts', kind => {+ const f = fixture(kind);+ f.add('b');+ if (kind === 'python') f.write('tests/test_a.py', `+import pytest[email protected]('number', range(12))+def test_many(number): assert number >= 0[email protected](reason='platform fixture')+def test_skip(): assert False+`);+ else f.write('test/e2e/a_e2e_fixture.spec.ts', `+import {test, expect} from ${JSON.stringify(require.resolve('@playwright/test'))};+for (let i = 0; i < 12; i++) test('case ' + i, () => expect(i).toBeGreaterThanOrEqual(0));+test.skip('platform fixture', () => expect(false).toBe(true));+`);+ const initial = f.run();+ expect(initial.status, initial.stdout + initial.stderr).toBe(0);+ const resumed = f.run(true);+ expect(resumed.status, resumed.stdout + resumed.stderr).toBe(0);+ expect(resumed.stdout).toContain('reused=3');+ const base = path.join(f.pc, 'test-results/checkpoints', path.dirname(f.stateFile()));+ const state = JSON.parse(fs.readFileSync(path.join(base, 'checkpoint.json'), 'utf8'));+ const results = Object.values(state.completed).flatMap((entry: any) => JSON.parse(fs.readFileSync(path.join(base, entry.report), 'utf8')).files);+ expect(results).toHaveLength(15);+ expect(results.filter((file: any) => file.skipped === 1)).toHaveLength(1);+}, 120_000);++it('invalidates Python results when an external import dependency changes', () => {+ const f = fixture('python');+ const library = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-external-python-'));+ roots.push(library);+ const source = path.join(library, 'checkpoint_dependency.py');+ fs.writeFileSync(source, 'value = 1\n');+ f.add('b', false, 'import checkpoint_dependency; assert checkpoint_dependency.value == 1');+ const env = { PYTHONPATH: library };+ const first = f.run(false, [], env);+ expect(first.status, first.stdout + first.stderr).toBe(0);+ expect(f.run(true, [], env).stdout).toContain('reused=3');+ fs.writeFileSync(source, 'value = 2\n');+ const changed = f.run(true, [], env);+ expect(changed.status, changed.stdout + changed.stderr).toBe(1);+ expect(changed.stdout).toContain('reused=0');+}, 120_000);++it.skipIf(process.platform === 'win32').each(['python', 'e2e'] as const)(+ '%s cancellation releases its lock and only resumes terminal batches', async kind => {+ const f = fixture(kind);+ f.add('b', false, kind === 'python'+ ? "if not Path('test-results/hold-ready').exists(): Path('test-results/hold-ready').write_text('1'); __import__('time').sleep(30)"+ : "if (!fs.existsSync('test-results/hold-ready')) { fs.writeFileSync('test-results/hold-ready', '1'); await new Promise(resolve => setTimeout(resolve, 30000)); }");+ const child = f.start();+ let stderr = '';+ child.stdout.resume();+ child.stderr.on('data', data => { stderr += data.toString(); });+ const closed = new Promise(resolve => child.once('close', (code, signal) => resolve({ code, signal })));+ try {+ const deadline = Date.now() + 30_000;+ while (!fs.existsSync(path.join(f.pc, 'test-results/hold-ready')) && Date.now() < deadline) {+ await new Promise(resolve => setTimeout(resolve, 50));+ }+ expect(fs.existsSync(path.join(f.pc, 'test-results/hold-ready')), stderr).toBe(true);+ const conflict = f.run(true);+ expect(conflict.status).toBe(1);+ expect(conflict.stderr).toContain('Checkpoint is locked');+ child.kill('SIGTERM');+ expect(await closed).toEqual({ code: null, signal: 'SIGTERM' });+ const resumed = f.run(true);+ expect(resumed.status, resumed.stdout + resumed.stderr).toBe(0);+ expect(resumed.stdout).toContain('reused=1');+ expect([f.count('a'), f.count('b'), f.count('c')]).toEqual([1, 2, 1]);+ } finally { if (child.exitCode === null && child.signalCode === null) child.kill('SIGTERM'); await closed; }+ }, 120_000,+);+++it('requires every Electron log attachment and retains independently verified recovery evidence', () => {+ const f = fixture('e2e');+ f.add('b', false, `await test.info().attach('electron-log-required', { body: Buffer.from('[]') });`);+ const missing = f.run();+ expect(missing.status, missing.stdout + missing.stderr).toBe(1);+ f.add('b', false, `+ await test.info().attach('electron-log-required', { body: Buffer.from(JSON.stringify([{ start: 1, end: 1, diagnosticIndex: 0 }])) });+ const log = test.info().outputPath('electron.log');+ fs.writeFileSync(log, '[renderer:pageerror] injected preview crash\\n');+ await test.info().attach('electron-logs', { path: log });+ `);+ const recovered = f.run(true);+ expect(recovered.status, recovered.stdout + recovered.stderr).toBe(0);+ expect(recovered.stdout).toContain('log analysis WARN');+ const resumed = f.run(true);+ expect(resumed.stdout).toContain('reused=3');+}, 120_000);+++// Run the actual dispose owner without importing Playwright or starting an app.+// A missing receipt must reject certification and still release every resource.+it.each(['electron-log-required', 'electron-logs'])(+ 'checkpoint %s attachment failures preserve cleanup and remain failures', async failedAttachment => {+ const file = ts.createSourceFile('orkas.ts', fs.readFileSync(+ path.join(pcRoot, 'test/e2e/fixtures/orkas.ts'), 'utf8'), ts.ScriptTarget.Latest, true);+ const owner = file.statements.find((statement): statement is ts.ClassDeclaration =>+ ts.isClassDeclaration(statement) && statement.name?.text === 'OrkasTestApp');+ if (!owner) throw new Error('Missing Electron fixture owner');+ const methods = owner.members.filter(member => member.name+ && ['dispose', 'recoveredDiagnosticRanges'].includes(member.name.getText(file)));+ const code = ts.transpileModule('class DisposeOwner {' + methods.map(method => method.getText(file)).join('\n') + '}', {+ compilerOptions: { target: ts.ScriptTarget.ES2022 },+ }).outputText;+ const Owner = new Function('process', 'expect', 'path', 'mkdirSync', 'writeFileSync', 'rmSync',+ 'unlinkSync', 'collectLogFiles', 'randomUUID', code + '; return DisposeOwner;')(+ { env: { ORKAS_TEST_BATCH_REPORT: 'fixture-report' } }, expect, path, fs.mkdirSync, fs.writeFileSync,+ fs.rmSync, fs.unlinkSync, () => '', () => 'snapshot');+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'orkas-checkpoint-dispose-'));+ roots.push(root);+ const appRoot = path.join(root, 'app');+ fs.mkdirSync(appRoot);+ fs.writeFileSync(path.join(appRoot, 'owned-file'), 'temporary fixture data');+ const released: string[] = [];+ const failure = new Error('injected checkpoint attachment failure');+ const fixture = Object.assign(new Owner(), {+ root: appRoot, page: null, apiResponses: [], expectedApiFailures: [], cliStub: false,+ rendererPageErrors: [], recoveredRendererErrors: new Set(), diagnostics: ['[fixture-cleanup] completed'], tracePaths: [],+ closeCurrentApp: async () => { released.push('app'); },+ closeStubServer: async () => { released.push('stub'); },+ testInfo: {+ status: 'passed', expectedStatus: 'passed', outputDir: path.join(root, 'output'),+ outputPath: (name: string) => path.join(root, 'output', name),+ attach: async (name: string) => {+ expect(released).toEqual(['app', 'stub']);+ if (name === failedAttachment) throw failure;+ },+ },+ });+ if (failedAttachment === 'electron-log-required') await expect(fixture.dispose()).rejects.toBe(failure);+ else await expect(fixture.dispose()).rejects.toMatchObject({+ message: 'Checkpoint Electron log capture failed', cause: failure,+ });+ expect(released).toEqual(['app', 'stub']);+ expect(fs.existsSync(appRoot)).toBe(false);+ expect(fs.existsSync(path.join(root, 'output', 'checkpoint-electron-snapshot.log'))).toBe(true);+ },+);diff --git a/test/main/util/test-runtime-env.test.ts b/test/main/util/test-runtime-env.test.tsindex ec50f84d..d4a6d2b9 100644--- a/test/main/util/test-runtime-env.test.ts+++ b/test/main/util/test-runtime-env.test.ts@@ -1,6 +1,8 @@import fs from 'node:fs';import os from 'node:os';import path from 'node:path';+import { spawnSync } from 'node:child_process';+import { pathToFileURL } from 'node:url';import { afterEach, describe, expect, it } from 'vitest';@@ -21,6 +23,25 @@ function temporaryRoot(): string {}describe('test runtime environment', () => {+ it.each(['win32', 'darwin', 'linux'])('resolves the flat checkout default Python instead of its parent for %s', platform => {+ const parent = fs.realpathSync(temporaryRoot());+ const root = path.join(parent, 'app');+ const suffix = platform === 'win32' ? ['venv', 'Scripts', 'python.exe'] : ['venv', 'bin', 'python'];+ for (const directory of [parent, root]) {+ const python = path.join(directory, ...suffix);+ fs.mkdirSync(path.dirname(python), { recursive: true });+ fs.writeFileSync(python, '');+ }+ const helper = path.join(root, 'scripts', 'test-runtime-env.mjs');+ fs.mkdirSync(path.dirname(helper), { recursive: true });+ fs.copyFileSync(path.resolve(__dirname, '../../../scripts/test-runtime-env.mjs'), helper);+ const result = spawnSync(process.env.ORKAS_TEST_NODE || process.execPath,+ ['--input-type=module', '-e', 'const { withTestPython } = await import(process.argv[1]); console.log(JSON.stringify(withTestPython({}, undefined, process.argv[2])));', pathToFileURL(helper).href, platform],+ { encoding: 'utf8', timeout: 10_000 });+ expect(result.status, result.stderr).toBe(0);+ expect(JSON.parse(result.stdout)).toEqual({ ORKAS_TEST_PYTHON: path.join(root, ...suffix), ORKAS_EVAL_PYTHON: path.join(root, ...suffix) });+ });+it.each(['win32', 'darwin', 'linux'])('uses the repository virtual environment for %s fixture subprocesses', platform => {const root = temporaryRoot();const python = platform === 'win32'diff --git a/vitest.config.ts b/vitest.config.tsindex 33546a5c..a194630b 100644--- a/vitest.config.ts+++ b/vitest.config.ts@@ -1,7 +1,10 @@import { defineConfig } from 'vitest/config';import { cpus } from 'node:os';+import { createRequire } from 'node:module';import { TestLogAnalysisReporter } from './test/log-analysis-reporter';+createRequire(import.meta.url)('./test/helpers/source-map-literals.cjs').install();+const logicalCpus = cpus().length || 1;// Test files exercise real Electron workers, shell helpers, Git, SQLite,// FFmpeg, and whisper.cpp. Four concurrent forks can exhaust the desktop