Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

ci: separate browser smoke from exact-head visual fidelity (#11687)

[._.]/3e220c8bff05
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6562370f46..a2c0fa8613 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -301,6 +301,7 @@ jobs:
- name: Run server tests
if: needs.impact.outputs.server_mode != 'skip'
env:
+ PORTOS_CI_BROWSER_OWNER: unit
CI_TEST_MODE: ${{ needs.impact.outputs.server_mode }}
CI_TEST_FILES: ${{ needs.impact.outputs.server_files }}
CI_TEST_SOURCES: ${{ needs.impact.outputs.server_sources }}
@@ -519,12 +520,9 @@ jobs:
run: node scripts/cancel-current-ci-run.js
database:
- # Also the host for the cross-workspace browser suites (issue #10312): the
- # server job installs no client workspace, so `server/**/*.browser.test.js`
- # skip there. This job already has ffmpeg and a runner with Chrome, so it
- # adds the client install and runs the files the planner selected with
- # PORTOS_REQUIRE_BROWSER_SUITES set, which turns a missing prerequisite into
- # a failure instead of a green skip. The DB steps run only when `db` is set.
+ # Owns the registered browser suites; the Linux unit job excludes that
+ # exact registry. PRs run smoke here and require exact-head fidelity evidence
+ # at the gate. Scheduled/release runs retain the complete browser profile.
name: DB tests
needs: impact
if: needs.impact.outputs.db == 'true' || needs.impact.outputs.browser_files != '[]'
@@ -710,6 +708,7 @@ jobs:
- name: Run cross-workspace browser suites
if: needs.impact.outputs.browser_files != '[]'
env:
+ PORTOS_BROWSER_TEST_PROFILE: ${{ github.event_name == 'pull_request' && github.base_ref != 'release' && inputs.full != true && 'smoke' || 'full' }}
CI_TEST_MODE: files
CI_TEST_FILES: ${{ needs.impact.outputs.browser_files }}
PORTOS_REQUIRE_BROWSER_SUITES: '1'
@@ -961,6 +960,7 @@ jobs:
permissions:
contents: read
actions: read
+ statuses: read
steps:
# The gate needs the repo only for scripts/ci-gate-report.js, which
# imports Node builtins only — no dependency install, no setup-node.
@@ -994,6 +994,15 @@ jobs:
CI_GATE_RESULT_LINUX_CONTAINMENT: ${{ needs.linux-containment.result }}
run: node scripts/ci-gate-report.js
+ - name: Require current browser fidelity evidence
+ if: github.event_name == 'pull_request' && github.base_ref != 'release' && inputs.full != true
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ CI_BROWSER_FILES: ${{ needs.impact.outputs.browser_files }}
+ CI_PULL_REQUEST: ${{ github.event.pull_request.number }}
+ CI_EXPECTED_HEAD: ${{ github.event.pull_request.head.sha }}
+ run: node scripts/verify-browser-fidelity.js
+
full-gate:
# A second check run published ONLY when the impact plan chose the complete
# suite. release.yml keys on this name to skip re-running CI on the push to
@@ -1012,7 +1021,7 @@ jobs:
# force-full path — nightly, the main -> release PR, release, an explicit
# full-CI request — keeps windows_mode == 'full', so the release
# optimization is unaffected.
- if: always() && needs.impact.outputs.full == 'true' && needs.impact.outputs.windows_mode == 'full'
+ if: always() && needs.impact.outputs.full == 'true' && needs.impact.outputs.windows_mode == 'full' && (github.event_name != 'pull_request' || github.base_ref == 'release' || inputs.full == true)
needs: [impact, gate]
runs-on: ubuntu-latest
timeout-minutes: 10
diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md
index a709889f24..05ad385791 100644
--- a/docs/CONTRIBUTING.md
+++ b/docs/CONTRIBUTING.md
@@ -89,6 +89,12 @@ See [VERSIONING.md](./VERSIONING.md) for full details.
Pregate proves only the stages it runs; it does not replace required CI or resource-dependent checks. A full-suite plan runs only the always-run guards unless `--full` is supplied; DB suites, Windows, client build, and boot smoke are reported but not run by pregate.
+Rendering PRs that select fidelity cases also run
+`node scripts/browser-fidelity.js --pr NUMBER` from the clean published head.
+The command runs the cases and publishes exact-head evidence using existing GitHub
+authentication; missing or stale evidence blocks CI Gate. See
+[Browser fidelity evidence](GITHUB_ACTIONS.md#browser-fidelity-evidence).
+
### Checkout-owned development caches
Vite dev mode stores reusable dependency data in `client/.cache/vite`. Vitest
diff --git a/docs/GITHUB_ACTIONS.md b/docs/GITHUB_ACTIONS.md
index 47d4223adc..023417513a 100644
--- a/docs/GITHUB_ACTIONS.md
+++ b/docs/GITHUB_ACTIONS.md
@@ -448,23 +448,18 @@ The selected work is split across parallel jobs:
`Cache client node_modules` with the client job, so on a hit it pays no install.
- **DB tests** — provisions only the isolated `portos_test` database and runs
the serial DB suite when database-sensitive files changed.
-- **Cross-workspace browser suites** (same `database` job) — the
- `BROWSER_SUITES` registered in `scripts/ci-test-plan.js` (`musicVideoProductionReview`,
- `musicVideoRichAuthoring`, `musicVideoSongRevision`) bundle real client
- components and drive them in Chrome. The `server` job installs no client
- workspace, so they skip there and would otherwise skip in every CI job (#10312:
- a `ProductionReviewPanel` prop change left all three harnesses blank behind a
- green CI). The planner emits `browser_files`: every registered suite on a full
- plan (a registry test fails when a suite using the gate is left out), otherwise the ones selected by a changed test or by the
- basename lookup of a changed client source in a harness `entry` string. When it
- is non-empty the `database` job also runs (even with `db` false), installs
- the client workspace (sharing `Cache client node_modules`) and runs those files
- with `PORTOS_REQUIRE_BROWSER_SUITES=1`, which makes
- `server/lib/browserSuiteGate.js` throw instead of skip when Chrome, ffmpeg or
- the client bundler is missing. Local runs without the prerequisites still skip.
- Limit: selection follows the basename named in a harness entry, so a change to
- a component the entry only imports transitively runs these suites on a full
- plan or when a suite file itself changes, not on a scoped PR.
+- **Cross-workspace browser suites** (same `database` job) — the exact
+ `BROWSER_SUITES` registry in `scripts/lib/browserSuites.js` owns these tests.
+ Linux server shards exclude that registry with `PORTOS_CI_BROWSER_OWNER=unit`;
+ other browser files, including `overlayTextService`, keep their existing owner.
+ The planner selects affected registered files and the browser job requires
+ Chrome, FFmpeg and bundler prerequisites instead of silently skipping them.
+ Ordinary PRs run the `smoke` profile: all existing assertions except the
+ explicitly tagged `[fidelity]` grading/authoring cases. Scheduled, release,
+ and dispatched full runs execute every case. Profiles do not change pixels,
+ render settings, tolerances, or cleanup assertions.
+ A PR selecting a fidelity-owning file also requires the exact-head receipt
+ described below; a smoke pass alone cannot admit it.
- **Windows server tests** — the same server selection, but only on full CI
(the `main` → `release` PR, nightly, release, workflow dispatch) or when a
Windows-sensitive surface changed (`.ps1` / `.cmd` spawn, PowerShell BOM,
@@ -478,10 +473,40 @@ The selected work is split across parallel jobs:
cancelled jobs when nothing actually failed — as far as the gate job itself
runs, which a run-wide cancel prevents. See "External cancellation and one
automatic retry" below.
-- **Full CI Gate** — published only when the plan chose the complete suite, and
- mirrors `CI Gate`'s result. This is the check the release workflow looks for;
+- **Full CI Gate** — published only when the plan chose the complete suite and
+ full Windows coverage, with the full browser profile executed in hosted CI.
+ Ordinary PR smoke runs cannot publish it, even with a local fidelity receipt.
+ It mirrors `CI Gate`'s result. This is the check the release workflow looks for;
see "Reusing the release PR's CI run" below.
+### Browser fidelity evidence
+
+After publishing a relevant PR, run `node scripts/browser-fidelity.js --pr NUMBER`
+from its clean checkout using existing `gh` authentication. This command runs the
+canonical fidelity cases automatically, retains a local receipt, and publishes
+an exact-commit status only after complete passing assertions and unchanged-source
+verification. It binds the PR head and that commit's source tree. It does not
+provision a runner, install credentials, or start PortOS.
+
+The required CI gate checks the current head and its source tree, profile digest,
+trusted repository-owner publisher and latest status. Missing, stale, skipped, failed or incomplete
+results block admission. Local receipt JSON is an audit record; the trusted
+publisher's status attests execution, not a cryptographic proof of test behavior.
+Keep credentials outside the test subprocess.
+
+If evidence finishes after CI Gate failed only for missing fidelity evidence,
+rerun that gate job on the unchanged run once its other required jobs have passed.
+A receipt cannot turn a cancelled or failed browser, Windows, or containment job
+green. A changed PR head requires new evidence; base-only movement does not. Hosted smoke tests the merge checkout; local fidelity proves the PR head only,
+not native integration of subsequent base changes. Scheduled/release full coverage
+remains the integration check for those fidelity cases. Default
+pregate is not a substitute for this command or required CI.
+
+Hardware playback/GPU throughput remains distinct from these deterministic visual
+assertions. Software WebGL results do not certify physical-GPU performance, and
+Mac-local results cannot replace Windows compatibility or Linux containment.
+The profile split does not explain or repair previously unidentified hosted stalls.
+
### Full-suite sharding
A full plan is the slow case: on the 4-vCPU public runners the client suite
diff --git a/scripts/browser-fidelity.js b/scripts/browser-fidelity.js
new file mode 100644
index 0000000000..7f2019c163
--- /dev/null
+++ b/scripts/browser-fidelity.js
@@ -0,0 +1,169 @@
+#!/usr/bin/env node
+
+import { execFileSync } from 'node:child_process';
+import { existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
+import { join, resolve } from 'node:path';
+import { createRequire } from 'node:module';
+import { fileURLToPath } from 'node:url';
+import { tmpdir, platform, arch } from 'node:os';
+import { runNpm, toRunnerPath } from './run-ci-tests.js';
+import { BROWSER_FIDELITY_FILES, browserTestProfileArgs, assertBrowserProfileExecuted } from './lib/browserTestProfiles.js';
+import { browserFidelityTarget, browserFidelityContext, browserReceiptDigest, browserFidelityActor } from './lib/browserFidelity.js';
+import { isDirectlyInvoked } from './lib/directInvocation.js';
+
+const repoRoot = fileURLToPath(new URL('../', import.meta.url));
+
+function gh(args, input) {
+ try { return execFileSync('gh', args, { encoding: 'utf8', input, timeout: 15000, maxBuffer: 2 * 1024 * 1024 }); }
+ catch { throw Error('GitHub CLI request failed; existing authentication/permissions are required.'); }
+}
+
+async function ghRequest(path, body) {
+ const args = ['api', path];
+ if (body) args.push('--method', 'POST', '--input', '-');
+ try { return JSON.parse(gh(args, body ? JSON.stringify(body) : undefined)); }
+ catch { throw Error('GitHub CLI evidence/status request failed.'); }
+}
+
+const gitRead = args => {
+ try { return execFileSync('git', args, { encoding: 'utf8', timeout: 15000, stdio: ['ignore', 'pipe', 'pipe'], cwd: repoRoot }).trim(); }
+ catch { throw Error('Git checkout identity unavailable.'); }
+};
+
+export function fidelityChildEnv(env, diagnostics) {
+ const child = Object.fromEntries(Object.entries(env).filter(([key]) => !/TOKEN|SECRET|PASSWORD|API_KEY|AUTH|COOKIE|CREDENTIAL/i.test(key)));
+ delete child.PORTOS_CI_BROWSER_OWNER;
+ return { ...child, PORTOS_CI_DIAGNOSTICS_DIR: diagnostics, PORTOS_REQUIRE_BROWSER_SUITES: '1',
+ PORTOS_BROWSER_TEST_PROFILE: 'fidelity', PORTOS_PREGATE_MAX_WORKERS: '1' };
+}
+
+async function fidelityToolchain(env) {
+ const require = createRequire(new URL('../server/package.json', import.meta.url));
+ const { chromium } = require('playwright-core');
+ const { findFfmpeg } = await import('../server/lib/ffmpeg.js');
+ const { _testChromeCaptureArgs } = await import('../server/services/htmlComposition/testBrowserCleanup.js');
+ const chromePath = [env.CHROME_PATH, chromium.executablePath(),
+ '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', '/usr/bin/google-chrome',
+ '/usr/bin/chromium', '/usr/bin/chromium-browser'].find(path => path && existsSync(path));
+ const ffmpegPath = await findFfmpeg();
+ if (!chromePath || !ffmpegPath) throw Error('Fidelity Chrome/FFmpeg toolchain unavailable.');
+ const version = (path, args) => {
+ const text = execFileSync(path, args, { encoding: 'utf8', timeout: 5000, maxBuffer: 65536,
+ stdio: ['ignore', 'pipe', 'ignore'], env: fidelityChildEnv(env, '') });
+ const line = text.split(/\r?\n/)[0].replace(/[^\x20-\x7E]/g, '').slice(0, 240);
+ if (!line) throw Error('Native tool version unavailable.');
+ return line;
+ };
+ return { chromePath, receipt: { node: process.version, os: platform(), arch: arch(),
+ chrome: version(chromePath, ['--version']), ffmpeg: version(ffmpegPath, ['-version']),
+ renderBackend: { kind: 'unknown', reason: 'Existing profile does not report runtime graphics backend.',
+ launchFlags: _testChromeCaptureArgs('receipt').filter(arg => arg.startsWith('--') && !arg.startsWith('--user-data-dir=')) } } };
+}
+
+function assertCheckout(git, target) {
+ if (git(['status', '--porcelain=v1', '--untracked-files=all'])) throw Error('Browser fidelity requires a clean checkout.');
+ if (git(['rev-parse', 'HEAD']) !== target.head) throw Error('Checkout must match the current PR head.');
+ if (git(['rev-parse', 'HEAD^{tree}']) !== target.tree) throw Error('Checkout tree differs from the current PR head tree.');
+}
+
+function readCompletedProfile(root) {
+ const attempts = readdirSync(root).filter(name => /^portos-vitest-[\w-]+$/.test(name));
+ if (attempts.length !== 1) throw Error('Expected one complete fidelity invocation, without retry.');
+ const directory = join(root, attempts[0]);
+ const read = (name, limit) => {
+ const path = join(directory, name);
+ const info = lstatSync(path);
+ if (!info.isFile() || info.isSymbolicLink() || info.size > limit) throw Error('Fidelity report is unavailable or exceeds its bound.');
+ return JSON.parse(readFileSync(path, 'utf8'));
+ };
+ const invocation = read('invocation.json', 100000);
+ const run = read('run.json', 160000);
+ if (invocation.completed !== true || invocation.status !== 0 || invocation.signal
+ || run.reason !== 'passed' || run.errorCount !== 0 || !Array.isArray(run.errors) || run.errors.length || run.teardownTimeout
+ || !Number.isSafeInteger(run.completedCount) || run.completedCount < BROWSER_FIDELITY_FILES.length
+ || !Array.isArray(run.activeModules) || run.activeModules.length) throw Error('Fidelity invocation/teardown did not complete cleanly.');
+ const report = read('failures.json', 32 * 1024 * 1024);
+ if (report.success !== true || Number(report.numFailedTests) !== 0 || Number(report.numFailedTestSuites) !== 0) {
+ throw Error('Fidelity structured result did not pass.');
+ }
+ return assertBrowserProfileExecuted('fidelity', report);
+}
+
+export async function publishBrowserFidelity({ pr, repository, receiptPath, env = process.env,
+ request = ghRequest, git = gitRead, run = runNpm, toolchain = fidelityToolchain, deadlineMs = 15 * 60_000,
+} = {}) {
+ // gh resolves the configured repository/host; never assume an upstream owner.
+ repository ??= JSON.parse(gh(['repo', 'view', '--json', 'nameWithOwner'])).nameWithOwner;
+ const target = await browserFidelityTarget(repository, pr, request);
+ assertCheckout(git, target);
+ const user = await request('user');
+ if (user?.login?.toLowerCase() !== browserFidelityActor(repository)) throw Error('Existing GitHub authentication is not a trusted fidelity publisher.');
+ const context = browserFidelityContext(target);
+ const root = mkdtempSync(join(tmpdir(), 'portos-browser-fidelity-'));
+ const diagnostics = join(root, 'diagnostics');
+ mkdirSync(diagnostics);
+ const output = receiptPath ? resolve(receiptPath) : join(root, 'receipt.json');
+ const statusPath = `repos/${repository}/statuses/${target.head}`;
+ const post = async (state, description) => {
+ const status = await request(statusPath, { state, context, description });
+ if (status?.state !== state || status?.context !== context
+ || status?.creator?.login?.toLowerCase() !== browserFidelityActor(repository)) throw Error('Fidelity status publication is unconfirmed.');
+ };
+ const controller = new AbortController();
+ const timer = setTimeout(() => controller.abort(), deadlineMs);
+ const abort = () => controller.abort();
+ process.once('SIGINT', abort);
+ process.once('SIGTERM', abort);
+ const startedAt = new Date().toISOString();
+ try {
+ await post('pending', 'Required local browser fidelity is running.');
+ const tools = await toolchain(env);
+ const childEnv = fidelityChildEnv(env, diagnostics);
+ childEnv.CHROME_PATH = tools.chromePath;
+ const result = await run('server', 'test:ci', [
+ ...BROWSER_FIDELITY_FILES.map(file => toRunnerPath('server', file)), ...browserTestProfileArgs('fidelity'),
+ ], { env: childEnv, signal: controller.signal });
+ controller.signal.throwIfAborted();
+ if (result.status !== 0 || result.error || result.interrupted) throw Error('Browser fidelity failed or owned cleanup is unverified.');
+ const executed = readCompletedProfile(diagnostics);
+ assertCheckout(git, target);
+ const current = await browserFidelityTarget(repository, pr, request);
+ if (browserFidelityContext(current) !== context) throw Error('PR head, head tree, or fidelity policy changed during execution.');
+ controller.signal.throwIfAborted();
+ const receipt = { version: 1, target, context, startedAt, completedAt: new Date().toISOString(),
+ toolchain: tools.receipt,
+ profile: { files: BROWSER_FIDELITY_FILES, args: browserTestProfileArgs('fidelity'), executed },
+ result: 'passed' };
+ // The receipt remains local. No cloud artifact or remotely verified log is
+ // implied by the status; its trusted publisher attests this completed run.
+ writeFileSync(output, `${JSON.stringify(receipt, null, 2)}\n`, { flag: 'wx', mode: 0o600 });
+ assertCheckout(git, target);
+ controller.signal.throwIfAborted();
+ await post('success', `receipt sha256=${browserReceiptDigest(receipt)}`);
+ return { receiptPath: output, diagnosticsPath: diagnostics, context, executed };
+ } catch {
+ await post('failure', 'Required browser fidelity failed, changed, or did not complete.').catch(() => {});
+ throw Object.assign(Error('Browser fidelity was not admitted; local diagnostics were retained.'), { diagnosticsPath: diagnostics });
+ } finally {
+ clearTimeout(timer);
+ process.removeListener('SIGINT', abort);
+ process.removeListener('SIGTERM', abort);
+ }
+}
+
+if (isDirectlyInvoked(import.meta.url)) {
+ const args = process.argv.slice(2);
+ const value = name => { const index = args.indexOf(name); return index < 0 ? undefined : args[index + 1]; };
+ try {
+ if (args.some((arg, index) => index % 2 === 0 && !['--pr', '--repo', '--receipt'].includes(arg))
+ || args.length % 2 || !/^\d+$/.test(value('--pr') ?? '')) throw Error('Invalid browser fidelity arguments.');
+ const result = await publishBrowserFidelity({ pr: Number(value('--pr')), repository: value('--repo'), receiptPath: value('--receipt') });
+ console.log(`✅ Browser fidelity receipt saved locally (${result.executed} executed tests); trusted status published.`);
+ console.log(`Receipt: ${result.receiptPath}`);
+ console.log(`Local diagnostics: ${result.diagnosticsPath}`);
+ } catch (error) {
+ console.error('❌ Browser fidelity was not admitted. Use --pr N on its clean current head with existing trusted gh authentication.');
+ if (error.diagnosticsPath) console.error(`Local diagnostics retained: ${error.diagnosticsPath}`);
+ process.exitCode = 1;
+ }
+}
diff --git a/scripts/browser-fidelity.test.js b/scripts/browser-fidelity.test.js
new file mode 100644
index 0000000000..820780ba32
--- /dev/null
+++ b/scripts/browser-fidelity.test.js
@@ -0,0 +1,121 @@
+import { mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { spawnSync } from 'node:child_process';
+import { afterEach, describe, expect, it, vi } from 'vitest';
+import { publishBrowserFidelity } from './browser-fidelity.js';
+import { BROWSER_FIDELITY_TESTS, BROWSER_FIDELITY_FILES } from './lib/browserTestProfiles.js';
+
+const head = 'a'.repeat(40), base = 'b'.repeat(40), merge = 'c'.repeat(40), tree = 'd'.repeat(40);
+const roots = [];
+afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
+
+function fixture({ outcome = 'passed', changed = false, dirty = false, wrongTree = false } = {}) {
+ const root = mkdtempSync(join(tmpdir(), 'browser-fidelity-contract-')); roots.push(root);
+ const posts = [];
+ let finished = false;
+ const request = vi.fn(async (path, body) => {
+ if (body) { posts.push(body); return { state: body.state, context: body.context, creator: { login: 'fixture' } }; }
+ if (path === 'user') return { login: 'fixture' };
+ if (path.endsWith('/pulls/12')) return { state: 'open', head: { sha: head, repo: { full_name: 'fixture/PortOS' } },
+ base: { sha: base, repo: { full_name: 'fixture/PortOS' } }, merge_commit_sha: merge };
+ if (path.endsWith(`/git/commits/${head}`)) return { sha: head, tree: { sha: tree }, parents: [{ sha: base }] };
+ throw Error('Unexpected synthetic API request');
+ });
+ const git = vi.fn(args => args[0] === 'status' ? (dirty ? ' M changed.js' : '')
+ : args.at(-1) === 'HEAD' ? (finished && changed ? 'e'.repeat(40) : head) : wrongTree ? 'f'.repeat(40) : tree);
+ const run = vi.fn(async (_scope, _script, args, { env, signal }) => {
+ expect(signal.aborted).toBe(false);
+ expect(args).toContain('--testNamePattern=\\[fidelity\\]');
+ expect(env.GITHUB_TOKEN).toBeUndefined();
+ expect(env.OPENAI_API_KEY).toBeUndefined();
+ expect(env.PORTOS_BROWSER_TEST_PROFILE).toBe('fidelity');
+ expect(env.PORTOS_REQUIRE_BROWSER_SUITES).toBe('1');
+ roots.push(dirname(env.PORTOS_CI_DIAGNOSTICS_DIR));
+ expect(env.PORTOS_CI_BROWSER_OWNER).toBeUndefined();
+ expect(env.CHROME_PATH).toBe('/synthetic/chrome');
+ const dir = join(env.PORTOS_CI_DIAGNOSTICS_DIR, 'portos-vitest-contract');
+ mkdirSync(dir);
+ const report = { success: true, numFailedTests: 0, numFailedTestSuites: 0, numPassedTests: 3,
+ testResults: Object.entries(BROWSER_FIDELITY_TESTS).map(([name, titles]) => ({ name,
+ assertionResults: titles.map(title => ({ title, fullName: `suite ${title}`, status: 'passed' })) })) };
+ if (outcome === 'skipped') report.testResults[0].assertionResults[0].status = 'pending';
+ if (outcome === 'failed') { report.success = false; report.numFailedTests = 1; }
+ if (outcome === 'missing') report.testResults.pop();
+ for (const [name, data] of Object.entries({
+ 'invocation.json': { completed: outcome !== 'incomplete', status: 0, signal: null },
+ 'run.json': { reason: 'passed', errorCount: 0, errors: [], completedCount: 2, activeModules: [], teardownTimeout: outcome === 'teardown' },
+ 'failures.json': report,
+ })) writeFileSync(join(dir, name), JSON.stringify(data));
+ finished = true;
+ if (outcome === 'aborted') return new Promise(resolve => signal.addEventListener('abort',
+ () => resolve({ status: 143, interrupted: 'SIGTERM' }), { once: true }));
+ return { status: outcome === 'launcher' ? 1 : 0 };
+ });
+ return { root, posts, run, options: { pr: 12, repository: 'fixture/PortOS', receiptPath: join(root, 'receipt.json'),
+ request, git, run, toolchain: async () => ({ chromePath: '/synthetic/chrome',
+ receipt: { chrome: 'Chrome synthetic', ffmpeg: 'FFmpeg synthetic', renderBackend: { kind: 'unknown' } } }), env: { PATH: process.env.PATH, PORTOS_CI_BROWSER_OWNER: 'unit', GITHUB_TOKEN: 'synthetic credential', OPENAI_API_KEY: 'synthetic key' } } };
+}
+
+describe('local fidelity publication through the existing runner and status boundary', () => {
+ it('publishes pending then success only after complete exact-head results, retaining a local receipt', async () => {
+ const f = fixture();
+ const result = await publishBrowserFidelity(f.options);
+ expect(f.posts.map(post => post.state)).toEqual(['pending', 'success']);
+ const receipt = JSON.parse(readFileSync(result.receiptPath, 'utf8'));
+ expect(receipt.target).toMatchObject({ head, base, tree, merge });
+ expect(receipt.profile.files).toEqual(BROWSER_FIDELITY_FILES);
+ expect(receipt.profile.executed).toBe(3);
+ expect(f.posts[1].context.length).toBeLessThanOrEqual(100);
+ expect(f.posts[1].description).toMatch(/^receipt sha256=[a-f0-9]{64}$/);
+ expect(JSON.stringify(receipt)).not.toMatch(/synthetic credential|synthetic key/);
+ });
+
+ it('binds default git identity to the script checkout when invoked from another directory', () => {
+ const foreign = mkdtempSync(join(tmpdir(), 'fidelity-foreign-cwd-')); roots.push(foreign);
+ const entry = new URL('./browser-fidelity.js', import.meta.url).href;
+ const result = spawnSync(process.execPath, ['--input-type=module', '-e', `
+ import { publishBrowserFidelity } from ${JSON.stringify(entry)};
+ const repository = 'fixture/PortOS';
+ const request = async path => path.includes('/pulls/')
+ ? { state: 'open', head: {sha: '${head}', repo: {full_name: repository}}, base: {sha: '${base}', repo: {full_name: repository}}, merge_commit_sha: '${merge}' }
+ : { sha: '${head}', tree: {sha: '${tree}'}, parents: [{sha: '${base}'}, {sha: '${head}'}] };
+ try { await publishBrowserFidelity({ pr: 12, repository, request, run: () => { throw Error('must not execute'); } }); }
+ catch (error) { console.log(error.message); }
+ `], { cwd: foreign, encoding: 'utf8', timeout: 10000 });
+ expect(result.error).toBeUndefined();
+ expect(result.stdout).toMatch(/clean checkout|match the current PR head/);
+ expect(result.stdout).not.toContain('Git checkout identity unavailable');
+ });
+
+ it.each(['skipped', 'failed', 'missing', 'incomplete', 'teardown', 'launcher'])('never publishes success for %s evidence', async outcome => {
+ const f = fixture({ outcome });
+ await expect(publishBrowserFidelity(f.options)).rejects.toThrow('not admitted');
+ expect(f.posts.map(post => post.state)).toEqual(['pending', 'failure']);
+ });
+
+ it('fails pending publication after its bounded runner cancellation settles', async () => {
+ vi.useFakeTimers();
+ try {
+ const f = fixture({ outcome: 'aborted' });
+ const pending = publishBrowserFidelity({ ...f.options, deadlineMs: 100 });
+ const rejected = expect(pending).rejects.toThrow('not admitted');
+ await vi.advanceTimersByTimeAsync(100);
+ await rejected;
+ expect(f.posts.map(post => post.state)).toEqual(['pending', 'failure']);
+ } finally { vi.useRealTimers(); }
+ });
+
+ it('invalidates pending evidence when the checkout changes during execution', async () => {
+ const f = fixture({ changed: true });
+ await expect(publishBrowserFidelity(f.options)).rejects.toThrow('not admitted');
+ expect(f.posts.map(post => post.state)).toEqual(['pending', 'failure']);
+ });
+
+ it.each([{ dirty: true }, { wrongTree: true }])('refuses an unreviewable checkout before any status or execution: %j', async options => {
+ const f = fixture(options);
+ await expect(publishBrowserFidelity(f.options)).rejects.toThrow();
+ expect(f.posts).toEqual([]);
+ expect(f.run).not.toHaveBeenCalled();
+ });
+});
diff --git a/scripts/browser-suite-registry.test.js b/scripts/browser-suite-registry.test.js
index d29f3041d5..26843a7ca6 100644
--- a/scripts/browser-suite-registry.test.js
+++ b/scripts/browser-suite-registry.test.js
@@ -1,14 +1,44 @@
import { execFileSync } from 'node:child_process';
import { fileURLToPath } from 'node:url';
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
import { describe, expect, it } from 'vitest';
-import { BROWSER_SUITES } from './ci-test-plan.js';
+import { BROWSER_SUITES } from './lib/browserSuites.js';
+import { BROWSER_FIDELITY_ONLY_FILES, BROWSER_FIDELITY_TESTS } from './lib/browserTestProfiles.js';
+
+const root = fileURLToPath(new URL('../', import.meta.url));
// Scans the tracked tree, so it is registered in ALWAYS_RUN_TESTS (#10312).
describe('cross-workspace browser suite registry', () => {
it('lists exactly the suites that guard canRun with the browser suite gate', () => {
- const root = fileURLToPath(new URL('../', import.meta.url));
const gated = execFileSync('git', ['grep', '-l', 'browserSuiteGate', '--', 'server/**/*.browser.test.js'], { cwd: root, encoding: 'utf8' })
.split('\n').filter(Boolean).sort();
expect(gated).toEqual([...BROWSER_SUITES].sort());
});
+
+ it('keeps the source tags bound to the manifest and the short render/export/cancel cases untagged', () => {
+ expect(Object.keys(BROWSER_FIDELITY_TESTS).every(file => BROWSER_SUITES.includes(file))).toBe(true);
+ for (const file of BROWSER_SUITES) {
+ const expected = BROWSER_FIDELITY_TESTS[file] ?? [];
+ const source = readFileSync(join(root, file), 'utf8');
+ const declarations = [...source.matchAll(/\bit\((['"`])(\[fidelity\][^\n]*?)\1/g)];
+ // Catch a marker in any registered suite, including a new registration
+ // shape this small literal-title extractor does not understand yet.
+ expect(source.match(/\[fidelity\]/g)?.length ?? 0, file).toBe(declarations.length);
+ const tags = declarations.flatMap(([, , title]) =>
+ title.includes('${fixture}') ? ['ramps', 'generated-shots'].map(fixture => title.replace('${fixture}', fixture)) : [title]);
+ expect(tags, file).toEqual(expected);
+ if (BROWSER_FIDELITY_ONLY_FILES.includes(file)) {
+ const all = [...source.matchAll(/\bit\((['"`])([^\n]*?)\1/g)].map(([, , title]) => title);
+ expect(all, `${file} must not hide a newly added smoke case`).toEqual(expected);
+ }
+ }
+ const documentSource = readFileSync(join(root, 'server/services/musicVideo/documentRender.browser.test.js'), 'utf8');
+ const smokeNames = [
+ 'authors a local Three.js world and renders the same deterministic scene through module preview and export',
+ 'renders a generated 3-second card/still/clip document with the selected performance in-point',
+ 'propagates a deadline through a pending seek and waits for owned Chrome cleanup',
+ ];
+ for (const name of smokeNames) expect(documentSource).toContain(`('${name}',`);
+ });
});
diff --git a/scripts/ci-test-plan.js b/scripts/ci-test-plan.js
index f93e3c2a50..620fc7b98e 100644
--- a/scripts/ci-test-plan.js
+++ b/scripts/ci-test-plan.js
@@ -7,6 +7,7 @@ import { staticImportSpecifiersFromSource } from '../server/lib/staticImportGrap
import { isDirectlyInvoked } from './lib/directInvocation.js';
import { writeStepOutput } from './lib/githubOutput.js';
import { resolveDbTestFiles } from './lib/dbTestFiles.js';
+import { BROWSER_SUITES } from './lib/browserSuites.js';
const TEST_FILE_RE = /\.(?:test|spec)\.[cm]?[jt]sx?$/i;
const MODULE_SOURCE_RE = /\.[cm]?[jt]sx?$/i;
@@ -28,31 +29,8 @@ const MAX_CHANGED_CODE_FILES = 30;
// editorial-leaf planner test sat exactly on it (#10312).
const MAX_TARGETED_TEST_FILES = 125;
-// Server-runner suites that drive real client components in Chrome. The server
-// job installs no client workspace, so there they always skip; the `database`
-// job (client deps + Chrome + ffmpeg, PORTOS_REQUIRE_BROWSER_SUITES=1) is where
-// the files selected here run for real (issue #10312).
-//
-// An explicit list, not `*.browser.test.js`: only suites that guard `canRun`
-// with server/lib/browserSuiteGate.js can be told to fail instead of skip. The
-// registry test (browser-suite-registry.test.js) fails when a suite importing
-// the gate is left out.
-export const BROWSER_SUITES = [
- 'server/lib/filmLook.browser.test.js',
- 'server/lib/filmLookGpu.browser.test.js',
- 'server/lib/musicVideoTitleCard.browser.test.js',
- 'server/routes/musicVideoProductionReview.browser.test.js',
- 'server/routes/musicVideoRichAuthoring.browser.test.js',
- 'server/routes/musicVideoSongRevision.browser.test.js',
- 'server/services/federatedMedia/sourceAudio.browser.test.js',
- 'server/services/musicVideo/animaticCapture.browser.test.js',
- 'server/services/musicVideo/documentRender.browser.test.js',
- 'server/services/musicVideo/documentTemplates/layered/engine.overlay.browser.test.js',
- 'server/services/musicVideo/documentTemplates/shared/lyricType.browser.test.js',
- 'server/services/musicVideo/documentTemplates/shared/toonWorld.browser.test.js',
- 'server/services/musicVideo/publish/linkedin.browser.test.js',
- 'server/services/musicVideo/publish/sunoHook.browser.test.js',
-];
+// Preserve the planner's public registry export for existing callers.
+export { BROWSER_SUITES } from './lib/browserSuites.js';
const browserSuitesIn = (paths) => uniqueSorted(paths.filter((path) => BROWSER_SUITES.includes(path)));
// Python sidecar scripts (`scripts/generate_ltx2.py`, …). Vitest's import graph
diff --git a/scripts/lib/browserFidelity.js b/scripts/lib/browserFidelity.js
new file mode 100644
index 0000000000..1d68fbdb6a
--- /dev/null
+++ b/scripts/lib/browserFidelity.js
@@ -0,0 +1,63 @@
+import { createHash } from 'node:crypto';
+import { readFileSync } from 'node:fs';
+import { BROWSER_FIDELITY_TESTS, BROWSER_FIDELITY_FILES, browserTestProfileArgs } from './browserTestProfiles.js';
+
+const REPOSITORY = /^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/;
+export function browserFidelityActor(repository) {
+ if (!REPOSITORY.test(repository)) throw Error('Invalid fidelity repository.');
+ return repository.split('/')[0].toLowerCase();
+}
+const SHA = /^[a-f0-9]{40}$/;
+const digest = value => createHash('sha256').update(value).digest('hex');
+
+// The gate has a scripts-only checkout. The exact tested tree binds renderer
+// source; this digest binds the canonical selection and its completeness rule.
+export function browserFidelityProfileDigest() {
+ const source = readFileSync(new URL('./browserTestProfiles.js', import.meta.url), 'utf8').replace(/\r\n?/g, '\n');
+ return digest(JSON.stringify({ version: 1, manifest: BROWSER_FIDELITY_TESTS,
+ args: browserTestProfileArgs('fidelity'), source }));
+}
+
+export function browserFidelityContext(identity) {
+ if (!REPOSITORY.test(identity.repository) || !Number.isSafeInteger(identity.pr) || identity.pr < 1
+ || !['head', 'tree'].every(key => SHA.test(identity[key]))
+ || !/^[a-f0-9]{64}$/.test(identity.profile)) throw Error('Invalid browser fidelity identity.');
+ return `browser-fidelity/${digest(JSON.stringify({ repository: identity.repository, pr: identity.pr,
+ head: identity.head, tree: identity.tree, profile: identity.profile }))}`;
+}
+
+export function needsBrowserFidelity(files) {
+ if (!Array.isArray(files) || !files.every(file => typeof file === 'string')) throw Error('Invalid browser suite selection.');
+ return files.some(file => BROWSER_FIDELITY_FILES.includes(file));
+}
+
+/** request uses repository-relative REST paths and never exposes raw API errors. */
+export async function browserFidelityTarget(repository, pr, request, expected = {}) {
+ if (!REPOSITORY.test(repository) || !Number.isSafeInteger(pr) || pr < 1) throw Error('Invalid fidelity repository or pull request.');
+ const pull = await request(`repos/${repository}/pulls/${pr}`);
+ const head = pull?.head?.sha, base = pull?.base?.sha, merge = pull?.merge_commit_sha;
+ if (pull?.state !== 'open' || pull?.base?.repo?.full_name !== repository
+ || ![head, base].every(value => SHA.test(value))) throw Error('Open same-repository PR identity unavailable.');
+ const commit = await request(`repos/${repository}/git/commits/${head}`);
+ const tree = commit?.tree?.sha;
+ if (commit?.sha !== head || !SHA.test(tree)) throw Error('PR head tree is unavailable.');
+ if (expected.head !== undefined && expected.head !== head) throw Error('CI run identity is stale for the current PR head.');
+ return { repository, pr, head, base, merge: SHA.test(merge) ? merge : null, tree, profile: browserFidelityProfileDigest() };
+}
+
+export function assertFidelityStatus(identity, statuses) {
+ if (!Array.isArray(statuses)) throw Error('Browser fidelity statuses unavailable.');
+ const context = browserFidelityContext(identity);
+ // GitHub returns newest first; numeric IDs also disambiguate same-second posts.
+ const matching = statuses.filter(status => status?.context === context);
+ if (matching.some(status => !Number.isSafeInteger(status.id))) throw Error('Invalid browser fidelity status identity.');
+ matching.sort((a, b) => b.id - a.id);
+ const latest = matching[0];
+ if (!latest || latest.state !== 'success' || latest.creator?.login?.toLowerCase() !== browserFidelityActor(identity.repository)
+ || !/^receipt sha256=[a-f0-9]{64}$/.test(latest.description ?? '')) {
+ throw Error('Current trusted browser fidelity receipt is missing, pending, or failed.');
+ }
+ return { context, receiptDigest: latest.description.slice('receipt sha256='.length) };
+}
+
+export const browserReceiptDigest = receipt => digest(JSON.stringify(receipt));
diff --git a/scripts/lib/browserSuites.js b/scripts/lib/browserSuites.js
new file mode 100644
index 0000000000..8f72dc7528
--- /dev/null
+++ b/scripts/lib/browserSuites.js
@@ -0,0 +1,18 @@
+// Registered suites run once in the browser job; Linux unit jobs exclude only
+// this registry. Prerequisite guards fail closed when browser coverage is required.
+export const BROWSER_SUITES = [
+ 'server/lib/filmLook.browser.test.js',
+ 'server/lib/filmLookGpu.browser.test.js',
+ 'server/lib/musicVideoTitleCard.browser.test.js',
+ 'server/routes/musicVideoProductionReview.browser.test.js',
+ 'server/routes/musicVideoRichAuthoring.browser.test.js',
+ 'server/routes/musicVideoSongRevision.browser.test.js',
+ 'server/services/federatedMedia/sourceAudio.browser.test.js',
+ 'server/services/musicVideo/animaticCapture.browser.test.js',
+ 'server/services/musicVideo/documentRender.browser.test.js',
+ 'server/services/musicVideo/documentTemplates/layered/engine.overlay.browser.test.js',
+ 'server/services/musicVideo/documentTemplates/shared/lyricType.browser.test.js',
+ 'server/services/musicVideo/documentTemplates/shared/toonWorld.browser.test.js',
+ 'server/services/musicVideo/publish/linkedin.browser.test.js',
+ 'server/services/musicVideo/publish/sunoHook.browser.test.js',
+];
diff --git a/scripts/lib/browserTestProfiles.js b/scripts/lib/browserTestProfiles.js
new file mode 100644
index 0000000000..c8ffe1a4e2
--- /dev/null
+++ b/scripts/lib/browserTestProfiles.js
@@ -0,0 +1,58 @@
+/** Browser profiles select existing assertions; they never change render settings. */
+export const BROWSER_FIDELITY_TESTS = {
+ 'server/services/musicVideo/documentRender.browser.test.js': [
+ '[fidelity] matches bounded grades across real composed/document ramps and song-time excerpts',
+ '[fidelity] matches bounded grades across real composed/document generated-shots and song-time excerpts',
+ ],
+ 'server/routes/musicVideoRichAuthoring.browser.test.js': [
+ '[fidelity] authors and accepts a Three.js world through the real UI and renders its reviewed proof',
+ ],
+};
+export const BROWSER_FIDELITY_FILES = Object.keys(BROWSER_FIDELITY_TESTS);
+// This suite has one fidelity case and no smoke assertions. Other registered
+// suites retain smoke work, even when they also contain fidelity cases.
+export const BROWSER_FIDELITY_ONLY_FILES = ['server/routes/musicVideoRichAuthoring.browser.test.js'];
+
+export function browserTestProfileArgs(profile = 'full') {
+ if (profile === 'full') return [];
+ if (profile === 'smoke') return ['--testNamePattern=^(?!.*\\[fidelity\\]).*$'];
+ if (profile === 'fidelity') return ['--testNamePattern=\\[fidelity\\]'];
+ throw new Error(`Unknown browser test profile: ${profile}`);
+}
+
+export function browserTestProfileFiles(profile, files) {
+ browserTestProfileArgs(profile);
+ if (profile === 'smoke') return files.filter(file => !BROWSER_FIDELITY_ONLY_FILES.includes(file));
+ if (profile === 'fidelity') return files.filter(file => BROWSER_FIDELITY_FILES.includes(file));
+ return [...files];
+}
+
+/** A successful process with only skipped tests is not a profile receipt. */
+export function assertBrowserProfileExecuted(profile, report) {
+ browserTestProfileArgs(profile);
+ const executed = Number(report?.numPassedTests) + Number(report?.numFailedTests);
+ if (!Number.isSafeInteger(executed) || executed < 1) {
+ throw new Error(`Browser ${profile} profile executed no tests`);
+ }
+ if (profile !== 'fidelity') return executed;
+
+ // Require every named fidelity assertion, including parameter-expanded cases.
+ // Also reject new/unregistered tags so adding a case cannot silently leave
+ // the receipt's expected inventory behind.
+ const remaining = new Set(Object.entries(BROWSER_FIDELITY_TESTS)
+ .flatMap(([file, titles]) => titles.map(title => `${file}\n${title}`)));
+ if (!Array.isArray(report.testResults)) throw new Error('Fidelity test results are missing');
+ for (const suite of report.testResults) {
+ const path = String(suite.name ?? '').replaceAll('\\', '/');
+ const file = BROWSER_FIDELITY_FILES.find(file => path === file || path.endsWith(`/${file}`));
+ if (!Array.isArray(suite.assertionResults)) throw new Error('Fidelity assertions are missing');
+ for (const assertion of suite.assertionResults) {
+ if (!String(assertion.fullName ?? assertion.title ?? '').includes('[fidelity]')) continue;
+ const key = `${file}\n${assertion.title}`;
+ if (!remaining.delete(key)) throw new Error(`Unexpected or duplicate fidelity assertion: ${assertion.title}`);
+ if (assertion.status !== 'passed') throw new Error(`Fidelity assertion did not pass: ${assertion.title}`);
+ }
+ }
+ if (remaining.size) throw new Error(`Missing fidelity assertions: ${[...remaining].join(', ')}`);
+ return executed;
+}
diff --git a/scripts/lib/browserTestProfiles.test.js b/scripts/lib/browserTestProfiles.test.js
new file mode 100644
index 0000000000..27f38051e1
--- /dev/null
+++ b/scripts/lib/browserTestProfiles.test.js
@@ -0,0 +1,106 @@
+import { describe, expect, it } from 'vitest';
+import { spawnSync } from 'node:child_process';
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { createRequire } from 'node:module';
+import { tmpdir } from 'node:os';
+import { dirname, join } from 'node:path';
+import { assertBrowserProfileExecuted, BROWSER_FIDELITY_ONLY_FILES, BROWSER_FIDELITY_TESTS, browserTestProfileArgs, browserTestProfileFiles } from './browserTestProfiles.js';
+
+const smokeNames = [
+ 'authors a local Three.js world and renders the same deterministic scene through module preview and export',
+ 'renders a generated 3-second card/still/clip document with the selected performance in-point',
+ 'propagates a deadline through a pending seek and waits for owned Chrome cleanup',
+];
+const completeReport = () => ({
+ numPassedTests: Object.values(BROWSER_FIDELITY_TESTS).flat().length, numFailedTests: 0,
+ testResults: Object.entries(BROWSER_FIDELITY_TESTS).map(([name, titles]) => ({
+ name: `/checkout/${name}`, assertionResults: titles.map(title => ({ title, fullName: `suite ${title}`, status: 'passed' })),
+ })),
+});
+
+describe('browser test profiles', () => {
+ it('rejects unknown profiles and receipts with no executed assertions', () => {
+ for (const profile of ['', 'fast', null]) expect(() => browserTestProfileArgs(profile)).toThrow('Unknown browser test profile');
+ for (const profile of ['smoke', 'fidelity', 'full']) {
+ expect(() => assertBrowserProfileExecuted(profile, { numTotalTests: 3, numPassedTests: 0, numFailedTests: 0 })).toThrow('executed no tests');
+ expect(() => assertBrowserProfileExecuted(profile, {})).toThrow('executed no tests');
+ }
+ });
+
+ it('omits only the explicitly fidelity-only suite from smoke selection', () => {
+ const mixed = 'server/services/musicVideo/documentRender.browser.test.js';
+ const smoke = 'server/lib/musicVideoTitleCard.browser.test.js';
+ const files = [mixed, smoke, ...BROWSER_FIDELITY_ONLY_FILES];
+ expect(browserTestProfileFiles('smoke', files)).toEqual([mixed, smoke]);
+ expect(browserTestProfileFiles('fidelity', files)).toEqual([mixed, ...BROWSER_FIDELITY_ONLY_FILES]);
+ expect(browserTestProfileFiles('full', files)).toEqual(files);
+ expect(browserTestProfileFiles('smoke', BROWSER_FIDELITY_ONLY_FILES)).toEqual([]);
+ expect(() => browserTestProfileFiles('', files)).toThrow('Unknown browser test profile');
+ });
+
+ it('requires every fidelity assertion to pass exactly once in its owning file', () => {
+ expect(assertBrowserProfileExecuted('fidelity', completeReport())).toBe(3);
+ const windows = completeReport();
+ windows.testResults.forEach(suite => { suite.name = `D:\\checkout\\${suite.name.slice('/checkout/'.length).replaceAll('/', '\\')}`; });
+ expect(assertBrowserProfileExecuted('fidelity', windows)).toBe(3);
+ for (const status of ['failed', 'skipped', 'pending', 'todo']) {
+ const report = completeReport();
+ report.testResults[0].assertionResults[0].status = status;
+ expect(() => assertBrowserProfileExecuted('fidelity', report)).toThrow('did not pass');
+ }
+ for (const mutate of [
+ report => report.testResults.pop(),
+ report => report.testResults[0].assertionResults.pop(),
+ report => { report.testResults[0].name = '/unrelated/documentRender.browser.test.js'; },
+ report => report.testResults[0].assertionResults.push(report.testResults[0].assertionResults[0]),
+ report => report.testResults[0].assertionResults.push({ title: '[fidelity] new case', status: 'passed' }),
+ ]) {
+ const report = completeReport(); mutate(report);
+ expect(() => assertBrowserProfileExecuted('fidelity', report)).toThrow(/fidelity assertion/i);
+ }
+ });
+
+
+ // Executes only empty synthetic Node test bodies. This pins Vitest's actual
+ // filtering and JSON receipt semantics, without importing browser fixtures.
+ it('partitions the full assertion set into nonempty smoke and fidelity runs with real Vitest', () => {
+ const temp = mkdtempSync(join(tmpdir(), 'browser-profile-contract-'));
+ const require = createRequire(new URL('../../server/package.json', import.meta.url));
+ const cli = join(dirname(require.resolve('vitest/package.json')), 'vitest.mjs');
+ try {
+ const host = join(temp, 'host'); mkdirSync(host);
+ const env = { ...process.env, NODE_DISABLE_COMPILE_CACHE: '1', TMPDIR: host, TMP: host, TEMP: host };
+ // A nested Vitest run owns its own temp root; it must never reuse or
+ // sweep the parent's root while sibling contract tests are still alive.
+ delete env.PORTOS_TEST_TEMP_ROOT;
+ const config = join(temp, 'vitest.config.mjs');
+ writeFileSync(config, `import { bootstrapVitestTempRoot } from ${JSON.stringify(new URL('./vitestTempRoot.js', import.meta.url).href)};
+bootstrapVitestTempRoot();
+export default { test: { globals: true, maxWorkers: 1, include: ['server/**/*.test.js'],
+globalSetup: [${JSON.stringify(new URL('../vitestTempRootSetup.js', import.meta.url).href)}] } };`);
+ for (const [file, titles] of Object.entries(BROWSER_FIDELITY_TESTS)) {
+ const path = join(temp, file); mkdirSync(dirname(path), { recursive: true });
+ const names = file.includes('documentRender') ? [...smokeNames, ...titles] : titles;
+ writeFileSync(path, names.map(title => `it(${JSON.stringify(title)}, () => {});`).join('\n'));
+ }
+ const passed = {};
+ for (const profile of ['smoke', 'fidelity', 'full']) {
+ const output = join(temp, `${profile}.json`);
+ const run = spawnSync(process.execPath, [cli, 'run', '--config', config, '--configLoader=runner',
+ ...browserTestProfileArgs(profile), '--reporter=json', `--outputFile=${output}`], {
+ cwd: temp, encoding: 'utf8', timeout: 20000,
+ env,
+ });
+ expect(run.error).toBeUndefined();
+ expect(run.status, run.stderr || run.stdout).toBe(0);
+ const report = JSON.parse(readFileSync(output, 'utf8'));
+ expect(assertBrowserProfileExecuted(profile, report)).toBeGreaterThan(0);
+ passed[profile] = report.testResults.flatMap(suite => suite.assertionResults)
+ .filter(assertion => assertion.status === 'passed').map(assertion => assertion.title).sort();
+ }
+ expect(passed.smoke).toEqual([...smokeNames].sort());
+ expect(passed.fidelity).toEqual(Object.values(BROWSER_FIDELITY_TESTS).flat().sort());
+ expect([...passed.smoke, ...passed.fidelity].sort()).toEqual(passed.full);
+ } finally { rmSync(temp, { recursive: true, force: true }); }
+ }, 70000);
+});
diff --git a/scripts/lib/workspaceCacheFixture.js b/scripts/lib/workspaceCacheFixture.js
index 4ef8e239f8..1b89a38df8 100644
--- a/scripts/lib/workspaceCacheFixture.js
+++ b/scripts/lib/workspaceCacheFixture.js
@@ -47,6 +47,7 @@ function checkout(host, name, workspace, configName, dependencies) {
mkdirSync(parent);
put(join(parent, 'package.json'), '{"type":"module"}');
const fixture = createVitestTempFixture(parent, workspace, 'expect(true).toBe(true);', configName);
+ delete fixture.options.env.PORTOS_CI_BROWSER_OWNER;
const root = fixture.args[fixture.args.indexOf('--root') + 1];
unlinkSync(join(root, 'node_modules'));
link(dependencies, join(root, 'node_modules'));
@@ -54,7 +55,7 @@ function checkout(host, name, workspace, configName, dependencies) {
// Copy the real config so import.meta.url names THIS checkout; its existing
// helpers stay shared read-only, just like dependencies in an ordinary worktree.
copyFileSync(join(REPO, workspace, configName), join(root, configName));
- for (const file of ['lib/vitestTempRoot.js', 'lib/dbTestFiles.js', 'vitestCiPool.js', 'vitestTempRootSetup.js']) {
+ for (const file of ['lib/vitestTempRoot.js', 'lib/dbTestFiles.js', 'lib/browserSuites.js', 'vitestCiPool.js', 'vitestTempRootSetup.js']) {
link(join(REPO, 'scripts', file), join(parent, 'scripts', file));
}
if (workspace === 'server') {
diff --git a/scripts/pregate.test.js b/scripts/pregate.test.js
index ca8ae2fbff..39b5798d78 100644
--- a/scripts/pregate.test.js
+++ b/scripts/pregate.test.js
@@ -121,6 +121,7 @@ describe('pregate hidden-content invocation', () => {
'scripts/pregate.js', 'scripts/checkNodeVersion.js', 'scripts/scan-diff-hidden-content.js',
'scripts/ci-test-plan.js', 'scripts/ci-base-sha.js', 'server/lib/staticImportGraph.js',
'scripts/lib/directInvocation.js', 'scripts/lib/githubOutput.js', 'scripts/lib/dbTestFiles.js',
+ 'scripts/lib/browserSuites.js',
'server/lib/diffHiddenContentScan.js', 'server/lib/modelAbuseGuard.js',
'server/lib/textUtils.js',
]) {
@@ -439,6 +440,7 @@ describe('the gate end to end', () => {
'scripts/pregate.js', 'scripts/checkNodeVersion.js', 'scripts/scan-diff-hidden-content.js',
'scripts/ci-test-plan.js', 'scripts/ci-base-sha.js', 'server/lib/staticImportGraph.js',
'scripts/lib/directInvocation.js', 'scripts/lib/githubOutput.js', 'scripts/lib/dbTestFiles.js',
+ 'scripts/lib/browserSuites.js',
'server/lib/diffHiddenContentScan.js', 'server/lib/modelAbuseGuard.js',
'server/lib/textUtils.js',
]) {
diff --git a/scripts/run-ci-tests.js b/scripts/run-ci-tests.js
index 6a156f1310..db62014054 100644
--- a/scripts/run-ci-tests.js
+++ b/scripts/run-ci-tests.js
@@ -12,6 +12,7 @@ import { sweepStaleRunRoots, writeOwnerFile } from './lib/vitestStaleRunRoots.js
import { writeStepEnv, writeStepSummary } from './lib/githubOutput.js';
import { reportWindowsCrashDiagnostics } from './ci-windows-crash-diagnostics.js';
import { diagnosticTestFile, persistTestDiagnostic } from './ci-test-diagnostics.js';
+import { browserTestProfileArgs, browserTestProfileFiles, assertBrowserProfileExecuted, BROWSER_FIDELITY_ONLY_FILES } from './lib/browserTestProfiles.js';
const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '..');
@@ -189,12 +190,13 @@ export function structuredFailureDiagnostics(reportPath, runReportPath) {
* stderr live (so a long CI job keeps showing progress) while also retaining
* a bounded tail of the combined output for post-run crash detection.
*
- * @returns {Promise<{status: number, output: string, interrupted?: NodeJS.Signals, error?: Error}>}
+ * @returns {Promise<{status: number, output: string, interrupted?: NodeJS.Signals, error?: Error, cleanupUnverified?: boolean, diagnosticsPath?: string}>}
*/
-export async function runNpm(scope, script, extraArgs) {
+export async function runNpm(scope, script, extraArgs, { env = process.env, signal, cancellationTimeoutMs = 10000 } = {}) {
+ signal?.throwIfAborted();
// Opt-in, job-owned storage survives an abruptly terminated launcher. Keep
// separate roots for the initial attempt and the existing single retry.
- const durableRoot = process.env.PORTOS_CI_DIAGNOSTICS_DIR;
+ const durableRoot = env.PORTOS_CI_DIAGNOSTICS_DIR;
let hostRoot = tmpdir();
if (durableRoot) {
try { mkdirSync(durableRoot, { recursive: true }); hostRoot = durableRoot; }
@@ -204,15 +206,31 @@ export async function runNpm(scope, script, extraArgs) {
if (!durable) sweepStaleRunRoots(hostRoot, { prefix: 'portos-vitest-' });
const reportDir = mkdtempSync(join(hostRoot, 'portos-vitest-'));
const reportPath = join(reportDir, 'failures.json');
+ let cleanupUnverified = false;
try {
writeOwnerFile(reportDir);
- return await runNpmWithReport(scope, script, extraArgs, reportPath);
+ const result = await runNpmWithReport(scope, script, extraArgs, reportPath, env, signal, cancellationTimeoutMs);
+ cleanupUnverified = result.cleanupUnverified === true;
+ if (cleanupUnverified) {
+ console.error(`Test launcher cleanup unverified; diagnostics retained at ${reportDir}`);
+ return { ...result, diagnosticsPath: reportDir };
+ }
+ const profile = env.PORTOS_BROWSER_TEST_PROFILE;
+ if (profile !== undefined && result.status === 0) {
+ try {
+ assertBrowserProfileExecuted(profile, JSON.parse(readFileSync(reportPath, 'utf8')));
+ } catch (error) {
+ console.error(`Browser profile evidence rejected: ${error.message}`);
+ return { ...result, status: 1 };
+ }
+ }
+ return result;
} finally {
- if (!durable) rmSync(reportDir, { recursive: true, force: true });
+ if (!durable && !cleanupUnverified) rmSync(reportDir, { recursive: true, force: true });
}
}
-function runNpmWithReport(scope, script, extraArgs, reportPath) {
+function runNpmWithReport(scope, script, extraArgs, reportPath, env, abortSignal, cancellationTimeoutMs) {
const runReportPath = join(dirname(reportPath), 'run.json');
const invocationPath = join(dirname(reportPath), 'invocation.json');
const invocation = {
@@ -227,12 +245,12 @@ function runNpmWithReport(scope, script, extraArgs, reportPath) {
// Wrapped, not a bare `npm.cmd`: Node refuses to spawn a `.cmd` under
// `shell:false` and throws EINVAL, so this script could never run on a
// Windows checkout. See server/lib/bufferedSpawn.js.
- const { command, args: spawnArgs } = prepareCliSpawn('npm', args);
+ const { command, args: spawnArgs } = prepareCliSpawn('npm', args, env);
return new Promise((resolve) => {
// npm is a launcher, not the entire test tree. Give this invocation its
// own POSIX group; Windows killProcessTree uses taskkill /T instead.
const processGroup = process.platform !== 'win32';
- const childEnv = { ...process.env, PORTOS_CI_DIAGNOSTICS_PATH: runReportPath };
+ const childEnv = { ...env, PORTOS_CI_DIAGNOSTICS_PATH: runReportPath };
// Only the workflow launcher owns durable attempts. Nested launcher
// fixtures must not overwrite or add to the job's failure provenance.
delete childEnv.PORTOS_CI_DIAGNOSTICS_DIR;
@@ -242,6 +260,9 @@ function runNpmWithReport(scope, script, extraArgs, reportPath) {
let interrupted;
let escalation;
let spawnError;
+ let cancellationDeadline;
+ let settled = false;
+ let cleanupUnverified = false;
const cancel = (signal) => {
if (interrupted) return;
interrupted = signal;
@@ -249,9 +270,20 @@ function runNpmWithReport(scope, script, extraArgs, reportPath) {
escalation = setTimeout(() => {
killProcessTree(child, 'SIGKILL', { processGroup });
}, 2000);
+ if (abortSignal) cancellationDeadline = setTimeout(() => {
+ cleanupUnverified = true;
+ spawnError = new Error('Owned test launcher cleanup unverified after cancellation.');
+ child.stdout.destroy();
+ child.stderr.destroy();
+ child.unref();
+ finish(null, null);
+ }, cancellationTimeoutMs);
};
const onSigint = () => cancel('SIGINT');
const onSigterm = () => cancel('SIGTERM');
+ const onAbort = () => cancel('SIGTERM');
+ abortSignal?.addEventListener('abort', onAbort, { once: true });
+ if (abortSignal?.aborted) onAbort();
process.on('SIGINT', onSigint);
process.on('SIGTERM', onSigterm);
let output = '';
@@ -263,8 +295,11 @@ function runNpmWithReport(scope, script, extraArgs, reportPath) {
child.stderr.on('data', (chunk) => tee(process.stderr, chunk));
// A spawn error also emits close. Settle only there, after the pipes close.
child.on('error', (error) => { spawnError = error; });
- child.on('close', (code, signal) => {
+ const finish = (code, signal) => {
+ if (settled) return;
+ settled = true;
clearTimeout(escalation);
+ clearTimeout(cancellationDeadline);
if (interrupted) {
// npm may close before a descendant that detached its stdio. Finish
// the owned group as well, including children that ignored SIGTERM.
@@ -272,8 +307,9 @@ function runNpmWithReport(scope, script, extraArgs, reportPath) {
}
process.removeListener('SIGINT', onSigint);
process.removeListener('SIGTERM', onSigterm);
+ abortSignal?.removeEventListener('abort', onAbort);
const status = interrupted ? (interrupted === 'SIGINT' ? 130 : 143) : code ?? 1;
- persistTestDiagnostic(invocationPath, { ...invocation, completed: true, status, signal: signal ?? interrupted ?? null });
+ persistTestDiagnostic(invocationPath, { ...invocation, completed: !cleanupUnverified, status, signal: signal ?? interrupted ?? null });
if (status !== 0) {
console.error(`Test launcher exited: code=${code}, signal=${signal ?? 'none'}${interrupted ? `, interrupted=${interrupted}` : ''}`);
console.error(structuredFailureDiagnostics(reportPath, runReportPath));
@@ -281,8 +317,10 @@ function runNpmWithReport(scope, script, extraArgs, reportPath) {
// hook, including exit codes unavailable through the reporter API.
console.error(`Vitest output tail (last 16000 characters):\n${stripVTControlCharacters(output).slice(-16_000)}`);
}
- resolve({ status, output, ...(interrupted ? { interrupted } : {}), ...(spawnError ? { error: spawnError } : {}) });
- });
+ resolve({ status, output, ...(interrupted ? { interrupted } : {}), ...(spawnError ? { error: spawnError } : {}),
+ ...(cleanupUnverified ? { cleanupUnverified: true } : {}) });
+ };
+ child.on('close', finish);
});
}
@@ -291,7 +329,9 @@ export async function spawnNpm(scope, script, extraArgs, label, {
} = {}) {
console.log(`Running ${scope} ${label}${extraArgs.length ? ` (${extraArgs.length} selector argument(s))` : ''}.`);
const startedAt = Date.now();
- const result = await run(scope, script, extraArgs);
+ const profileArgs = process.env.PORTOS_BROWSER_TEST_PROFILE !== undefined
+ ? browserTestProfileArgs(process.env.PORTOS_BROWSER_TEST_PROFILE) : [];
+ const result = await run(scope, script, [...extraArgs, ...profileArgs]);
recordVitestDuration(scope, label, startedAt);
if (result.error) {
console.error(result.error.message);
@@ -315,7 +355,7 @@ export async function spawnNpm(scope, script, extraArgs, label, {
// reproduces or was a one-off runner fault.
console.warn(`⚠️ Vitest worker crashed natively on ${relPath} — retrying that file once`);
const retryStartedAt = Date.now();
- const retry = await run(scope, script, [selector]);
+ const retry = await run(scope, script, [selector, ...profileArgs]);
recordVitestDuration(scope, `crash retry: ${relPath}`, retryStartedAt);
if (!retry.error && retry.status === 0) {
if (hasOtherTestFailures(result.output)) {
@@ -348,7 +388,9 @@ async function main() {
}
const mode = process.env.CI_TEST_MODE || 'full';
- const repoFiles = JSON.parse(process.env.CI_TEST_FILES || '[]');
+ const inputFiles = JSON.parse(process.env.CI_TEST_FILES || '[]');
+ const repoFiles = process.env.PORTOS_BROWSER_TEST_PROFILE !== undefined
+ ? browserTestProfileFiles(process.env.PORTOS_BROWSER_TEST_PROFILE, inputFiles) : inputFiles;
const repoSources = JSON.parse(process.env.CI_TEST_SOURCES || '[]');
const selectedFiles = repoFiles.map((path) => toRunnerPath(scope, path));
const sourceFiles = repoSources.map((path) => toRunnerPath(scope, path));
@@ -377,7 +419,11 @@ async function main() {
if (mode === 'files') {
if (selectedFiles.length === 0) {
- console.log(`No ${scope} tests selected.`);
+ const profile = process.env.PORTOS_BROWSER_TEST_PROFILE;
+ const deferredOnly = profile === 'smoke' && inputFiles.length > 0
+ && inputFiles.every(file => BROWSER_FIDELITY_ONLY_FILES.includes(file));
+ if (profile !== undefined && !deferredOnly) throw Error('Browser profile selected no executable cases.');
+ console.log(`No ${scope} tests selected.${deferredOnly ? ' Fidelity-only inputs require separate evidence.' : ''}`);
process.exitCode = 0;
return;
}
diff --git a/scripts/run-ci-tests.test.js b/scripts/run-ci-tests.test.js
index 9f60d3e785..e7f827a36c 100644
--- a/scripts/run-ci-tests.test.js
+++ b/scripts/run-ci-tests.test.js
@@ -1,4 +1,6 @@
import { spawn, spawnSync } from 'node:child_process';
+import { EventEmitter } from 'node:events';
+import { PassThrough } from 'node:stream';
import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, utimesSync, writeFileSync } from 'fs';
import { tmpdir } from 'os';
import { delimiter, join } from 'path';
@@ -76,6 +78,22 @@ it('forwards the related matrix slice and all selectors through the real launche
}
});
+it('rejects empty browser coverage except explicitly deferred smoke-only inputs', () => {
+ for (const [profile, files, expected] of [
+ ['fidelity', [], 1], ['fidelity', ['server/lib/filmLook.browser.test.js'], 1],
+ ['full', [], 1], ['smoke', [], 1],
+ ['smoke', ['server/routes/musicVideoRichAuthoring.browser.test.js'], 0],
+ ]) {
+ const result = spawnSync(process.execPath, [join(import.meta.dirname, 'run-ci-tests.js'), 'server'], {
+ encoding: 'utf8', timeout: 10000,
+ env: { ...process.env, PORTOS_BROWSER_TEST_PROFILE: profile, CI_TEST_MODE: 'files',
+ CI_TEST_FILES: JSON.stringify(files), GITHUB_ENV: '', GITHUB_STEP_SUMMARY: '' },
+ });
+ expect(result.error).toBeUndefined();
+ expect(result.status, `${profile}: ${result.stderr}`).toBe(expected);
+ }
+});
+
describe('ci.yml shard wiring', () => {
const runners = Object.entries(workflowJobs(WORKFLOW)).filter(([, body]) => body.includes('run-ci-tests.js'));
@@ -110,6 +128,21 @@ describe('ci.yml shard wiring', () => {
expect(body.indexOf('npm ci --prefix client')).toBeLessThan(start);
});
+ it('deduplicates only Linux browser ownership and admits PR fidelity separately', () => {
+ const jobs = workflowJobs(WORKFLOW);
+ expect(jobs.server).toContain('PORTOS_CI_BROWSER_OWNER: unit');
+ expect(jobs['windows-server']).not.toContain('PORTOS_CI_BROWSER_OWNER');
+ expect(jobs.database).toContain("github.event_name == 'pull_request' && github.base_ref != 'release' && inputs.full != true && 'smoke' || 'full'");
+ expect(jobs.gate.indexOf('node scripts/ci-gate-report.js')).toBeLessThan(
+ jobs.gate.indexOf('node scripts/verify-browser-fidelity.js'));
+ for (const name of ['CI_BROWSER_FILES', 'CI_EXPECTED_HEAD']) {
+ expect(jobs.gate).toContain(`${name}:`);
+ }
+ expect(jobs['full-gate']).toContain("(github.event_name != 'pull_request' || github.base_ref == 'release' || inputs.full == true)");
+ expect(jobs.gate).toContain('CI_GATE_RESULT_LINUX_CONTAINMENT:');
+ expect(jobs.gate).toContain('CI_GATE_RESULT_WINDOWS_SERVER:');
+ });
+
it('runs once-only steps on the first shard alone', () => {
// Smoke boot, the production build, and the Scalar-removal pin are not
// sharded work; on every shard they would triple the cost for no coverage.
@@ -330,6 +363,32 @@ describe('native crash diagnostics before retry/cancellation (#8152)', () => {
expect(run.mock.calls[1]).toEqual(['server', 'test:ci', ['./services/sprites/walk.test.js']]);
});
+ it('refuses an explicitly empty browser profile before invoking tests', async () => {
+ vi.stubEnv('PORTOS_BROWSER_TEST_PROFILE', '');
+ const run = vi.fn();
+ try {
+ await expect(spawnNpm('server', 'test:ci', ['selected.browser.test.js'], 'invalid', { run }))
+ .rejects.toThrow('Unknown browser test profile');
+ expect(run).not.toHaveBeenCalled();
+ } finally { vi.unstubAllEnvs(); }
+ });
+
+ it('keeps browser profile selection on a recovered worker retry', async () => {
+ vi.stubEnv('PORTOS_BROWSER_TEST_PROFILE', 'smoke');
+ try {
+ const run = vi.fn().mockResolvedValueOnce({ status: 1, output })
+ .mockResolvedValueOnce({ status: 0, output: 'passed' });
+ expect(await spawnNpm('server', 'test:ci', ['selected.browser.test.js'], 'smoke', {
+ run, diagnose: vi.fn(),
+ })).toBe(0);
+ const filter = '--testNamePattern=^(?!.*\\[fidelity\\]).*$';
+ expect(run.mock.calls[0][2]).toEqual(['selected.browser.test.js', filter]);
+ expect(run.mock.calls[1][2]).toEqual(['./services/sprites/walk.test.js', filter]);
+ } finally {
+ vi.unstubAllEnvs();
+ }
+ });
+
it('retains diagnostic evidence for recovered crashes without clearing unrelated assertion failures', async () => {
for (const [originalOutput, expected] of [[output, 0], [output.replace('798 passed', '797 passed | 1 failed'), 1]]) {
const run = vi.fn().mockResolvedValueOnce({ status: 1, output: originalOutput })
@@ -448,7 +507,49 @@ describe('structured failure diagnostics', () => {
describe('report launcher subprocess lifecycle (#9804)', () => {
const runnerUrl = new URL('./run-ci-tests.js', import.meta.url).href;
- function launch(host, name, { blocking = false } = {}) {
+ it('retains temporary diagnostics when cancellation cannot verify child cleanup', async () => {
+ const host = mkdtempSync(join(tmpdir(), 'launcher-unverified-'));
+ const child = Object.assign(new EventEmitter(), {
+ stdout: new PassThrough(), stderr: new PassThrough(), unref: vi.fn(),
+ });
+ const errors = vi.spyOn(console, 'error').mockImplementation(() => {});
+ vi.useFakeTimers();
+ vi.doMock('child_process', () => ({ spawn: () => child }));
+ vi.doMock('node:os', async importOriginal => ({ ...await importOriginal(), tmpdir: () => host }));
+ vi.doMock('../server/lib/bufferedSpawn.js', async importOriginal => ({
+ ...await importOriginal(), killProcessTree: vi.fn(),
+ }));
+ vi.resetModules();
+ try {
+ const { runNpm } = await import('./run-ci-tests.js');
+ const controller = new AbortController();
+ const pending = runNpm('server', 'test:ci:related', [], {
+ env: { ...process.env, PORTOS_CI_DIAGNOSTICS_DIR: '' },
+ signal: controller.signal, cancellationTimeoutMs: 100,
+ });
+ controller.abort();
+ await vi.advanceTimersByTimeAsync(100);
+ const result = await pending;
+ expect(result).toMatchObject({ status: 143, interrupted: 'SIGTERM', cleanupUnverified: true });
+ expect(result.error.message).toContain('cleanup unverified');
+ expect(result.diagnosticsPath).toBe(join(host, readdirSync(host)[0]));
+ expect(JSON.parse(readFileSync(join(result.diagnosticsPath, 'invocation.json'), 'utf8')))
+ .toMatchObject({ completed: false, status: 143, signal: 'SIGTERM' });
+ expect(errors).toHaveBeenCalledWith(expect.stringContaining(result.diagnosticsPath));
+ } finally {
+ vi.useRealTimers();
+ vi.doUnmock('child_process');
+ vi.doUnmock('node:os');
+ vi.doUnmock('../server/lib/bufferedSpawn.js');
+ vi.resetModules();
+ errors.mockRestore();
+ child.stdout.destroy();
+ child.stderr.destroy();
+ rmSync(host, { recursive: true, force: true });
+ }
+ });
+
+ function launch(host, name, { blocking = false, abortOnSignal = false } = {}) {
const fixture = join(host, name);
mkdirSync(fixture);
const ready = join(fixture, 'ready.json');
@@ -473,7 +574,10 @@ describe('report launcher subprocess lifecycle (#9804)', () => {
delete env.PORTOS_TEST_TEMP_ROOT;
const child = spawn(process.execPath, ['--input-type=module', '-e', `
import { runNpm } from ${JSON.stringify(runnerUrl)};
- const result = await runNpm('server', 'test:ci:related', ${JSON.stringify(args)});
+ const controller = new AbortController();
+ if (${abortOnSignal}) process.once('SIGUSR2', () => controller.abort());
+ const result = await runNpm('server', 'test:ci:related', ${JSON.stringify(args)},
+ ${abortOnSignal} ? { signal: controller.signal } : {});
process.exitCode = result.status;`], { env, stdio: ['ignore', 'pipe', 'pipe'] });
let output = '';
child.stdout.on('data', chunk => { output += chunk; });
@@ -519,13 +623,13 @@ describe('report launcher subprocess lifecycle (#9804)', () => {
// Windows cannot deliver POSIX SIGINT/SIGTERM to a Node process; ordinary
// outcomes and recovery below still exercise its real npm.cmd boundary.
describe.skipIf(process.platform === 'win32')('graceful signals', () => {
- for (const signal of ['SIGINT', 'SIGTERM']) {
+ for (const signal of ['SIGINT', 'SIGTERM', 'SIGUSR2']) {
it(`${signal} fails the stage, stops descendants, and removes the report root`, async () => {
const host = mkdtempSync(join(tmpdir(), 'rcl-'));
let run;
let pids;
try {
- run = launch(host, 'cancel', { blocking: true });
+ run = launch(host, 'cancel', { blocking: true, abortOnSignal: signal === 'SIGUSR2' });
await waitFor(() => existsSync(run.ready), 'test worker did not start');
pids = JSON.parse(readFileSync(run.ready, 'utf8'));
expect(ownedRoot(host, run.child.pid)).toBeTruthy();
diff --git a/scripts/verify-browser-fidelity.js b/scripts/verify-browser-fidelity.js
new file mode 100644
index 0000000000..19eb6c7aaa
--- /dev/null
+++ b/scripts/verify-browser-fidelity.js
@@ -0,0 +1,41 @@
+#!/usr/bin/env node
+
+import { readFileSync } from 'node:fs';
+import { browserFidelityTarget, assertFidelityStatus, needsBrowserFidelity } from './lib/browserFidelity.js';
+import { githubJson, repoApiPath } from './lib/githubActionsApi.js';
+import { isDirectlyInvoked } from './lib/directInvocation.js';
+
+export async function verifyBrowserFidelity({ env = process.env, fetchImpl = fetch } = {}) {
+ const files = JSON.parse(env.CI_BROWSER_FILES);
+ if (!needsBrowserFidelity(files)) return { required: false };
+ const repository = env.GITHUB_REPOSITORY;
+ const repoPath = repoApiPath(env), token = env.GITHUB_TOKEN;
+ let pr = Number(env.CI_PULL_REQUEST);
+ if (!pr && env.GITHUB_EVENT_PATH) pr = Number(JSON.parse(readFileSync(env.GITHUB_EVENT_PATH, 'utf8')).number);
+ if (!repoPath || !token) throw Error('Browser fidelity verification credentials unavailable.');
+ const request = async path => {
+ const prefix = `repos/${repository}`;
+ if (!path.startsWith(`${prefix}/`)) throw Error('Invalid fidelity API target.');
+ const body = await githubJson(fetchImpl, `${repoPath}${path.slice(prefix.length)}`, token);
+ if (body === null) throw Error('Browser fidelity API evidence unavailable.');
+ return body;
+ };
+ if (!/^[a-f0-9]{40}$/.test(env.CI_EXPECTED_HEAD ?? '')) {
+ throw Error('CI browser fidelity checkout identity unavailable.');
+ }
+ const target = await browserFidelityTarget(repository, pr, request, {
+ head: env.CI_EXPECTED_HEAD,
+ });
+ const statuses = await request(`repos/${repository}/commits/${target.head}/statuses?per_page=100`);
+ return { required: true, ...assertFidelityStatus(target, statuses) };
+}
+
+if (isDirectlyInvoked(import.meta.url)) {
+ try {
+ const result = await verifyBrowserFidelity();
+ console.log(result.required ? '✅ Current trusted browser fidelity receipt verified.' : '✅ No deferred browser fidelity selected.');
+ } catch {
+ console.error('❌ Browser fidelity admission failed: current trusted exact-head/tree/profile evidence is required.');
+ process.exitCode = 1;
+ }
+}
diff --git a/scripts/verify-browser-fidelity.test.js b/scripts/verify-browser-fidelity.test.js
new file mode 100644
index 0000000000..a80de9346f
--- /dev/null
+++ b/scripts/verify-browser-fidelity.test.js
@@ -0,0 +1,74 @@
+import { describe, expect, it, vi } from 'vitest';
+import { verifyBrowserFidelity } from './verify-browser-fidelity.js';
+import { browserFidelityContext, browserFidelityProfileDigest } from './lib/browserFidelity.js';
+import { BROWSER_FIDELITY_FILES } from './lib/browserTestProfiles.js';
+
+const target = { repository: 'fixture/PortOS', pr: 12, head: 'a'.repeat(40), base: 'b'.repeat(40),
+ merge: 'c'.repeat(40), tree: 'd'.repeat(40), profile: browserFidelityProfileDigest() };
+const status = (overrides = {}) => ({ id: 1, context: browserFidelityContext(target), state: 'success',
+ description: `receipt sha256=${'e'.repeat(64)}`, creator: { login: 'fixture' }, ...overrides });
+
+function fixture({ statuses = [status()], expected = {}, mergeTree = target.tree, forkHead = false } = {}) {
+ const env = { GITHUB_REPOSITORY: target.repository, GITHUB_TOKEN: 'synthetic token', CI_PULL_REQUEST: '12',
+ CI_BROWSER_FILES: JSON.stringify(BROWSER_FIDELITY_FILES), CI_EXPECTED_HEAD: target.head,
+ CI_EXPECTED_BASE: target.base, CI_EXPECTED_MERGE: target.merge, ...expected };
+ const fetchImpl = vi.fn(async url => {
+ let body;
+ if (url.endsWith('/pulls/12')) body = { state: 'open', head: { sha: target.head, repo: { full_name: forkHead ? 'contributor/PortOS' : target.repository } },
+ base: { sha: target.base, repo: { full_name: target.repository } }, merge_commit_sha: target.merge };
+ else if (url.includes('/git/commits/')) body = { sha: target.head, tree: { sha: mergeTree },
+ parents: [{ sha: target.base }, { sha: target.head }] };
+ else if (url.includes('/statuses?')) body = statuses;
+ else throw Error('Unexpected synthetic API URL');
+ return new Response(JSON.stringify(body), { status: 200 });
+ });
+ return { env, fetchImpl };
+}
+
+describe('required browser fidelity admission', () => {
+ it('accepts the personal repository owner and does not imply a remotely stored receipt', async () => {
+ const f = fixture();
+ await expect(verifyBrowserFidelity(f)).resolves.toEqual({ required: true,
+ context: status().context, receiptDigest: 'e'.repeat(64) });
+ expect(f.fetchImpl.mock.calls.at(-1)[0]).toContain(`/commits/${target.head}/statuses`);
+ });
+
+ it('accepts a fork-origin PR only through the base repository owner status', async () => {
+ await expect(verifyBrowserFidelity(fixture({ forkHead: true }))).resolves.toMatchObject({ required: true });
+ await expect(verifyBrowserFidelity(fixture({ forkHead: true,
+ statuses: [status({ creator: { login: 'contributor' } })] }))).rejects.toThrow('receipt');
+ });
+
+ it('rejects a stale PR head', async () => {
+ await expect(verifyBrowserFidelity(fixture({ expected: { CI_EXPECTED_HEAD: 'f'.repeat(40) } }))).rejects.toThrow('stale');
+ });
+
+ it('retains head-only fidelity when only the observed base and merge advance', async () => {
+ const f = fixture({ expected: { CI_EXPECTED_BASE: 'f'.repeat(40), CI_EXPECTED_MERGE: 'f'.repeat(40) } });
+ await expect(verifyBrowserFidelity(f)).resolves.toMatchObject({ required: true });
+ expect(browserFidelityContext({ ...target, base: 'f'.repeat(40), merge: 'f'.repeat(40) })).toBe(status().context);
+ });
+
+ it.each([
+ [],
+ [status({ context: browserFidelityContext({ ...target, profile: '0'.repeat(64) }) })],
+ [status({ creator: { login: 'untrusted' } })],
+ [status(), status({ id: 2, state: 'pending' })],
+ [status(), status({ id: 2, state: 'failure' })],
+ ].map(statuses => ({ statuses })))('fails closed on missing, wrong-profile, untrusted or superseded evidence: %j', async ({ statuses }) => {
+ await expect(verifyBrowserFidelity(fixture({ statuses }))).rejects.toThrow('receipt');
+ });
+
+ it('rejects a receipt for a different head tree', async () => {
+ await expect(verifyBrowserFidelity(fixture({ mergeTree: 'f'.repeat(40) }))).rejects.toThrow('receipt');
+ });
+
+ it('requires selection input, but does not request receipts for smoke-only suites', async () => {
+ const f = fixture();
+ delete f.env.CI_BROWSER_FILES;
+ await expect(verifyBrowserFidelity(f)).rejects.toThrow();
+ f.env.CI_BROWSER_FILES = '["server/lib/filmLook.browser.test.js"]';
+ await expect(verifyBrowserFidelity(f)).resolves.toEqual({ required: false });
+ expect(f.fetchImpl).not.toHaveBeenCalled();
+ });
+});
diff --git a/scripts/vitestCiPool.js b/scripts/vitestCiPool.js
index 6c3bbf022d..06a3472f2e 100644
--- a/scripts/vitestCiPool.js
+++ b/scripts/vitestCiPool.js
@@ -47,6 +47,7 @@ export const EXCLUSIVE_CAPTURE_TESTS = [
'services/musicVideo/documentTemplates/shared/lyricType.browser.test.js',
'services/musicVideo/documentTemplates/shared/toonWorld.browser.test.js',
'services/musicVideo/overlayTextService.browser.test.js',
+ 'services/musicVideo/publish/sunoHook.browser.test.js',
'routes/musicVideoProductionReview.browser.test.js',
'routes/musicVideoRichAuthoring.browser.test.js',
];
diff --git a/scripts/vitestCiPool.test.js b/scripts/vitestCiPool.test.js
index b51e0a2d4f..cf35021365 100644
--- a/scripts/vitestCiPool.test.js
+++ b/scripts/vitestCiPool.test.js
@@ -125,6 +125,7 @@ it('runs parallel units before exclusive, nonduplicated captures and cutover pro
['film-look', 'lib/filmLook.browser.test.js'],
['html', 'services/htmlComposition/index.test.js'],
['music', 'services/musicVideo/documentRender.browser.test.js'],
+ ['suno', 'services/musicVideo/publish/sunoHook.browser.test.js'],
['cutover', 'services/databaseMaintenanceCutover.test.js'],
['assembly', 'services/creativeDirector/videoAssembly.test.js'],
['process-audit-cleanup', '../scripts/persistentMindProcessAuditCleanup.test.js'],
@@ -143,6 +144,7 @@ it('runs parallel units before exclusive, nonduplicated captures and cutover pro
const env = { ...process.env, NODE_ENV: 'development', TMPDIR: scratch, TMP: scratch, TEMP: scratch };
delete env.PORTOS_TEST_TEMP_ROOT;
delete env.VITEST_FAST;
+ delete env.PORTOS_CI_BROWSER_OWNER;
const result = spawnSync(process.execPath, [join(server, 'node_modules/vitest/vitest.mjs'), 'run', '--configLoader', 'runner',
'--config', join(server, 'vitest.config.js'), '--root', fixture, '--maxWorkers', '2',
], { cwd: server, env, encoding: 'utf8', timeout: 30000 });
@@ -150,7 +152,7 @@ it('runs parallel units before exclusive, nonduplicated captures and cutover pro
expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(0);
const events = readFileSync(join(fixture, 'events'), 'utf8').trim().split('\n');
expect(events.slice(0, 2).sort()).toEqual(['unit-a', 'unit-b']);
- expect(events.slice(2).sort()).toEqual(['assembly', 'cutover', 'film-look', 'html', 'music', 'process-audit-cleanup']);
+ expect(events.slice(2).sort()).toEqual(['assembly', 'cutover', 'film-look', 'html', 'music', 'process-audit-cleanup', 'suno']);
} finally {
rmSync(scratch, { recursive: true, force: true });
}
diff --git a/server/envExampleDrift.test.js b/server/envExampleDrift.test.js
index eac55206b8..57a73ea8b4 100644
--- a/server/envExampleDrift.test.js
+++ b/server/envExampleDrift.test.js
@@ -113,6 +113,7 @@ const INHERITED_ENV = {
PGTESTDATABASE: 'test-harness override naming portos_test',
PORTOS_REQUIRE_DB: 'CI flag that turns a skipped DB suite into a failure',
PORTOS_REQUIRE_BROWSER_SUITES: 'CI flag that turns a skipped cross-workspace browser suite into a failure',
+ PORTOS_CI_BROWSER_OWNER: 'CI-only test ownership selector; Linux unit shards exclude suites owned by the browser job',
PORTOS_TEST_PYTHON: 'test-harness interpreter override',
PORTOS_TEST_QUIET: 'test-harness log silencer',
};
diff --git a/server/routes/musicVideoRichAuthoring.browser.test.js b/server/routes/musicVideoRichAuthoring.browser.test.js
index 5eedf1fc22..3a61a43311 100644
--- a/server/routes/musicVideoRichAuthoring.browser.test.js
+++ b/server/routes/musicVideoRichAuthoring.browser.test.js
@@ -54,7 +54,7 @@ afterAll(async () => {
});
describe.skipIf(!canRun)('rich document authoring in a real browser (Chrome, ffmpeg and client dependencies required)', () => {
- it('authors and accepts a Three.js world through the real UI and renders its reviewed proof', async () => {
+ it('[fidelity] authors and accepts a Three.js world through the real UI and renders its reviewed proof', async () => {
// Skipped suites do not run cleanup hooks: load data-owning services only here.
const { PATHS } = await import('../lib/paths.js');
const { default: router } = await import('./musicVideo.js');
diff --git a/server/services/musicVideo/documentRender.browser.test.js b/server/services/musicVideo/documentRender.browser.test.js
index d5c687ad8b..cf10dd5a54 100644
--- a/server/services/musicVideo/documentRender.browser.test.js
+++ b/server/services/musicVideo/documentRender.browser.test.js
@@ -664,7 +664,7 @@ describe.skipIf(!canRun)('layered template with real Chrome and ffmpeg', () => {
await rm(second.outputPath, { force: true });
}, 120000);
- ['ramps', 'generated-shots'].forEach(fixture => it(`matches bounded grades across real composed/document ${fixture} and song-time excerpts`, async () => {
+ ['ramps', 'generated-shots'].forEach(fixture => it(`[fidelity] matches bounded grades across real composed/document ${fixture} and song-time excerpts`, async () => {
// render.js initializes the media registry. Import it only when this test
// runs: a skipped browser suite never executes afterAll cleanup.
const { buildMusicVideoFfmpegArgs } = await import('./render.js');
diff --git a/server/vitest.config.js b/server/vitest.config.js
index bcfbd55c4b..073a1d087c 100644
--- a/server/vitest.config.js
+++ b/server/vitest.config.js
@@ -2,6 +2,7 @@ import { defineConfig } from 'vitest/config';
import { fileURLToPath } from 'node:url';
import { vitestCaptureProjects, vitestCiPool } from '../scripts/vitestCiPool.js';
import { DB_TEST_INCLUDE } from '../scripts/lib/dbTestFiles.js';
+import { BROWSER_SUITES } from '../scripts/lib/browserSuites.js';
import { bootstrapVitestTempRoot } from '../scripts/lib/vitestTempRoot.js';
// `npm run test:fast` is the Windows-safe way to set the flag (a
@@ -39,6 +40,12 @@ process.env.NODE_ENV = 'test';
// engines), so every supported Node build respects it.
process.env.NODE_DISABLE_COMPILE_CACHE = '1';
+// Linux CI gives these registered contracts one browser-owner job. Only the
+// explicitly opted-in unit profile excludes them; local and Windows defaults
+// retain their current collection, including browser suites outside the registry.
+const browserOwnerExcludes = process.platform === 'linux' && process.env.PORTOS_CI_BROWSER_OWNER === 'unit'
+ ? BROWSER_SUITES.map(path => path.slice('server/'.length)) : [];
+
const config = defineConfig({
// Anchor to the config, even when --root or the caller cwd differs.
cacheDir: fileURLToPath(new URL('./.cache/vitest', import.meta.url)),
@@ -138,6 +145,7 @@ const config = defineConfig({
'**/node_modules/**',
'../lib/slashdo/**',
...DB_TEST_INCLUDE,
+ ...browserOwnerExcludes,
...(process.env.VITEST_FAST ? [
'services/worktreeReap.test.js',
'services/sprites/atlas.test.js',
diff --git a/server/vitest.config.test.js b/server/vitest.config.test.js
index 5415ed0a9c..bd435b7f4c 100644
--- a/server/vitest.config.test.js
+++ b/server/vitest.config.test.js
@@ -15,12 +15,18 @@
* `development` value each one starts from.
*/
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
+import { BROWSER_SUITES } from '../scripts/lib/browserSuites.js';
+import { pinPlatform } from './lib/testHelper.js';
+
+let restorePlatform;
beforeEach(() => {
vi.resetModules();
});
afterEach(() => {
+ restorePlatform?.();
+ restorePlatform = undefined;
vi.doUnmock('./vitest.config.db.js');
// Restores the pre-stub value even though the config under test reassigns
// process.env.NODE_ENV directly.
@@ -53,3 +59,39 @@ describe('vitest.config.db.js', () => {
await expectForces('./vitest.config.db.js');
});
});
+
+// Root exclusions apply before full, explicit-file and related selection. Both
+// projects must retain them: the capture group has its own explicit include list.
+describe('browser suite CI ownership', () => {
+ it.each([
+ ['linux', undefined, false],
+ ['linux', 'browser', false],
+ ['linux', 'unit', true],
+ ['win32', 'unit', false],
+ ])('collects the intended contracts on %s with owner %s', async (platform, owner, excluded) => {
+ restorePlatform = pinPlatform(platform);
+ vi.stubEnv('PORTOS_CI_BROWSER_OWNER', owner);
+ const { default: config } = await import('./vitest.config.js');
+ const suites = BROWSER_SUITES.map(path => path.slice('server/'.length));
+ const projects = config.test.projects.map(project => project.test);
+ const capture = projects.find(project => project.name === 'capture');
+ const parallel = projects.find(project => project.name === 'parallel');
+ const expected = excluded ? suites : [];
+ expect(config.test.exclude.filter(path => suites.includes(path))).toEqual(expected);
+ expect(capture.exclude.filter(path => suites.includes(path))).toEqual(expected);
+ for (const suite of suites) {
+ if (excluded) expect(parallel.exclude).toContain(suite);
+ else if (capture.include.includes(suite)) expect(capture.exclude).not.toContain(suite);
+ else {
+ expect(parallel.include).toContain('**/*.test.js');
+ expect(parallel.exclude).not.toContain(suite);
+ }
+ }
+ // This independent browser contract has no registry owner and must still run.
+ const overlay = 'services/musicVideo/overlayTextService.browser.test.js';
+ expect(suites).not.toContain(overlay);
+ expect(config.test.exclude).not.toContain(overlay);
+ expect(capture.include).toContain(overlay);
+ expect(capture.exclude).not.toContain(overlay);
+ });
+});