Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

ci: the interaction harness and the lint plugin narrow only the Vitest lanes (#10065)

Cindy Zhangbf044739328d
diff --git a/.github/actions/test-scope/action.yml b/.github/actions/test-scope/action.yml
index a9d5b434c5..eb8aa34cfb 100644
--- a/.github/actions/test-scope/action.yml
+++ b/.github/actions/test-scope/action.yml
@@ -72,6 +72,12 @@ outputs:
full_suite:
description: The import graph cannot scope this change; run every test.
value: ${{ steps.scope.outputs.full_suite }}
+ vitest_full_suite:
+ description: >-
+ The Vitest lanes run every test. `false` also for shared infrastructure
+ only Vitest's module graph reaches (spec:AST-030 FR13); a base whose
+ classifier predates it reports nothing, which runs every test.
+ value: ${{ steps.scope.outputs.vitest_full_suite }}
release_bump:
description: The PR targets a release branch (the generated version bump).
value: ${{ steps.scope.outputs.release_bump }}
@@ -136,6 +142,7 @@ runs:
tooling_only: 'false',
lab_only: 'false',
full_suite: 'true',
+ vitest_full_suite: 'true',
release_bump: 'false',
touches: 'true',
applies: 'true',
diff --git a/.github/scripts/change-scope.cjs b/.github/scripts/change-scope.cjs
index 9ed398e98a..7c992caf6f 100644
--- a/.github/scripts/change-scope.cjs
+++ b/.github/scripts/change-scope.cjs
@@ -232,6 +232,22 @@ const PACKAGE_INFRA_PATTERNS = [
/^packages\/[^/]+\/[^/]+\.(?:c|m)?js$/,
];
+// Shared infrastructure only Vitest's module graph reaches: the interaction
+// harness and the repository's lint plugin, both ES modules. A change there
+// still runs every other owner as shared infrastructure (the interaction
+// proofs included), but the Vitest lanes run the tests that import it and the
+// guards that read sources (spec:AST-030 FR13). Scripts stay shared: package
+// scripts run generators whose output tests read, and Vitest's graph does not
+// follow `require`.
+const VITEST_GRAPH_INFRA_PATTERNS = [
+ /^internal\/interaction-harness\//,
+ /^internal\/eslint-plugin-astryx\//,
+];
+
+function isVitestGraphInfraPath(filePath) {
+ return VITEST_GRAPH_INFRA_PATTERNS.some(pattern => pattern.test(filePath));
+}
+
// Surfaces a pull request's tests can be scoped to by the import graph (the
// components and modules it changed, and what imports them). Anything else,
// or any infrastructure path above, runs the full suite; Node tooling keeps
@@ -342,6 +358,7 @@ function classifyChanges(
labOnly: false,
// No paths to scope from runs everything (spec:AST-030 FR4).
fullSuite: true,
+ vitestFullSuite: true,
surfaces: complete ? [] : [SURFACES.SHARED_OR_UNKNOWN],
complete,
reason: complete ? 'no changed files' : 'changed-file list is incomplete',
@@ -420,12 +437,16 @@ function classifyChanges(
);
// An incomplete list, or any shared infrastructure, runs everything. A Lab
// entry file of a Lab-only change is not shared infrastructure for it.
- const fullSuite =
+ const sharedInfra = allPaths.filter(
+ filePath =>
+ isSharedInfraPath(filePath) && !(labOnly && labEntries.has(filePath)),
+ );
+ const fullSuite = !complete || sharedInfra.length > 0;
+ // The Vitest lanes run every test unless every shared path is one the
+ // module graph reaches.
+ const vitestFullSuite =
!complete ||
- allPaths.some(
- filePath =>
- isSharedInfraPath(filePath) && !(labOnly && labEntries.has(filePath)),
- );
+ sharedInfra.some(filePath => !isVitestGraphInfraPath(filePath));
return {
specOnly,
toolingOnly,
@@ -435,6 +456,7 @@ function classifyChanges(
docsiteOnly,
labOnly,
fullSuite,
+ vitestFullSuite,
surfaces,
complete,
reason: !complete
@@ -529,7 +551,7 @@ if (require.main === module) {
throw new Error('GITHUB_OUTPUT is required with --github-output.');
fs.appendFileSync(
outputPath,
- `spec_only=${result.specOnly}\ndocsite_only=${result.docsiteOnly}\ntooling_only=${result.toolingOnly}\nlab_only=${result.labOnly}\nfull_suite=${result.fullSuite}\n`,
+ `spec_only=${result.specOnly}\ndocsite_only=${result.docsiteOnly}\ntooling_only=${result.toolingOnly}\nlab_only=${result.labOnly}\nfull_suite=${result.fullSuite}\nvitest_full_suite=${result.vitestFullSuite}\n`,
);
} else {
process.stdout.write(`${JSON.stringify(result)}\n`);
@@ -549,6 +571,7 @@ module.exports = {
isNodeToolingPath,
isPackageReleasePath,
isSharedInfraPath,
+ isVitestGraphInfraPath,
isLabEntryChange,
isSpecRecordPath,
labRouteSlug,
diff --git a/.github/scripts/change-scope.test.mjs b/.github/scripts/change-scope.test.mjs
index 09016f4c3b..7b0d092740 100644
--- a/.github/scripts/change-scope.test.mjs
+++ b/.github/scripts/change-scope.test.mjs
@@ -1,6 +1,10 @@
// Copyright (c) Meta Platforms, Inc. and affiliates.
+import {spawnSync} from 'node:child_process';
+import fs from 'node:fs';
import {createRequire} from 'node:module';
+import os from 'node:os';
+import path from 'node:path';
import {describe, expect, it} from 'vitest';
const require = createRequire(import.meta.url);
@@ -506,6 +510,86 @@ describe('full-suite scope', () => {
});
});
+describe('shared infrastructure only the Vitest module graph reaches (spec:AST-030 FR13)', () => {
+ const scope = files => classifyChanges(files.map(filename => ({filename})));
+
+ it('narrows only the Vitest lanes for the interaction harness and the lint plugin', () => {
+ for (const file of [
+ 'internal/interaction-harness/src/busy.ts',
+ 'internal/eslint-plugin-astryx/rules/no-internal-barrel-import.js',
+ ]) {
+ const result = scope(['packages/core/src/Button/Button.tsx', file]);
+ // Every other owner still runs as for shared infrastructure.
+ expect(result.fullSuite, file).toBe(true);
+ expect(result.vitestFullSuite, file).toBe(false);
+ }
+ });
+
+ it('keeps every other shared path, and a mix with one, on every Vitest test', () => {
+ for (const file of [
+ 'internal/a11y-spec/src/index.ts',
+ 'internal/test-utils/src/setup.ts',
+ 'eslint.config.js',
+ 'package.json',
+ 'vitest.config.ts',
+ '.github/workflows/ci.yml',
+ // Scripts stay shared: generators run by package scripts, and
+ // CommonJS files Vitest's graph does not follow.
+ 'scripts/generate-i18n-runtime.mjs',
+ 'scripts/build-css.mjs',
+ '.github/scripts/run-labels.cjs',
+ '.github/scripts/fs-guard-tests.mjs',
+ 'apps/sandbox/scripts/check-sandbox.mjs',
+ 'scripts/score-ledger.mjs',
+ 'internal/scripts/lib/a-future-strategy.mjs',
+ ]) {
+ expect(scope([file]).vitestFullSuite, file).toBe(true);
+ expect(
+ scope(['internal/interaction-harness/src/busy.ts', file])
+ .vitestFullSuite,
+ file,
+ ).toBe(true);
+ }
+ });
+
+ it('agrees with the full suite when nothing is shared or the list is incomplete', () => {
+ expect(scope(['packages/core/src/Button/Button.tsx']).vitestFullSuite).toBe(
+ false,
+ );
+ expect(
+ classifyChanges(
+ [{filename: 'internal/interaction-harness/src/busy.ts'}],
+ {expectedCount: 2},
+ ).vitestFullSuite,
+ ).toBe(true);
+ expect(classifyChanges([]).vitestFullSuite).toBe(true);
+ });
+
+ it('reports vitest_full_suite to the workflow', () => {
+ const out = path.join(
+ fs.mkdtempSync(path.join(os.tmpdir(), 'scope-')),
+ 'out',
+ );
+ const result = spawnSync(
+ 'node',
+ [
+ new URL('./change-scope.cjs', import.meta.url).pathname,
+ '--github-output',
+ ],
+ {
+ input:
+ 'M\tpackages/core/src/Button/Button.tsx\nM\tinternal/interaction-harness/src/busy.ts\n',
+ env: {...process.env, GITHUB_OUTPUT: out},
+ encoding: 'utf8',
+ },
+ );
+ expect(result.status).toBe(0);
+ const text = fs.readFileSync(out, 'utf8');
+ expect(text).toContain('full_suite=true\n');
+ expect(text).toContain('vitest_full_suite=false\n');
+ });
+});
+
describe('Lab-only change scope (spec:AST-030 FR26)', () => {
const lab = [
'packages/lab/src/Tour/Tour.tsx',
diff --git a/.github/scripts/ci-test-routing.test.mjs b/.github/scripts/ci-test-routing.test.mjs
index 79b0fd8cd7..4e42c2a9fb 100644
--- a/.github/scripts/ci-test-routing.test.mjs
+++ b/.github/scripts/ci-test-routing.test.mjs
@@ -229,7 +229,7 @@ describe.each(Object.entries(WORKFLOWS))(
].join('\n'),
);
expect(share.env.SCOPED).toBe(
- "${{ github.event_name == 'pull_request' && steps.scope.outputs.full_suite == 'false' }}",
+ "${{ github.event_name == 'pull_request' && steps.scope.outputs.vitest_full_suite == 'false' }}",
);
// The Node lane runs the same scoped share, guards included.
const node = workflow.jobs['test-node'].steps.find(
@@ -402,6 +402,29 @@ describe.each(Object.entries(WORKFLOWS))(
},
);
+describe('Vitest scoping for shared infrastructure the module graph reaches (spec:AST-030 FR13)', () => {
+ const workflow = load('ci.yml');
+ const scopedEnv = (job, name) =>
+ workflow.jobs[job].steps.find(step => step.name === name).env.SCOPED;
+
+ it('scopes the UI and Node Vitest lanes by vitest_full_suite', () => {
+ for (const [job, name] of [
+ ['test-ui', 'Run this shard of the UI tests'],
+ ['test-node', 'Run this shard of the Node tests'],
+ ]) {
+ expect(scopedEnv(job, name)).toBe(
+ "${{ github.event_name == 'pull_request' && steps.scope.outputs.vitest_full_suite == 'false' }}",
+ );
+ }
+ });
+
+ it('keeps every other owner on full_suite', () => {
+ const text = read('.github/workflows/ci.yml');
+ expect(text.match(/outputs\.vitest_full_suite/g)).toHaveLength(2);
+ expect(text).toContain("steps.scope.outputs.full_suite == 'false'");
+ });
+});
+
describe('ci.yml pr-a11y browser specs', () => {
const steps = load('ci.yml').jobs['pr-a11y-specs'].steps;
const run = steps.find(
diff --git a/.github/scripts/spec-owner-workflow.test.mjs b/.github/scripts/spec-owner-workflow.test.mjs
index e4e56155ba..77bf612833 100644
--- a/.github/scripts/spec-owner-workflow.test.mjs
+++ b/.github/scripts/spec-owner-workflow.test.mjs
@@ -93,7 +93,7 @@ describe('spec-only workflow contract', () => {
);
expect(result.status, result.stderr).toBe(0);
expect(fs.readFileSync(output, 'utf8')).toBe(
- 'spec_only=true\ndocsite_only=false\ntooling_only=false\nlab_only=false\nfull_suite=false\n',
+ 'spec_only=true\ndocsite_only=false\ntooling_only=false\nlab_only=false\nfull_suite=false\nvitest_full_suite=false\n',
);
fs.rmSync(path.join(classifierDir, 'knowledge-paths.cjs'));
diff --git a/.github/scripts/test-scope-action.test.mjs b/.github/scripts/test-scope-action.test.mjs
index 615a27296a..caa564da49 100644
--- a/.github/scripts/test-scope-action.test.mjs
+++ b/.github/scripts/test-scope-action.test.mjs
@@ -181,6 +181,7 @@ describe('per-job scope step', () => {
'spec_only',
'tooling_only',
'touches',
+ 'vitest_full_suite',
].sort(),
);
});
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 97f9b22de4..051ce168ff 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -965,11 +965,13 @@ jobs:
# A scoped pull request runs the tests Vitest's module graph relates to
# its diff against the base (`--changed`): the changed tests, and every
- # test that imports a changed module, however indirectly. Anything else
- # runs every test.
+ # test that imports a changed module, however indirectly. That includes
+ # shared infrastructure only the module graph reaches (the interaction
+ # harness and the lint plugin: `vitest_full_suite`).
+ # Anything else runs every test.
- name: Run this shard of the UI tests
env:
- SCOPED: ${{ github.event_name == 'pull_request' && steps.scope.outputs.full_suite == 'false' }}
+ SCOPED: ${{ github.event_name == 'pull_request' && steps.scope.outputs.vitest_full_suite == 'false' }}
LAB_ONLY: ${{ steps.scope.outputs.lab_only }}
run: |
if [ "$LAB_ONLY" = true ]; then
@@ -1109,7 +1111,7 @@ jobs:
# Scoped like test-ui's run.
- name: Run this shard of the Node tests
env:
- SCOPED: ${{ github.event_name == 'pull_request' && steps.scope.outputs.full_suite == 'false' }}
+ SCOPED: ${{ github.event_name == 'pull_request' && steps.scope.outputs.vitest_full_suite == 'false' }}
run: |
if [ "$SCOPED" = true ]; then
.github/scripts/run-named.sh "Tests the change reaches" -- pnpm test:node --changed=HEAD^1 --passWithNoTests --shard=${{ matrix.shard }}/${{ strategy.job-total }}
diff --git a/docs/specs/AST-030/spec.md b/docs/specs/AST-030/spec.md
index e8a4f2511c..2ae8b5d088 100644
--- a/docs/specs/AST-030/spec.md
+++ b/docs/specs/AST-030/spec.md
@@ -331,7 +331,13 @@ they are separately named.
`node-tooling` paths (FR8 keeps their whole Node project), and every other
`shared-or-unknown` path. Changesets are excepted, and so, in a Lab-only
change, are a carried Lab component's entry files (FR26), the route lock
- under `internal/` included. Every fail-closed
+ under `internal/` included. The interaction harness
+ (`internal/interaction-harness/`) and the repository's lint plugin
+ (`internal/eslint-plugin-astryx/`) narrow only the Vitest test owner, to the
+ tests whose module graph reaches the change and every test that reads
+ repository sources; every other owner runs as for shared infrastructure, all
+ interaction proofs and the repository-wide lint included.
+ A change with any other shared path runs every Vitest test. Every fail-closed
condition of FR3–FR4 also runs the full suite. `main`, release branches,
merge-group entries and `release-check` dispatches always run the full suite,
and release checks keep FR12's full scope.
@@ -644,7 +650,7 @@ The nightly perf lane is the one scheduled measurement outside them (FR18).
| FR12 | Storybook visual-plan, canonical workflow, and repository-owner guard tests | PR scope; full-plan maintenance; report-only publication; deployment/a11y/RTL/vibe artifacts | another workflow captures or compares pixels, a renamed or aliased owner escapes the guard, or legitimate non-visual artifacts are rejected |
| FR12 | `touched-components.test.mjs` changed-spec routing; `a11y-pr-contracts.test.mjs` `changedOnly` cases; `a11y-changed-stories.test.mjs` and the touched job's story-audit contract | a pull request that adds a browser accessibility spec; one that changes an `always` spec; one that changes a `release-only` or an `audit` spec; one that changes more spec files than the limit; a component change with no spec change; a pull request that adds an `a11y/` story no component owns; one that adds a story to an untouched component's story file beside a touched component | a pull request that adds a browser accessibility spec that fails is not caught on that pull request, a spec of a component that is neither changed, touched nor rendering a touched component runs in the contract job, or an unchanged `audit` spec does, a not-run file reads as passed or fails the join, a failing contract leaves the `test` join green, or a new violation in a changed story no component owns passes the touched check |
| FR12 | `a11y-pr-contracts.test.mjs` ownership cases; the CI workflow contract; contract-job and unit-test-job timings on 17 component pull requests | a pull request that touches one component with an owned `release-only` contract, passing and failing; a contract file that cannot finish within the budget; a wide change; a reverse dependent of the touched component; measured: setup up to about 1.5 minutes plus the 8-minute stop, against the slowest unit-test job finishing 9.9 to 25 minutes after the Storybook build; a wide change touching Spinner (its own specs run); the folders that render a touched component, nearest first, type-only imports left out, within a one-minute share, and one that does not fit (reported as not run) | a touched component's owned `release-only` contract does not run on its pull request, a failing contract leaves the `test` join green, a skipped file is split, does not show as "not run" in the check's summary or fails the join, the contract job finishes after the slowest unit-test job, or a touched component's own spec does not run on a wide change, or a rendering folder's spec runs before an owned one |
-| FR13 | classifier, workflow contract, component-analysis and RTL-scope tests | component change; shared module; Changeset beside code; infrastructure path; incomplete list; no build | an infrastructure path is scoped, a file-system guard test is skipped, a reverse dependent leaves visual/RTL or enters PR accessibility, or a full-suite event is narrowed |
+| FR13 | classifier, workflow contract, component-analysis and RTL-scope tests | component change; shared module; Changeset beside code; infrastructure path; incomplete list; no build; an interaction-harness or lint-plugin path beside code, alone and with another shared path, and a script or other shared path beside one | an infrastructure path is scoped, a file-system guard test is skipped, a reverse dependent leaves visual/RTL or enters PR accessibility, or a full-suite event is narrowed |
| FR14 | `workflow-paths.test.mjs` over every workflow | path allowlist; required-context workflow | a `paths-ignore`, a match-all or exclusion glob, or a path filter on a workflow that reports a required context fails |
| FR15 | workflow contract and reconciler tests | ENGOWNER spec PR; owner command; other comment; fork review; non-spec PR; run over the request budget; maintainer dispatch; fork PR, code-only: an owner's exact-head approval plus the owner command clears the code half | `review-required` succeeds while spec approval is missing, another workflow publishes a review context, a fork review starts the spec checks, a non-command comment starts a runner, an over-budget run decides, or a fork's approved code-only PR stays pending after the owner command |
| FR16 | `heavy-caps-action.test.mjs` cap cases and heavy-job contract; `ci-admission.test.mjs` | sixteenth-oldest PR; 101st heavy run of the day, beside cancelled, light and other-branch runs; agent lane and its five-PR and forty-run backstops; an engineering owner, including the shared login past those backstops, a commented-out or trailing-comment handle and an unreadable roster; bot; the deciding run; re-run; failed lookup; a run with room, without room, past its longest wait, beside a waiting `main` check, as a draft or unlabelled fork, with opt-in labels, beside small workflows past their measured most, and with the API unreadable; each workflow's most jobs; every required context's rule; a full queue: the capacity tool and admission agree there is no room | a heavy suite that runs ahead of the author's fifteen oldest PRs, or past the daily cap, a capped job that passes, an engineering owner or a re-run that is held, or a roster read from the pull request's head, a run that starts without room before its longest wait, a pull-request run that starts while `main` has waited under fifteen minutes, a job admission could skip, a peak below the jobs its run holds, or a job on a slim runner planned past its fifteen-minute cap fails |
@@ -816,7 +822,9 @@ A pull request's Vitest legs and browser audits follow what its change reaches:
the module graph for tests and built-bundle reach for components. Tests that
read sources through the file system run whole, because no import reaches what
they read. Shared infrastructure, uncertainty, `main`, merge-group entries and
-release checks run everything, so no landing or release gate narrows; a pull
+release checks run everything, except where FR13 narrows the Vitest owner for
+the interaction harness and the lint plugin, so no landing or release gate
+narrows; a pull
request runs the visual, accessibility and RTL owners at their opted-in scope
only with the suite's `ci:run-*` label, and otherwise only the
touched-components check for the components it touches (DEC-7).