Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
TypeScript automationchromechromiume2e-testingelectronend-to-end-testingfirefoxjavascriptplaywrighttesttest-automationtestingtesting-toolswebwebkit
feat(snapshots): add --update-snapshots=missing that passes tests (#42662)
Pavel Feldman858526b31def
diff --git a/docs/src/aria-snapshots.md b/docs/src/aria-snapshots.mdindex e1361133ef..26c37e9f48 100644--- a/docs/src/aria-snapshots.md+++ b/docs/src/aria-snapshots.md@@ -369,9 +369,10 @@ accessible names to aid snapshot creation and review.When using the Playwright test runner (`@playwright/test`), you can automatically update snapshots with the `--update-snapshots` flag, `-u` for short.-Running without the flag uses `missing` mode and generates only absent snapshots. Supplying `-u` uses `changed` mode-and updates snapshots that do not match. Use `all` to regenerate every snapshot, including matching ones, or `none`-to prevent snapshot updates.+Running without the flag uses `default` mode: absent snapshots are generated, but the tests that generate them fail,+so that a run with new snapshots does not silently pass. Use `missing` to generate absent snapshots and keep those+tests passing. Supplying `-u` uses `changed` mode and updates snapshots that do not match. Use `all` to regenerate+every snapshot, including matching ones, or `none` to prevent snapshot updates.```bashnpx playwright test --update-snapshots@@ -389,7 +390,7 @@ Passing an empty string as the template in an assertion generates a snapshot on-await expect(locator).toMatchAriaSnapshot('');```-This works without `-u` because the default mode is `missing`.+This works without `-u` because the default mode generates missing snapshots.#### Snapshot patch filesdiff --git a/docs/src/test-api/class-fullconfig.md b/docs/src/test-api/class-fullconfig.mdindex f9edb15435..0e89fc9fb8 100644--- a/docs/src/test-api/class-fullconfig.md+++ b/docs/src/test-api/class-fullconfig.md@@ -143,7 +143,7 @@ Resolved global tags. See [`property: TestConfig.tag`].## property: FullConfig.updateSnapshots* since: v1.10-- type: <[UpdateSnapshots]<"all"|"changed"|"missing"|"none">>+- type: <[UpdateSnapshots]<"all"|"changed"|"missing"|"none"|"default">>See [`property: TestConfig.updateSnapshots`].diff --git a/docs/src/test-api/class-testconfig.md b/docs/src/test-api/class-testconfig.mdindex 0696f545a1..443333a8f2 100644--- a/docs/src/test-api/class-testconfig.md+++ b/docs/src/test-api/class-testconfig.md@@ -664,13 +664,14 @@ export default defineConfig({## property: TestConfig.updateSnapshots* since: v1.10-- type: ?<[UpdateSnapshots]<"all"|"changed"|"missing"|"none">>+- type: ?<[UpdateSnapshots]<"all"|"changed"|"missing"|"none"|"default">>-Whether to update expected snapshots with the actual results produced by the test run. Defaults to `'missing'`.+Whether to update expected snapshots with the actual results produced by the test run. Defaults to `'default'`.* `'all'` - All tests that are executed will update snapshots.* `'changed'` - All tests that are executed will update snapshots that did not match. Matching snapshots will not be updated. Also creates missing snapshots.-* `'missing'` - Missing snapshots are created, for example when authoring a new test and running it for the first time. This is the default.+* `'missing'` - Missing snapshots are created, for example when authoring a new test and running it for the first time. Tests that only create missing snapshots pass.* `'none'` - No snapshots are updated.+* `'default'` - Missing snapshots are created, but the tests that create them fail, so that the run does not silently pass in CI. This is the default.Learn more about [snapshots](../test-snapshots.md).@@ -680,7 +681,7 @@ Learn more about [snapshots](../test-snapshots.md).import { defineConfig } from '@playwright/test';export default defineConfig({- updateSnapshots: 'missing',+ updateSnapshots: 'default',});```diff --git a/docs/src/test-cli-js.md b/docs/src/test-cli-js.mdindex 7c187b1433..637d075a32 100644--- a/docs/src/test-cli-js.md+++ b/docs/src/test-cli-js.md@@ -113,7 +113,7 @@ npx playwright test --ui| `--ui` | Run tests in interactive UI mode. || `--ui-host <host>` | Host to serve UI on; specifying this option opens UI in a browser tab. || `--ui-port <port>` | Port to serve UI on, 0 for any free port; specifying this option opens UI in a browser tab. |-| `-u` or `--update-snapshots [mode]` | Update snapshots with actual results. Possible values are "all", "changed", "missing", and "none". Running tests without the flag defaults to "missing"; running tests with the flag but without a value defaults to "changed". |+| `-u` or `--update-snapshots [mode]` | Update snapshots with actual results. Possible values are "all", "changed", "missing", "none" and "default". Running tests without the flag defaults to "default"; running tests with the flag but without a value defaults to "changed". || `--update-source-method [mode]` | Update snapshots with actual results. Possible values are "patch" (default), "3way" and "overwrite". "Patch" creates a unified diff file that can be used to update the source code later. "3way" generates merge conflict markers in source code. "Overwrite" overwrites the source code with the new snapshot values.|| `-x` | Stop after the first failure. |diff --git a/packages/playwright/src/common/config.ts b/packages/playwright/src/common/config.tsindex 87713e0404..e139194f08 100644--- a/packages/playwright/src/common/config.ts+++ b/packages/playwright/src/common/config.ts@@ -105,7 +105,7 @@ export class FullConfigInternal {reportSlowTests: takeFirst(userConfig.reportSlowTests, { max: 5, threshold: 300_000 /* 5 minutes */ }),shard: takeFirst(configCLIOverrides.shard, userConfig.shard, null),tags: globalTags,- updateSnapshots: takeFirst(configCLIOverrides.updateSnapshots, userConfig.updateSnapshots, 'missing'),+ updateSnapshots: takeFirst(configCLIOverrides.updateSnapshots, userConfig.updateSnapshots, 'default'),updateSourceMethod: takeFirst(configCLIOverrides.updateSourceMethod, userConfig.updateSourceMethod, 'patch'),version: packageJSON.version,workers: resolveWorkers(takeFirst((configCLIOverrides.debug || configCLIOverrides.pause) ? 1 : undefined, configCLIOverrides.workers, userConfig.workers, '50%')),diff --git a/packages/playwright/src/common/configLoader.ts b/packages/playwright/src/common/configLoader.tsindex 5461d4014c..2cc4606103 100644--- a/packages/playwright/src/common/configLoader.ts+++ b/packages/playwright/src/common/configLoader.ts@@ -238,8 +238,8 @@ function validateConfig(file: string, config: Config) {}if ('updateSnapshots' in config && config.updateSnapshots !== undefined) {- if (typeof config.updateSnapshots !== 'string' || !['all', 'changed', 'missing', 'none'].includes(config.updateSnapshots))- throw errorWithFile(file, `config.updateSnapshots must be one of "all", "changed", "missing" or "none"`);+ if (typeof config.updateSnapshots !== 'string' || !['all', 'changed', 'missing', 'none', 'default'].includes(config.updateSnapshots))+ throw errorWithFile(file, `config.updateSnapshots must be one of "all", "changed", "missing", "none" or "default"`);}if ('retryStrategy' in config && config.retryStrategy !== undefined) {diff --git a/packages/playwright/src/common/ipc.ts b/packages/playwright/src/common/ipc.tsindex 8ca7824f9e..975fa56d0b 100644--- a/packages/playwright/src/common/ipc.ts+++ b/packages/playwright/src/common/ipc.ts@@ -41,7 +41,7 @@ export type ConfigCLIOverrides = {timeout?: number;tsconfig?: string;ignoreSnapshots?: boolean;- updateSnapshots?: 'all' | 'changed' | 'missing' | 'none';+ updateSnapshots?: 'all' | 'changed' | 'missing' | 'none' | 'default';updateSourceMethod?: 'overwrite' | 'patch' | '3way';workers?: number | string;projects?: { name: string, use?: any }[],diff --git a/packages/playwright/src/isomorphic/teleReceiver.ts b/packages/playwright/src/isomorphic/teleReceiver.tsindex 845366c006..f4c76d5b2d 100644--- a/packages/playwright/src/isomorphic/teleReceiver.ts+++ b/packages/playwright/src/isomorphic/teleReceiver.ts@@ -833,7 +833,7 @@ export const baseFullConfig: reporterTypes.FullConfig = {quiet: false,shard: null,tags: [],- updateSnapshots: 'missing',+ updateSnapshots: 'default',updateSourceMethod: 'patch',version: '',workers: 0,diff --git a/packages/playwright/src/isomorphic/testServerInterface.ts b/packages/playwright/src/isomorphic/testServerInterface.tsindex 2450cbdc67..0e3fe05899 100644--- a/packages/playwright/src/isomorphic/testServerInterface.ts+++ b/packages/playwright/src/isomorphic/testServerInterface.ts@@ -86,7 +86,7 @@ export interface TestServerInterface {headed?: boolean;workers?: number | string;maxFailures?: number;- updateSnapshots?: 'all' | 'changed' | 'missing' | 'none';+ updateSnapshots?: 'all' | 'changed' | 'missing' | 'none' | 'default';updateSourceMethod?: 'overwrite' | 'patch' | '3way';reporters?: string[],trace?: 'on' | 'off';diff --git a/packages/playwright/src/matchers/expect.ts b/packages/playwright/src/matchers/expect.tsindex 1a2938aa5a..a9e8191b08 100644--- a/packages/playwright/src/matchers/expect.ts+++ b/packages/playwright/src/matchers/expect.ts@@ -114,7 +114,7 @@ export type ExpectConfig = {testInfo: ExpectTestInfo | null;filteredStackTrace: (rawStack: string[]) => StackFrame[];ignoreSnapshots: boolean;- updateSnapshots: 'all' | 'changed' | 'missing' | 'none';+ updateSnapshots: 'all' | 'changed' | 'missing' | 'none' | 'default';timeout?: number;toHaveScreenshot?: {threshold?: number;@@ -144,7 +144,7 @@ function unfilteredStackTrace(rawStack: string[]): StackFrame[] {return rawStack.map(frame => parseStackFrame(frame)).filter(f => !!f);}-let _expectConfig: ExpectConfig = { testInfo: null, filteredStackTrace: unfilteredStackTrace, ignoreSnapshots: false, updateSnapshots: 'missing' };+let _expectConfig: ExpectConfig = { testInfo: null, filteredStackTrace: unfilteredStackTrace, ignoreSnapshots: false, updateSnapshots: 'default' };export function setExpectConfig(config: ExpectConfig) {_expectConfig = config;diff --git a/packages/playwright/src/matchers/toMatchAriaSnapshot.ts b/packages/playwright/src/matchers/toMatchAriaSnapshot.tsindex d4d5c2a941..97fc02331a 100644--- a/packages/playwright/src/matchers/toMatchAriaSnapshot.ts+++ b/packages/playwright/src/matchers/toMatchAriaSnapshot.ts@@ -74,7 +74,7 @@ export async function toMatchAriaSnapshot(timeout = expectedParam?.timeout ?? this.timeout;}- const isMissingBaseline = updateSnapshots === 'missing' && !expected;+ const isMissingBaseline = (updateSnapshots === 'missing' || updateSnapshots === 'default') && !expected;if (isMissingBaseline && this.isNot) {const message = `Matchers using ".not" can't generate new baselines`;return { pass: this.isNot, message: () => message, name: 'toMatchAriaSnapshot' };@@ -137,7 +137,11 @@ export async function toMatchAriaSnapshot(const relativePath = path.relative(process.cwd(), expectedPath);if (isMissingBaseline) {const message = `A snapshot doesn't exist at ${relativePath}, writing actual.`;- return { pass: true, message: () => '', name: 'toMatchAriaSnapshot', softError: new Error(message), shouldNotRetryTest: true };+ if (updateSnapshots === 'default')+ return { pass: true, message: () => '', name: 'toMatchAriaSnapshot', softError: new Error(message), shouldNotRetryTest: true };+ /* eslint-disable no-console */+ console.log(message);+ return { pass: true, message: () => '', name: 'toMatchAriaSnapshot' };}const message = `A snapshot is generated at ${relativePath}.`;/* eslint-disable no-console */@@ -145,7 +149,7 @@ export async function toMatchAriaSnapshot(return { pass: true, message: () => '', name: 'toMatchAriaSnapshot' };} else {const suggestedRebaseline = `\`\n${escapeTemplateString(indent(typedReceived.regex, '{indent} '))}\n{indent}\``;- if (isMissingBaseline) {+ if (isMissingBaseline && updateSnapshots === 'default') {const message = 'A snapshot is not provided, generating new baseline.';return { pass: true, message: () => '', name: 'toMatchAriaSnapshot', suggestedRebaseline, softError: new Error(message), shouldNotRetryTest: true };}diff --git a/packages/playwright/src/matchers/toMatchSnapshot.ts b/packages/playwright/src/matchers/toMatchSnapshot.tsindex 95dec3bf30..0b54eb5af0 100644--- a/packages/playwright/src/matchers/toMatchSnapshot.ts+++ b/packages/playwright/src/matchers/toMatchSnapshot.ts@@ -80,7 +80,7 @@ class SnapshotHelper {readonly diffPath: string;readonly mimeType: string;readonly kind: 'Screenshot'|'Snapshot';- readonly updateSnapshots: 'all' | 'changed' | 'missing' | 'none';+ readonly updateSnapshots: 'all' | 'changed' | 'missing' | 'none' | 'default';readonly comparator: Comparator;readonly options: Omit<ToHaveScreenshotOptions, '_comparator'> & { comparator?: string };readonly matcherName: string;@@ -196,12 +196,12 @@ class SnapshotHelper {writeFileSync(this.actualPath, actual);attachments.push({ name: addSuffixToFilePath(this.attachmentBaseName, '-actual'), contentType: this.mimeType, path: this.actualPath });const message = `A snapshot doesn't exist at ${this.expectedPath}${isWriteMissingMode ? ', writing actual.' : '.'}`;- if (this.updateSnapshots === 'all' || this.updateSnapshots === 'changed') {+ if (this.updateSnapshots === 'all' || this.updateSnapshots === 'changed' || this.updateSnapshots === 'missing') {/* eslint-disable no-console */console.log(message);return this.createMatcherResult(message, true, undefined, attachments);}- if (this.updateSnapshots === 'missing') {+ if (this.updateSnapshots === 'default') {return {...this.createMatcherResult('', true, undefined, attachments),softError: new Error(message),diff --git a/packages/playwright/src/program.ts b/packages/playwright/src/program.tsindex 4d88f7440a..2bedd91609 100644--- a/packages/playwright/src/program.ts+++ b/packages/playwright/src/program.ts@@ -241,7 +241,7 @@ const testOptions: [string, { description: string, choices?: string[], preset?:['--ui', { description: `Run tests in interactive UI mode` }],['--ui-host <host>', { description: `Host to serve UI on; specifying this option opens UI in a browser tab` }],['--ui-port <port>', { description: `Port to serve UI on, 0 for any free port; specifying this option opens UI in a browser tab` }],- ['-u, --update-snapshots [mode]', { description: `Update snapshots with actual results. Running tests without the flag defaults to "missing"`, choices: ['all', 'changed', 'missing', 'none'], preset: 'changed' }],+ ['-u, --update-snapshots [mode]', { description: `Update snapshots with actual results. Running tests without the flag defaults to "default"`, choices: ['all', 'changed', 'missing', 'none', 'default'], preset: 'changed' }],['--update-source-method <method>', { description: `Chooses the way source is updated (default: "patch")`, choices: ['overwrite', '3way', 'patch'] }],['-j, --workers <workers>', { description: `Number of concurrent workers or percentage of logical CPU cores, use 1 to run in a single worker (default: 50%)` }],['-x', { description: `Stop after the first failure` }],diff --git a/packages/playwright/src/runner/testRunner.ts b/packages/playwright/src/runner/testRunner.tsindex d8f8451927..5bef691fc9 100644--- a/packages/playwright/src/runner/testRunner.ts+++ b/packages/playwright/src/runner/testRunner.ts@@ -68,7 +68,7 @@ export type RunTestsParams = {headed?: boolean;workers?: number | string;maxFailures?: number;- updateSnapshots?: 'all' | 'changed' | 'missing' | 'none';+ updateSnapshots?: 'all' | 'changed' | 'missing' | 'none' | 'default';updateSourceMethod?: 'overwrite' | 'patch' | '3way';reporters?: string[],trace?: 'on' | 'off';diff --git a/packages/playwright/src/runner/testServer.ts b/packages/playwright/src/runner/testServer.tsindex 5a31cd6f98..a88157bb79 100644--- a/packages/playwright/src/runner/testServer.ts+++ b/packages/playwright/src/runner/testServer.ts@@ -100,7 +100,7 @@ export type RunTestsParams = {testIds?: string[];headed?: boolean;workers?: number | string;- updateSnapshots?: 'all' | 'changed' | 'missing' | 'none';+ updateSnapshots?: 'all' | 'changed' | 'missing' | 'none' | 'default';updateSourceMethod?: 'overwrite' | 'patch' | '3way';reporters?: string[],trace?: 'on' | 'off';diff --git a/packages/playwright/src/worker/workerMain.ts b/packages/playwright/src/worker/workerMain.tsindex bd73bcdbac..8703d27a69 100644--- a/packages/playwright/src/worker/workerMain.ts+++ b/packages/playwright/src/worker/workerMain.ts@@ -526,7 +526,7 @@ export class WorkerMain extends ProcessRunner {this._currentTest = null;globals.setCurrentTestInfo(null);- setExpectConfig({ testInfo: null, filteredStackTrace, ignoreSnapshots: false, updateSnapshots: 'missing' });+ setExpectConfig({ testInfo: null, filteredStackTrace, ignoreSnapshots: false, updateSnapshots: 'default' });this.dispatchEvent('testEnd', buildTestEndPayload(testInfo));const preserveOutput = this._config.config.preserveOutput === 'always' ||diff --git a/packages/playwright/types/test.d.ts b/packages/playwright/types/test.d.tsindex e95a0f6015..26f666c7fb 100644--- a/packages/playwright/types/test.d.ts+++ b/packages/playwright/types/test.d.ts@@ -1967,13 +1967,15 @@ interface TestConfig<TestArgs = {}, WorkerArgs = {}> {tsconfig?: string;/**- * Whether to update expected snapshots with the actual results produced by the test run. Defaults to `'missing'`.+ * Whether to update expected snapshots with the actual results produced by the test run. Defaults to `'default'`.* - `'all'` - All tests that are executed will update snapshots.* - `'changed'` - All tests that are executed will update snapshots that did not match. Matching snapshots will not* be updated. Also creates missing snapshots.* - `'missing'` - Missing snapshots are created, for example when authoring a new test and running it for the first- * time. This is the default.+ * time. Tests that only create missing snapshots pass.* - `'none'` - No snapshots are updated.+ * - `'default'` - Missing snapshots are created, but the tests that create them fail, so that the run does not+ * silently pass in CI. This is the default.** Learn more about [snapshots](https://playwright.dev/docs/test-snapshots).*@@ -1984,12 +1986,12 @@ interface TestConfig<TestArgs = {}, WorkerArgs = {}> {* import { defineConfig } from '@playwright/test';** export default defineConfig({- * updateSnapshots: 'missing',+ * updateSnapshots: 'default',* });* ```**/- updateSnapshots?: "all"|"changed"|"missing"|"none";+ updateSnapshots?: "all"|"changed"|"missing"|"none"|"default";/*** Defines how to update snapshots in the source code.@@ -2173,7 +2175,7 @@ export interface FullConfig<TestArgs = {}, WorkerArgs = {}> {/*** See [testConfig.updateSnapshots](https://playwright.dev/docs/api/class-testconfig#test-config-update-snapshots).*/- updateSnapshots: "all"|"changed"|"missing"|"none";+ updateSnapshots: "all"|"changed"|"missing"|"none"|"default";/*** Seediff --git a/packages/trace-viewer/src/ui/uiModeView.tsx b/packages/trace-viewer/src/ui/uiModeView.tsxindex 3bf380dd27..a345a15dd3 100644--- a/packages/trace-viewer/src/ui/uiModeView.tsx+++ b/packages/trace-viewer/src/ui/uiModeView.tsx@@ -61,7 +61,7 @@ const queryParams = {reporters: searchParams.has('reporter') ? searchParams.getAll('reporter') : undefined,pathSeparator: searchParams.get('pathSeparator') || '/',};-if (queryParams.updateSnapshots && !['all', 'changed', 'none', 'missing'].includes(queryParams.updateSnapshots))+if (queryParams.updateSnapshots && !['all', 'changed', 'none', 'missing', 'default'].includes(queryParams.updateSnapshots))queryParams.updateSnapshots = undefined;const isMac = navigator.platform === 'MacIntel';@@ -129,7 +129,7 @@ export const UIModeView: React.FC<{}> = ({const onRevealSource = React.useCallback(() => setRevealSource(true), [setRevealSource]);const [singleWorker, setSingleWorker] = useSetting<boolean>('single-worker', false);- const [updateSnapshots, setUpdateSnapshots] = useSetting<reporterTypes.FullConfig['updateSnapshots']>('updateSnapshots', 'missing');+ const [updateSnapshots, setUpdateSnapshots] = useSetting<reporterTypes.FullConfig['updateSnapshots']>('updateSnapshots', 'default');const [onlyChanged, setOnlyChanged] = useSetting<boolean>('only-changed', false);const [stopOnFailure, setStopOnFailure] = useSetting<boolean>('stop-on-failure', false);const [mergeFiles] = useSetting('mergeFiles', false);@@ -553,6 +553,7 @@ export const UIModeView: React.FC<{}> = ({{ type: 'check', value: singleWorker, set: setSingleWorker, name: 'Single worker' },{ type: 'check', value: stopOnFailure, set: setStopOnFailure, name: 'Stop on first failure' },{ type: 'select', options: [+ { label: 'Default', value: 'default' },{ label: 'All', value: 'all' },{ label: 'Changed', value: 'changed' },{ label: 'Missing', value: 'missing' },diff --git a/tests/playwright-test/aria-snapshot-file.spec.ts b/tests/playwright-test/aria-snapshot-file.spec.tsindex db9d61278d..8881e4db16 100644--- a/tests/playwright-test/aria-snapshot-file.spec.ts+++ b/tests/playwright-test/aria-snapshot-file.spec.ts@@ -59,6 +59,29 @@ test('should generate multiple missing', async ({ runInlineTest }, testInfo) =>expect(snapshot2).toBe('- heading "hello world 2" [level=1]');});+test('should pass when generating multiple missing with update-snapshots=missing', async ({ runInlineTest }, testInfo) => {+ const result = await runInlineTest({+ 'a.spec.ts': `+ import { test, expect } from '@playwright/test';+ test('test', async ({ page }) => {+ await page.setContent(\`<h1>hello world</h1>\`);+ await expect(page).toMatchAriaSnapshot({ name: 'test-1.aria.yml' });+ await page.setContent(\`<h1>hello world 2</h1>\`);+ await expect(page).toMatchAriaSnapshot({ name: 'test-2.aria.yml' });+ });+ `+ }, { 'update-snapshots': 'missing' });++ expect(result.exitCode).toBe(0);+ expect(result.passed).toBe(1);+ expect(result.output).toContain(`A snapshot doesn't exist at a.spec.ts-snapshots${path.sep}test-1.aria.yml, writing actual`);+ expect(result.output).toContain(`A snapshot doesn't exist at a.spec.ts-snapshots${path.sep}test-2.aria.yml, writing actual`);+ const snapshot1 = await fs.promises.readFile(testInfo.outputPath('a.spec.ts-snapshots/test-1.aria.yml'), 'utf8');+ expect(snapshot1).toBe('- heading "hello world" [level=1]');+ const snapshot2 = await fs.promises.readFile(testInfo.outputPath('a.spec.ts-snapshots/test-2.aria.yml'), 'utf8');+ expect(snapshot2).toBe('- heading "hello world 2" [level=1]');+});+test('should rebaseline all', async ({ runInlineTest }, testInfo) => {const result = await runInlineTest({'a.spec.ts-snapshots/test-1.aria.yml': `@@ -146,7 +169,7 @@ test('backwards compat with .yml extension', async ({ runInlineTest }) => {expect(result.output).toContain(`A snapshot is generated at a.spec.ts-snapshots${path.sep}test-1.yml.`);});-for (const updateSnapshots of ['all', 'changed', 'missing', 'none']) {+for (const updateSnapshots of ['all', 'changed', 'missing', 'none', 'default']) {test(`should update snapshot with the update-snapshots=${updateSnapshots} (config)`, async ({ runInlineTest }, testInfo) => {const result = await runInlineTest({'playwright.config.ts': `diff --git a/tests/playwright-test/config.spec.ts b/tests/playwright-test/config.spec.tsindex 58e69363e2..24a709c51d 100644--- a/tests/playwright-test/config.spec.ts+++ b/tests/playwright-test/config.spec.ts@@ -595,7 +595,7 @@ test('should work with undefined values and base', async ({ runInlineTest }) =>'a.test.ts': `import { test, expect } from '@playwright/test';test('pass', async ({}, testInfo) => {- expect(testInfo.config.updateSnapshots).toBe('missing');+ expect(testInfo.config.updateSnapshots).toBe('default');});`});diff --git a/tests/playwright-test/golden.spec.ts b/tests/playwright-test/golden.spec.tsindex e234b95ffb..9ce9f0a9ea 100644--- a/tests/playwright-test/golden.spec.ts+++ b/tests/playwright-test/golden.spec.ts@@ -360,7 +360,7 @@ test('should update snapshot with the update-snapshots flag', async ({ runInlineexpect(data.toString()).toBe(ACTUAL_SNAPSHOT);});-for (const updateSnapshots of ['all', 'changed', 'missing', 'none']) {+for (const updateSnapshots of ['all', 'changed', 'missing', 'none', 'default']) {test(`should update snapshot with the update-snapshots=${updateSnapshots} (config)`, async ({ runInlineTest }, testInfo) => {const result = await runInlineTest({'playwright.config.ts': `export default { updateSnapshots: '${updateSnapshots}' };`,@@ -450,6 +450,62 @@ test('should silently write missing expectations locally with the update-snapshoexpect(data.toString()).toBe(ACTUAL_SNAPSHOT);});+test('should pass when writing missing expectations with the update-snapshots=missing flag', async ({ runInlineTest }, testInfo) => {+ const ACTUAL_SNAPSHOT = 'Hello world new';+ const result = await runInlineTest({+ ...files,+ 'a.spec.js': `+ const { test, expect } = require('./helper');+ test('is a test', ({}) => {+ expect('${ACTUAL_SNAPSHOT}').toMatchSnapshot('snapshot.txt');+ });+ `+ }, { 'update-snapshots': 'missing' });++ expect(result.exitCode).toBe(0);+ expect(result.passed).toBe(1);+ const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.txt');+ expect(result.output).toContain(`A snapshot doesn't exist at ${snapshotOutputPath}, writing actual`);+ expect(fs.readFileSync(snapshotOutputPath, 'utf-8')).toBe(ACTUAL_SNAPSHOT);+});++test('should fail when writing missing expectations with the update-snapshots=default flag', async ({ runInlineTest }, testInfo) => {+ const ACTUAL_SNAPSHOT = 'Hello world new';+ const result = await runInlineTest({+ ...files,+ 'a.spec.js': `+ const { test, expect } = require('./helper');+ test('is a test', ({}) => {+ expect('${ACTUAL_SNAPSHOT}').toMatchSnapshot('snapshot.txt');+ });+ `+ }, { 'update-snapshots': 'default' });++ expect(result.exitCode).toBe(1);+ expect(result.failed).toBe(1);+ const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.txt');+ expect(result.output).toContain(`A snapshot doesn't exist at ${snapshotOutputPath}, writing actual`);+ expect(fs.readFileSync(snapshotOutputPath, 'utf-8')).toBe(ACTUAL_SNAPSHOT);+});++test('should still fail on a mismatching expectation with the update-snapshots=missing flag', async ({ runInlineTest }, testInfo) => {+ const result = await runInlineTest({+ ...files,+ 'a.spec.js-snapshots/snapshot.txt': 'Hello world',+ 'a.spec.js': `+ const { test, expect } = require('./helper');+ test('is a test', ({}) => {+ expect('Hello world updated').toMatchSnapshot('snapshot.txt');+ });+ `+ }, { 'update-snapshots': 'missing' });++ expect(result.exitCode).toBe(1);+ expect(result.failed).toBe(1);+ const snapshotOutputPath = testInfo.outputPath('a.spec.js-snapshots/snapshot.txt');+ expect(fs.readFileSync(snapshotOutputPath, 'utf-8')).toBe('Hello world');+});+test('should silently write missing expectations locally with the update-snapshots flag for negated matcher', async ({ runInlineTest }, testInfo) => {const result = await runInlineTest({...files,diff --git a/tests/playwright-test/to-have-screenshot.spec.ts b/tests/playwright-test/to-have-screenshot.spec.tsindex 9e83293d77..642f991c5b 100644--- a/tests/playwright-test/to-have-screenshot.spec.ts+++ b/tests/playwright-test/to-have-screenshot.spec.ts@@ -1022,6 +1022,66 @@ test('should silently write missing expectations locally with the update-snapshoexpect(comparePNGs(data, whiteImage)).toBe(null);});+test('should pass when writing missing expectations with the update-snapshots=missing flag', async ({ runInlineTest }, testInfo) => {+ const result = await runInlineTest({+ ...playwrightConfig({+ snapshotPathTemplate: '__screenshots__/{testFilePath}/{arg}{ext}',+ }),+ 'a.spec.js': `+ const { test, expect } = require('@playwright/test');+ test('is a test', async ({ page }) => {+ await expect(page).toHaveScreenshot('snapshot.png');+ });+ `+ }, { 'update-snapshots': 'missing' });++ expect(result.exitCode).toBe(0);+ expect(result.passed).toBe(1);+ const snapshotOutputPath = testInfo.outputPath('__screenshots__/a.spec.js/snapshot.png');+ expect(result.output).toContain(`A snapshot doesn't exist at ${snapshotOutputPath}, writing actual`);+ expect(comparePNGs(fs.readFileSync(snapshotOutputPath), whiteImage)).toBe(null);+});++test('should fail when writing missing expectations with the update-snapshots=default flag', async ({ runInlineTest }, testInfo) => {+ const result = await runInlineTest({+ ...playwrightConfig({+ snapshotPathTemplate: '__screenshots__/{testFilePath}/{arg}{ext}',+ }),+ 'a.spec.js': `+ const { test, expect } = require('@playwright/test');+ test('is a test', async ({ page }) => {+ await expect(page).toHaveScreenshot('snapshot.png');+ });+ `+ }, { 'update-snapshots': 'default' });++ expect(result.exitCode).toBe(1);+ expect(result.failed).toBe(1);+ const snapshotOutputPath = testInfo.outputPath('__screenshots__/a.spec.js/snapshot.png');+ expect(result.output).toContain(`A snapshot doesn't exist at ${snapshotOutputPath}, writing actual`);+ expect(comparePNGs(fs.readFileSync(snapshotOutputPath), whiteImage)).toBe(null);+});++test('should still fail on a mismatching screenshot with the update-snapshots=missing flag', async ({ runInlineTest }, testInfo) => {+ const result = await runInlineTest({+ ...playwrightConfig({+ snapshotPathTemplate: '__screenshots__/{testFilePath}/{arg}{ext}',+ }),+ '__screenshots__/a.spec.js/snapshot.png': redImage,+ 'a.spec.js': `+ const { test, expect } = require('@playwright/test');+ test('is a test', async ({ page }) => {+ await expect(page).toHaveScreenshot('snapshot.png', { timeout: 2000 });+ });+ `+ }, { 'update-snapshots': 'missing' });++ expect(result.exitCode).toBe(1);+ expect(result.failed).toBe(1);+ const snapshotOutputPath = testInfo.outputPath('__screenshots__/a.spec.js/snapshot.png');+ expect(comparePNGs(fs.readFileSync(snapshotOutputPath), redImage)).toBe(null);+});+test('should not write missing expectations locally with the update-snapshots flag for negated matcher', async ({ runInlineTest }, testInfo) => {const result = await runInlineTest({...playwrightConfig({diff --git a/tests/playwright-test/update-aria-snapshot.spec.ts b/tests/playwright-test/update-aria-snapshot.spec.tsindex bf9289a56d..98edc25a53 100644--- a/tests/playwright-test/update-aria-snapshot.spec.ts+++ b/tests/playwright-test/update-aria-snapshot.spec.ts@@ -117,6 +117,40 @@ test('should update missing snapshots', async ({ runInlineTest }, testInfo) => {expect(result2.exitCode).toBe(0);});+test('should pass while updating missing snapshots with update-snapshots=missing', async ({ runInlineTest }, testInfo) => {+ const result = await runInlineTest({+ '.git/marker': '',+ 'a.spec.ts': `+ import { test, expect } from '@playwright/test';+ test('test', async ({ page }) => {+ await page.setContent(\`<h1>hello</h1>\`);+ await expect(page).toMatchAriaSnapshot(\`\`);+ });+ `+ }, { 'update-snapshots': 'missing' });++ expect(result.exitCode).toBe(0);+ expect(result.passed).toBe(1);++ const patchPath = testInfo.outputPath('test-results/rebaselines.patch');+ const data = fs.readFileSync(patchPath, 'utf-8');+ expect(trimPatch(data)).toBe(`diff --git a/a.spec.ts b/a.spec.ts+--- a/a.spec.ts++++ b/a.spec.ts+@@ -2,6 +2,8 @@+ import { test, expect } from '@playwright/test';+ test('test', async ({ page }) => {+ await page.setContent(\`<h1>hello</h1>\`);+- await expect(page).toMatchAriaSnapshot(\`\`);++ await expect(page).toMatchAriaSnapshot(\`++ - heading "hello" [level=1]++ \`);+ });++\\ No newline at end of file+`);+});+test('should update multiple missing snapshots', async ({ runInlineTest }, testInfo) => {const result = await runInlineTest({'.git/marker': '',