Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
TypeScript automationchromechromiume2e-testingelectronend-to-end-testingfirefoxjavascriptplaywrighttesttest-automationtestingtesting-toolswebwebkit
feat(reporter): and `printOnlyFailures` and `--reporter-only-failures` (#42766)
Devin Rousso4e049eefe32d
diff --git a/docs/src/test-api/class-fullconfig.md b/docs/src/test-api/class-fullconfig.mdindex 596cda945a..0e49d2e04d 100644--- a/docs/src/test-api/class-fullconfig.md+++ b/docs/src/test-api/class-fullconfig.md@@ -107,7 +107,7 @@ See [`property: TestConfig.quiet`].## property: FullConfig.reporter* since: v1.10-- type: <[string]|[Array]<[Object]>|[BuiltInReporter]<"list"|"dot"|"line"|"github"|"json"|"junit"|"null"|"html"|"blob"|"perfetto">>+- type: <[string]|[Array]<[Object]>|[BuiltInReporter]<"list"|"dot"|"failures"|"line"|"github"|"json"|"junit"|"null"|"html"|"blob"|"perfetto">>- `0` <[string]> Reporter name or module or file path- `1` <[Object]> An object with reporter options if anydiff --git a/docs/src/test-api/class-testconfig.md b/docs/src/test-api/class-testconfig.mdindex 39b0f1f53a..1191937e4a 100644--- a/docs/src/test-api/class-testconfig.md+++ b/docs/src/test-api/class-testconfig.md@@ -449,7 +449,7 @@ export default defineConfig({## property: TestConfig.reporter* since: v1.10-- type: ?<[string]|[Array]<[Object]>|[BuiltInReporter]<"list"|"dot"|"line"|"github"|"json"|"junit"|"null"|"html"|"blob"|"perfetto">>+- type: ?<[string]|[Array]<[Object]>|[BuiltInReporter]<"list"|"dot"|"failures"|"line"|"github"|"json"|"junit"|"null"|"html"|"blob"|"perfetto">>- `0` <[string]> Reporter name or module or file path- `1` <[Object]> An object with reporter options if anydiff --git a/docs/src/test-cli-js.md b/docs/src/test-cli-js.mdindex fadedf6b74..f7509beaee 100644--- a/docs/src/test-cli-js.md+++ b/docs/src/test-cli-js.md@@ -102,7 +102,8 @@ npx playwright test --ui| `--project <project-name...>` | Only run tests from the specified list of projects, supports '*' wildcard (default: run all projects). || `--quiet` | Suppress stdio. || `--repeat-each <N>` | Run each test `N` times (default: 1). |-| `--reporter <reporter>` | Reporter to use, comma-separated, can be "dot", "line", "list", or others (default: "list" locally and "dot" on CI). You can also pass a path to a custom reporter file. |+| `--reporter <reporter>` | Reporter to use, comma-separated, can be "dot", "line", "list", "failures", or others (default: "list" locally and "dot" on CI). You can also pass a path to a custom reporter file. |+| `--reporter-only-failures` | Include only failed, flaky, and interrupted tests in JSON, JUnit, and HTML reports. || `--retries <retries>` | Maximum retry count for flaky tests, zero for no retries (default: no retries). || `--run-agents <mode>` | Run agents to generate the code for `page.perform`. Possible values are "missing", "all" and "none" (default: "none"). See [test-agents](./test-agents.md). || `--shard <shard>` | Shard tests and execute only the selected shard, specified in the form "current/all", 1-based, e.g., "3/5". |@@ -321,7 +322,8 @@ npx playwright merge-reports ./reports| Option | Description || :--- | :--- || `-c, --config <file>` | Configuration file. Can be used to specify additional configuration for the output report |-| `--reporter <reporter>` | Reporter to use, comma-separated, can be "list", "line", "dot", "json", "junit", "null", "github", "html", "blob", "perfetto" (default: "list" locally and "dot" on CI) |+| `--reporter <reporter>` | Reporter to use, comma-separated, can be "list", "line", "dot", "failures", "json", "junit", "null", "github", "html", "blob", "perfetto" (default: "list" locally and "dot" on CI) |+| `--reporter-only-failures` | Include only failed, flaky, and interrupted tests in JSON, JUnit, and HTML reports. |### Clear Cachediff --git a/docs/src/test-reporter-api/class-reporter.md b/docs/src/test-reporter-api/class-reporter.mdindex 68a2a21dab..8b92b9e326 100644--- a/docs/src/test-reporter-api/class-reporter.md+++ b/docs/src/test-reporter-api/class-reporter.md@@ -74,6 +74,8 @@ export default defineConfig({});```+When `--reporter-only-failures` is set, custom reporters receive `onlyFailures: true` in their constructor options.+Here is a typical order of reporter calls:* [`method: Reporter.onBegin`] is called once with a root suite that contains all other suites and tests. Learn more about [suites hierarchy][Suite].* [`method: Reporter.onTestBegin`] is called for each test run. It is given a [TestCase] that is executed, and a [TestResult] that is almost empty. Test result will be populated while the test runs (for example, with steps and stdio) and will get final `status` once the test finishes.diff --git a/docs/src/test-reporters-js.md b/docs/src/test-reporters-js.mdindex fb72f14753..14af3d9ab0 100644--- a/docs/src/test-reporters-js.md+++ b/docs/src/test-reporters-js.md@@ -50,6 +50,35 @@ export default defineConfig({});```+### Report only failures++Use `onlyFailures` with the JSON, JUnit, or HTML reporter to include only failed, flaky, and interrupted tests. Every attempt of an included test is retained, including successful retries:++```js title="playwright.config.ts"+import { defineConfig } from '@playwright/test';++export default defineConfig({+ reporter: [['html', { onlyFailures: true }]],+});+```++The command line flag enables this option for all configured reporters that support it:++```bash+npx playwright test --reporter=failures,html --reporter-only-failures+npx playwright merge-reports --reporter=html --reporter-only-failures ./blob-report+```++Use the `failures` reporter for live terminal diagnostics without progress output. For example, retain a complete HTML report alongside concise terminal output:++```js title="playwright.config.ts"+import { defineConfig } from '@playwright/test';++export default defineConfig({+ reporter: [['failures'], ['html']],+});+```+## Built-in reportersAll built-in reporters show detailed information about failures, and mostly differ in verbosity for successful runs.@@ -212,6 +241,33 @@ Dot report supports the following configuration options and environment variable| `FORCE_COLOR` | | Whether to produce colored output. | `true` when terminal is in TTY mode, `false` otherwise.| `NO_COLOR` | | Whether to disable colored output ([no-color.org](https://no-color.org/)). Any non-empty value disables colors. | unset+### Failures reporter++The failures reporter prints failures and successful retries as they occur, followed by a summary.++```bash+npx playwright test --reporter=failures+```++```js title="playwright.config.ts"+import { defineConfig } from '@playwright/test';++export default defineConfig({+ reporter: 'failures',+});+```++Use [`property: TestConfig.quiet`] to suppress test stdout and stderr.++Failures report supports the following configuration options and environment variables:++| Environment Variable Name | Reporter Config Option | Description | Default |+|---|---|---|---|+| `PLAYWRIGHT_FAILURES_OMIT_TAGS` | `omitTags` | Whether to omit test tags appended to test titles. | `false` |+| `PLAYWRIGHT_FORCE_TTY` | | Whether to produce output suitable for a live terminal. | `true` when terminal is in TTY mode, `false` otherwise |+| `FORCE_COLOR` | | Whether to produce colored output. | `true` when terminal is in TTY mode, `false` otherwise |+| `NO_COLOR` | | Whether to disable colored output. | unset |+### HTML reporterHTML reporter produces a self-contained folder that contains report for the test run that can be served as a web page.diff --git a/packages/playwright/src/cli/reportActions.ts b/packages/playwright/src/cli/reportActions.tsindex 3b8943ebc5..599de78429 100644--- a/packages/playwright/src/cli/reportActions.ts+++ b/packages/playwright/src/cli/reportActions.ts@@ -29,7 +29,8 @@ export async function showReport(report: string | undefined, host: string, port:export async function mergeReports(reportDir: string | undefined, opts: { [key: string]: any }) {const configFile = opts.config;- const config = configFile ? await configLoader.loadConfigFromFile(configFile) : await configLoader.loadEmptyConfigForMergeReports();+ const overrides = { reporterOnlyFailures: opts.reporterOnlyFailures ? true : undefined };+ const config = configFile ? await configLoader.loadConfigFromFile(configFile, overrides) : await configLoader.loadEmptyConfigForMergeReports(overrides);const dir = path.resolve(process.cwd(), reportDir || '');const dirStat = await fs.promises.stat(dir).catch(e => null);diff --git a/packages/playwright/src/cli/testActions.ts b/packages/playwright/src/cli/testActions.tsindex b6fd25facb..c9f529e450 100644--- a/packages/playwright/src/cli/testActions.ts+++ b/packages/playwright/src/cli/testActions.ts@@ -121,6 +121,7 @@ function overridesFromOptions(options: { [key: string]: any }): ipc.ConfigCLIOverepeatEach: options.repeatEach ? parseInt(options.repeatEach, 10) : undefined,retries: options.retries ? parseInt(options.retries, 10) : undefined,reporter: resolveReporterOption(options.reporter),+ reporterOnlyFailures: options.reporterOnlyFailures ? true : undefined,additionalReporters: resolveReporterOption(options.addReporter),shard: resolveShardOption(options.shard),shuffle: resolveShuffleOption(options.shuffle),diff --git a/packages/playwright/src/common/config.ts b/packages/playwright/src/common/config.tsindex bb6f1b2bc4..c27be2079f 100644--- a/packages/playwright/src/common/config.ts+++ b/packages/playwright/src/common/config.ts@@ -293,7 +293,7 @@ export function toReporters(reporters: BuiltInReporter | ReporterDescription[] |return reporters;}-export const builtInReporters = ['list', 'line', 'dot', 'json', 'junit', 'null', 'github', 'html', 'blob', 'perfetto', 'coverage'] as const;+export const builtInReporters = ['list', 'line', 'dot', 'failures', 'json', 'junit', 'null', 'github', 'html', 'blob', 'perfetto', 'coverage'] as const;export type BuiltInReporter = typeof builtInReporters[number];export type ContextReuseMode = 'none' | 'when-possible';diff --git a/packages/playwright/src/common/configLoader.ts b/packages/playwright/src/common/configLoader.tsindex 5e74dc254a..5d931a847a 100644--- a/packages/playwright/src/common/configLoader.ts+++ b/packages/playwright/src/common/configLoader.ts@@ -366,7 +366,7 @@ export async function loadConfigFromFile(configFile: string | undefined, overridreturn await loadConfig(resolveConfigLocation(configFile), overrides, ignoreDeps);}-export async function loadEmptyConfigForMergeReports() {+export async function loadEmptyConfigForMergeReports(overrides?: ConfigCLIOverrides) {// Merge reports is "different" for no good reason. It should not pick up local config from the cwd.- return await loadConfig({ configDir: process.cwd() });+ return await loadConfig({ configDir: process.cwd() }, overrides);}diff --git a/packages/playwright/src/common/ipc.ts b/packages/playwright/src/common/ipc.tsindex 512a919b5d..f282d85bc3 100644--- a/packages/playwright/src/common/ipc.ts+++ b/packages/playwright/src/common/ipc.ts@@ -36,6 +36,7 @@ export type ConfigCLIOverrides = {repeatEach?: number;retries?: number;reporter?: ReporterDescription[];+ reporterOnlyFailures?: boolean;additionalReporters?: ReporterDescription[];shard?: { current: number, total: number };shuffle?: string;diff --git a/packages/playwright/src/program.ts b/packages/playwright/src/program.tsindex 9042f96522..95db695b62 100644--- a/packages/playwright/src/program.ts+++ b/packages/playwright/src/program.ts@@ -129,6 +129,7 @@ function addMergeReportsCommand(program: Command) {});command.option('-c, --config <file>', `Configuration file. Can be used to specify additional configuration for the output report.`);command.option('--reporter <reporter>', `Reporter to use, comma-separated, can be ${builtInReporters.map(name => `"${name}"`).join(', ')} (default: "${config.defaultReporter}")`);+ command.option('--reporter-only-failures', 'Only include tests with failures in JSON, JUnit, and HTML reports');command.addHelpText('afterAll', `Arguments [dir]:Directory containing blob reports.@@ -230,6 +231,7 @@ const testOptions: [string, { description: string, choices?: string[], preset?:['--quiet', { description: `Suppress stdio` }],['--repeat-each <N>', { description: `Run each test N times (default: 1)` }],['--reporter <reporter>', { description: `Reporter to use, comma-separated, can be ${builtInReporters.map(name => `"${name}"`).join(', ')} (default: "${config.defaultReporter}")` }],+ ['--reporter-only-failures', { description: `Only include tests with failures in JSON, JUnit, and HTML reports` }],['--retries <retries>', { description: `Maximum retry count for flaky tests, zero for no retries (default: no retries)` }],['--run-agents <mode>', { description: `Run agents to generate the code for page.perform`, choices: ['missing', 'all', 'none'], preset: 'none' }],['--shard <shard>', { description: `Shard tests and execute only the selected shard, specify in the form "current/all", 1-based, for example "3/5"` }],diff --git a/packages/playwright/src/reporters/base.ts b/packages/playwright/src/reporters/base.tsindex 00bbb0172b..87ce46efb7 100644--- a/packages/playwright/src/reporters/base.ts+++ b/packages/playwright/src/reporters/base.ts@@ -51,6 +51,7 @@ type TestSummary = {export type CommonReporterOptions = {configDir: string,+ onlyFailures?: boolean,_mode?: 'list' | 'test' | 'merge',_commandHash?: string,};@@ -389,6 +390,30 @@ export class TerminalReporter implements ReporterV2 {}}+export function isFailure(test: TestCase): boolean {+ const outcome = test.outcome();+ return outcome === 'unexpected' || outcome === 'flaky' || test.results.some(result => result.status === 'interrupted');+}++export type TestFilter = (test: TestCase) => boolean;++export function createTestFilter(options: CommonReporterOptions): TestFilter | undefined {+ return options.onlyFailures ? isFailure : undefined;+}++export function* visitTests(suite: Suite, filter?: TestFilter): Generator<TestCase> {+ for (const entry of suite.entries()) {+ if (entry.type !== 'test')+ yield* visitTests(entry, filter);+ else if (!filter || filter(entry))+ yield entry;+ }+}++export function filterTests<T extends Suite | TestCase>(entries: T[], filter?: TestFilter): T[] {+ return filter ? entries.filter(entry => entry.type === 'test' ? filter(entry) : !visitTests(entry, filter).next().done) : entries;+}+function formatResultErrors(screen: Screen, test: TestCase, result: TestResult): string {const lines: string[] = [];if (test.outcome() === 'unexpected') {diff --git a/packages/playwright/src/reporters/failures.ts b/packages/playwright/src/reporters/failures.tsnew file mode 100644index 0000000000..cf063e5d09--- /dev/null+++ b/packages/playwright/src/reporters/failures.ts@@ -0,0 +1,114 @@+/**+ * Copyright (c) Microsoft Corporation.+ *+ * Licensed under the Apache License, Version 2.0 (the "License");+ * you may not use this file except in compliance with the License.+ * You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */++import { msToString } from '@isomorphic/formatUtils';+import { getAsBooleanFromENV } from '@utils/env';++import { markErrorsAsReported, TerminalReporter } from './base';++import type { FailuresReporterOptions } from '../../types/test';+import type { FullResult, TestCase, TestError, TestResult } from '../../types/testReporter';+import type { CommonReporterOptions, TerminalReporterOptions } from './base';++class FailuresReporter extends TerminalReporter {+ private _failureIndex = new Map<TestCase, number>();+ private _needNewLine = false;++ constructor(options?: FailuresReporterOptions & CommonReporterOptions & TerminalReporterOptions) {+ super({ ...options, omitTags: getAsBooleanFromENV('PLAYWRIGHT_FAILURES_OMIT_TAGS', options?.omitTags), lastResult: true });+ }++ override onStdOut(chunk: string | Buffer, test?: TestCase, result?: TestResult) {+ super.onStdOut(chunk, test, result);+ this._dumpToStdio(chunk, this.screen.stdout);+ }++ override onStdErr(chunk: string | Buffer, test?: TestCase, result?: TestResult) {+ super.onStdErr(chunk, test, result);+ this._dumpToStdio(chunk, this.screen.stderr);+ }++ private _dumpToStdio(chunk: string | Buffer, stream: NodeJS.WriteStream) {+ if (this.config.quiet)+ return;+ stream.write(chunk);+ if (chunk.length)+ this._needNewLine = !chunk.toString().endsWith('\n');+ }++ override onTestEnd(test: TestCase, result: TestResult) {+ super.onTestEnd(test, result);+ if (result.status !== 'skipped' && result.status !== test.expectedStatus) {+ this._printFailure(test);+ } else if (result.retry && result.status === 'passed') {+ this._maybeWriteNewLine();+ this.writeLine(this.screen.colors.green(this.formatTestHeader(test, { indent: ' ', index: this._failureIndex.get(test) })));+ this.writeLine(this.screen.colors.green(` Passed on retry #${result.retry} (${msToString(result.duration)}).`));+ this.writeLine();+ }+ }++ private _printFailure(test: TestCase) {+ let index = this._failureIndex.get(test);+ if (index === undefined) {+ index = this._failureIndex.size + 1;+ this._failureIndex.set(test, index);+ }+ const message = this.formatFailure(test, index);+ if (message.trim()) {+ this._maybeWriteNewLine();+ this.writeLine(message);+ }+ }++ override onError(error: TestError) {+ super.onError(error);+ this._maybeWriteNewLine();+ this.writeLine(this.formatError(error).message);+ }++ async onTestPaused(test: TestCase, result: TestResult) {+ // Without TTY, the user cannot interrupt the pause.+ if (!process.stdin.isTTY && !process.env.PW_TEST_DEBUG_REPORTERS)+ return;++ this._maybeWriteNewLine();+ if (test.outcome() === 'unexpected') {+ this._printFailure(test);+ markErrorsAsReported(result);+ this.writeLine(this.screen.colors.yellow(' Paused on error. Press Ctrl+C to end.'));+ } else {+ this.writeLine(this.screen.colors.yellow(this.formatTestHeader(test, { indent: ' ' })));+ this.writeLine(this.screen.colors.yellow(' Paused at test end. Press Ctrl+C to end.'));+ }+ await new Promise<void>(() => {});+ }++ override async onEnd(result: FullResult) {+ await super.onEnd(result);+ this._maybeWriteNewLine();+ this.epilogue(false);+ }++ private _maybeWriteNewLine() {+ if (this._needNewLine) {+ this.writeLine();+ this._needNewLine = false;+ }+ }+}++export default FailuresReporter;diff --git a/packages/playwright/src/reporters/html.ts b/packages/playwright/src/reporters/html.tsindex da7ad610da..647107d6e8 100644--- a/packages/playwright/src/reporters/html.ts+++ b/packages/playwright/src/reporters/html.ts@@ -36,11 +36,12 @@ import { extractZip } from '@utils/third_party/extractZip';// `import('vite')`) is DCE'd in release builds.declare const __PW_HMR__: boolean;-import { CommonReporterOptions, formatError, formatResultFailure, internalScreen } from './base';+import { createTestFilter, filterTests, formatError, formatResultFailure, internalScreen } from './base';import * as babel from '../transform/babelBundle';import { resolveReporterOutputPath, stripAnsiEscapes } from '../util';import type { ReportConfigureParams, ReportEndParams, ReporterV2 } from './reporterV2';+import type { CommonReporterOptions, TestFilter } from './base';import type { HtmlReporterOptions as HtmlReporterConfigOptions, Metadata, TestAnnotation } from '../../types/test';import type * as api from '../../types/testReporter';import type { HTMLReport, HTMLReportOptions, Location, Stats, TestAttachment, TestCase, TestCaseSummary, TestFile, TestFileSummary, TestResult, TestStep } from '@html-reporter/types';@@ -164,7 +165,7 @@ class HtmlReporter implements ReporterV2 {noSnippets,noCopyPrompt,mergeFiles,- });+ }, createTestFilter(this._options));this._buildResult = await builder.build(this.config.metadata, projectSuites, result, this._topLevelErrors, this._machines);}@@ -339,13 +340,15 @@ class HtmlBuilder {private _attachmentsBaseURL: string;private _options: HTMLReportOptions;private _doNotInlineAssets: boolean;+ private _testFilter: TestFilter | undefined;- constructor(yazl: typeof import('yazl'), config: api.FullConfig, outputDir: string, attachmentsBaseURL: string, doNotInlineAssets: boolean, options: HTMLReportOptions) {+ constructor(yazl: typeof import('yazl'), config: api.FullConfig, outputDir: string, attachmentsBaseURL: string, doNotInlineAssets: boolean, options: HTMLReportOptions, testFilter?: TestFilter) {this._dataZipFile = new yazl.ZipFile();this._config = config;this._reportFolder = outputDir;this._options = options;this._doNotInlineAssets = doNotInlineAssets;+ this._testFilter = testFilter;fs.mkdirSync(this._reportFolder, { recursive: true });this._attachmentsBaseURL = attachmentsBaseURL;}@@ -354,7 +357,7 @@ class HtmlBuilder {const data: DataMap = new Map();for (const projectSuite of projectSuites) {const projectName = projectSuite.project()!.name;- for (const fileSuite of projectSuite.suites) {+ for (const fileSuite of filterTests(projectSuite.suites, this._testFilter)) {const fileName = this._relativeLocation(fileSuite.location)!.file;this._createEntryForSuite(data, projectName, fileSuite, fileName, true);}@@ -505,7 +508,7 @@ class HtmlBuilder {private _processSuite(suite: api.Suite, projectName: string, path: string[], deep: boolean, outTests: TestEntry[]) {const newPath = [...path, suite.title];- suite.entries().forEach(e => {+ filterTests(suite.entries(), this._testFilter).forEach(e => {if (e.type === 'test')outTests.push(this._createTestEntry(e, projectName, newPath));else if (deep)diff --git a/packages/playwright/src/reporters/json.ts b/packages/playwright/src/reporters/json.tsindex 406df9ba87..a044069205 100644--- a/packages/playwright/src/reporters/json.ts+++ b/packages/playwright/src/reporters/json.ts@@ -20,10 +20,11 @@ import path from 'path';import { MultiMap } from '@isomorphic/multimap';import { toPosixPath } from '@utils/fileUtils';-import { formatError, nonTerminalScreen, prepareErrorStack, resolveOutputFile, CommonReporterOptions } from './base';+import { createTestFilter, filterTests, formatError, nonTerminalScreen, prepareErrorStack, resolveOutputFile, visitTests } from './base';import { config } from '../common';import type { ReporterV2 } from './reporterV2';+import type { CommonReporterOptions, TestFilter } from './base';import type { JsonReporterOptions } from '../../types/test';import type { FullConfig, FullResult, JSONReport, JSONReportError, JSONReportSpec, JSONReportSuite, JSONReportTest, JSONReportTestResult, JSONReportTestStep, Location, Suite, TestCase, TestError, TestResult, TestStep } from '../../types/testReporter';@@ -32,9 +33,11 @@ class JSONReporter implements ReporterV2 {suite!: Suite;private _errors: TestError[] = [];private _resolvedOutputFile: string | undefined;+ private _testFilter: TestFilter | undefined;constructor(options: JsonReporterOptions & CommonReporterOptions) {this._resolvedOutputFile = resolveOutputFile('JSON', options)?.outputFile;+ this._testFilter = createTestFilter(options);}version(): 'v2' {@@ -92,7 +95,7 @@ class JSONReporter implements ReporterV2 {flaky: 0,},};- for (const test of this.suite.allTests())+ for (const test of visitTests(this.suite, this._testFilter))++report.stats[test.outcome()];return report;}@@ -162,13 +165,15 @@ class JSONReporter implements ReporterV2 {}private _serializeSuite(projectId: string, projectName: string, suite: Suite): null | JSONReportSuite {- if (!suite.allTests().length)+ const entries = filterTests(suite.entries(), this._testFilter);+ const suites = entries.filter(entry => entry.type !== 'test').map(suite => this._serializeSuite(projectId, projectName, suite)).filter(s => s) as JSONReportSuite[];+ const tests = entries.filter(entry => entry.type === 'test');+ if (!tests.length && !suites.length)return null;- const suites = suite.suites.map(suite => this._serializeSuite(projectId, projectName, suite)).filter(s => s) as JSONReportSuite[];return {title: suite.title,...this._relativeLocation(suite.location),- specs: suite.tests.map(test => this._serializeTestSpec(projectId, projectName, test)),+ specs: tests.map(test => this._serializeTestSpec(projectId, projectName, test)),suites: suites.length ? suites : undefined,};}diff --git a/packages/playwright/src/reporters/junit.ts b/packages/playwright/src/reporters/junit.tsindex 82df282283..cc4cca59f1 100644--- a/packages/playwright/src/reporters/junit.ts+++ b/packages/playwright/src/reporters/junit.ts@@ -19,10 +19,11 @@ import path from 'path';import { getAsBooleanFromENV } from '@utils/env';-import { CommonReporterOptions, formatFailure, nonTerminalScreen, resolveOutputFile } from './base';+import { createTestFilter, filterTests, formatFailure, nonTerminalScreen, resolveOutputFile, visitTests } from './base';import { stripAnsiEscapes } from '../util';import type { ReporterV2 } from './reporterV2';+import type { CommonReporterOptions, TestFilter } from './base';import type { JUnitReporterOptions } from '../../types/test';import type { FullConfig, FullResult, Suite, TestCase, TestResult } from '../../types/testReporter';@@ -40,11 +41,13 @@ class JUnitReporter implements ReporterV2 {private includeProjectInTestName = false;private includeRetries = false;private omitTags = false;+ private testFilter: TestFilter | undefined;constructor(options: JUnitReporterOptions & CommonReporterOptions) {+ this.testFilter = createTestFilter(options);this.stripANSIControlSequences = getAsBooleanFromENV('PLAYWRIGHT_JUNIT_STRIP_ANSI', !!options.stripANSIControlSequences);this.includeProjectInTestName = getAsBooleanFromENV('PLAYWRIGHT_JUNIT_INCLUDE_PROJECT_IN_TEST_NAME', !!options.includeProjectInTestName);- this.includeRetries = getAsBooleanFromENV('PLAYWRIGHT_JUNIT_INCLUDE_RETRIES', !!options.includeRetries);+ this.includeRetries = !!options.onlyFailures || getAsBooleanFromENV('PLAYWRIGHT_JUNIT_INCLUDE_RETRIES', !!options.includeRetries);this.omitTags = getAsBooleanFromENV('PLAYWRIGHT_JUNIT_OMIT_TAGS', !!options.omitTags);this.configDir = options.configDir;this.resolvedOutputFile = resolveOutputFile('JUNIT', options)?.outputFile;@@ -70,7 +73,7 @@ class JUnitReporter implements ReporterV2 {async onEnd(result: FullResult) {const children: XMLEntry[] = [];for (const projectSuite of this.suite.suites) {- for (const fileSuite of projectSuite.suites)+ for (const fileSuite of filterTests(projectSuite.suites, this.testFilter))children.push(await this._buildTestSuite(projectSuite.title, fileSuite));}const tokens: string[] = [];@@ -110,7 +113,7 @@ class JUnitReporter implements ReporterV2 {const children: XMLEntry[] = [];const testCaseNamePrefix = projectName && this.includeProjectInTestName ? `[${projectName}] ` : '';- for (const test of suite.allTests()){+ for (const test of visitTests(suite, this.testFilter)) {++tests;if (test.outcome() === 'skipped')++skipped;diff --git a/packages/playwright/src/runner/reporters.ts b/packages/playwright/src/runner/reporters.tsindex 34cff70b12..7b9bef3c18 100644--- a/packages/playwright/src/runner/reporters.ts+++ b/packages/playwright/src/runner/reporters.ts@@ -22,6 +22,7 @@ import { BlobReporter } from '../reporters/blob';import CoverageReporter from '../reporters/coverage';import DotReporter from '../reporters/dot';import EmptyReporter from '../reporters/empty';+import FailuresReporter from '../reporters/failures';import GitHubReporter from '../reporters/github';import HtmlReporter from '../reporters/html';import JSONReporter from '../reporters/json';@@ -44,6 +45,7 @@ export async function createReporters(config: FullConfigInternal, mode: 'list' |'blob': BlobReporter,'perfetto': PerfettoReporter,'dot': mode === 'list' ? ListModeReporter : DotReporter,+ 'failures': mode === 'list' ? ListModeReporter : FailuresReporter,'line': mode === 'list' ? ListModeReporter : LineReporter,'list': mode === 'list' ? ListModeReporter : ListReporter,'github': GitHubReporter,@@ -58,9 +60,9 @@ export async function createReporters(config: FullConfigInternal, mode: 'list' |const reportOptions = reporterCommandOptions(config, mode, runOptions);for (const r of descriptions) {const [name, arg] = r;- const options = { ...reportOptions, ...arg };+ const options = resolveReporterOptions(reportOptions, { ...reportOptions, ...arg });if (name in defaultReporters) {- reporters.push(new defaultReporters[name as keyof typeof defaultReporters](options));+ reporters.push(new defaultReporters[name as commonConfig.BuiltInReporter](options));} else {const reporterConstructor = await loadReporter(config, name);reporters.push(wrapReporterAsV2(new reporterConstructor(options)));@@ -69,7 +71,7 @@ export async function createReporters(config: FullConfigInternal, mode: 'list' |if (process.env.PW_TEST_REPORTER) {const name = process.env.PW_TEST_REPORTER;if (name in defaultReporters) {- reporters.push(new defaultReporters[name as keyof typeof defaultReporters](reportOptions));+ reporters.push(new defaultReporters[name as commonConfig.BuiltInReporter](reportOptions));} else {const reporterConstructor = await loadReporter(config, name);reporters.push(wrapReporterAsV2(new reporterConstructor(reportOptions)));@@ -109,9 +111,19 @@ function reporterCommandOptions(config: FullConfigInternal, mode: 'list' | 'testconfigDir: config.configDir,_mode: mode,_commandHash: computeCommandHash(config, runOptions),+ onlyFailures: mode !== 'list' && config.configCLIOverrides.reporterOnlyFailures,};}+function resolveReporterOptions(commonOptions: CommonReporterOptions, options: CommonReporterOptions): CommonReporterOptions {+ let onlyFailures = options.onlyFailures;+ if (commonOptions._mode === 'list')+ onlyFailures = false;+ else if (commonOptions.onlyFailures)+ onlyFailures = true;+ return { ...options, onlyFailures };+}+function computeCommandHash(config: FullConfigInternal, runOptions?: TestRunOptions) {const parts = [];// Include project names for readability.diff --git a/packages/playwright/types/test.d.ts b/packages/playwright/types/test.d.tsindex 144b352090..44c7700894 100644--- a/packages/playwright/types/test.d.ts+++ b/packages/playwright/types/test.d.ts@@ -20,13 +20,15 @@ export * from 'playwright-core';export type BlobReporterOptions = { outputDir?: string, fileName?: string };export type DotReporterOptions = { omitTags?: boolean };+export type FailuresReporterOptions = { omitTags?: boolean };export type LineReporterOptions = { omitTags?: boolean };export type ListReporterOptions = { printSteps?: boolean, printFailuresInline?: boolean, omitTags?: boolean };export type GitHubReporterOptions = { omitTags?: boolean };-export type JUnitReporterOptions = { outputFile?: string, stripANSIControlSequences?: boolean, includeProjectInTestName?: boolean, includeRetries?: boolean, omitTags?: boolean };-export type JsonReporterOptions = { outputFile?: string };+export type JUnitReporterOptions = { outputFile?: string, stripANSIControlSequences?: boolean, includeProjectInTestName?: boolean, includeRetries?: boolean, omitTags?: boolean, onlyFailures?: boolean };+export type JsonReporterOptions = { outputFile?: string, onlyFailures?: boolean };export type PerfettoReporterOptions = { outputFile?: string };export type HtmlReporterOptions = {+ onlyFailures?: boolean;outputFolder?: string;open?: 'always' | 'never' | 'on-failure';host?: string;@@ -42,6 +44,7 @@ export type HtmlReporterOptions = {export type ReporterDescription = Readonly<['blob'] | ['blob', BlobReporterOptions] |['dot'] | ['dot', DotReporterOptions] |+ ['failures'] | ['failures', FailuresReporterOptions] |['line'] | ['line', LineReporterOptions] |['list'] | ['list', ListReporterOptions] |['github'] | ['github', GitHubReporterOptions] |@@ -961,7 +964,7 @@ interface TestConfig<TestArgs = {}, WorkerArgs = {}> {* ```**/- reporter?: LiteralUnion<'list'|'dot'|'line'|'github'|'json'|'junit'|'null'|'html'|'blob'|'perfetto', string> | ReporterDescription[];+ reporter?: LiteralUnion<'list'|'dot'|'failures'|'line'|'github'|'json'|'junit'|'null'|'html'|'blob'|'perfetto', string> | ReporterDescription[];/*** Global options for all tests, for example* [testOptions.browserName](https://playwright.dev/docs/api/class-testoptions#test-options-browser-name). Learn morediff --git a/packages/playwright/types/testReporter.d.ts b/packages/playwright/types/testReporter.d.tsindex a7962739df..12a5618bb1 100644--- a/packages/playwright/types/testReporter.d.ts+++ b/packages/playwright/types/testReporter.d.ts@@ -92,6 +92,8 @@ export interface FullResult {* });* ```*+ * When `--reporter-only-failures` is set, custom reporters receive `onlyFailures: true` in their constructor options.+ ** Here is a typical order of reporter calls:* - [reporter.onBegin(config, suite)](https://playwright.dev/docs/api/class-reporter#reporter-on-begin) is called* once with a root suite that contains all other suites and tests. Learn more aboutdiff --git a/tests/playwright-test/reporter-only-failures.spec.ts b/tests/playwright-test/reporter-only-failures.spec.tsnew file mode 100644index 0000000000..6442b2b313--- /dev/null+++ b/tests/playwright-test/reporter-only-failures.spec.ts@@ -0,0 +1,402 @@+/**+ * Copyright (c) Microsoft Corporation.+ *+ * Licensed under the Apache License, Version 2.0 (the "License");+ * you may not use this file except in compliance with the License.+ * You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */++import fs from 'fs';+import path from 'path';+import xml2js from 'xml2js';+import { test, expect, stripAnsi } from './playwright-test-fixtures';+import { extractZip } from '../../packages/utils/third_party/extractZip';++import type { HTMLReport, TestFile } from '../../packages/html-reporter/src/types';+import type { JSONReport, JSONReportSpec, JSONReportSuite } from '@playwright/test/reporter';++const passingTest = `+ import { test } from '@playwright/test';+ test('passes', () => {});+`;++const testFiles = {+ 'passing.test.ts': `+ import fs from 'fs';+ import { test } from '@playwright/test';+ test('passes', async ({}, testInfo) => {+ fs.writeFileSync('passing-ran.txt', 'ran');+ const attachmentPath = testInfo.outputPath('passing.txt');+ fs.writeFileSync(attachmentPath, 'passing attachment');+ await testInfo.attach('passing', { path: attachmentPath });+ });+ `,+ 'failures.test.ts': `+ import fs from 'fs';+ import { test, expect } from '@playwright/test';+ test.describe('failures', () => {+ test('fails', async ({}, testInfo) => {+ const attachmentPath = testInfo.outputPath('failure.txt');+ fs.writeFileSync(attachmentPath, 'failure attachment');+ await testInfo.attach('failure', { path: attachmentPath });+ expect(1).toBe(2);+ });+ test('flaky', ({}, testInfo) => {+ console.log('flaky attempt ' + testInfo.retry);+ console.error('flaky stderr ' + testInfo.retry);+ if (!testInfo.retry)+ throw new Error('flaky failure');+ });+ test('unexpectedly passes', () => {+ test.fail();+ });+ });+ test.describe('ignored', () => {+ test.skip('skipped', () => {});+ test('expected failure', () => {+ test.fail();+ throw new Error('expected failure');+ });+ });+ `,+};++test('--reporter-only-failures leaves terminal reporters unchanged', async ({ runInlineTest }) => {+ const result = await runInlineTest({+ 'a.test.ts': passingTest,+ }, { 'reporter': 'list,line,dot', 'reporter-only-failures': true }, { PLAYWRIGHT_FORCE_TTY: '0' });+ expect(result.exitCode).toBe(0);+ expect(result.output.match(/Running 1 test using/g)).toHaveLength(3);+ expect(result.output.match(/› passes/g)).toHaveLength(2);+ expect(result.output).toContain('·');+});++test('failures reporter accounts for passing and failing test durations', async ({ runInlineTest }) => {+ const result = await runInlineTest({+ 'playwright.config.ts': `+ export default {+ reporter: 'failures',+ fullyParallel: false,+ reportSlowTests: { max: 1, threshold: 300 },+ };+ `,+ 'a.test.ts': `+ import { test } from '@playwright/test';+ test('passes', async () => {+ await new Promise(resolve => setTimeout(resolve, 200));+ });+ test('fails', async () => {+ await new Promise(resolve => setTimeout(resolve, 200));+ throw new Error('failure');+ });+ `,+ });+ expect(result.exitCode).toBe(1);+ expect(result.passed).toBe(1);+ expect(result.failed).toBe(1);+ expect(result.output).toContain('Slow test file: a.test.ts');+});++test('failures reporter respects quiet and reports errors outside tests', async ({ runInlineTest }) => {+ const result = await runInlineTest({+ 'playwright.config.ts': `+ export default {+ reporter: 'failures',+ quiet: true,+ globalTeardown: './global-teardown.ts',+ };+ `,+ 'global-teardown.ts': `+ export default () => {+ throw new Error('global teardown failed');+ };+ `,+ 'a.test.ts': `+ import { test } from '@playwright/test';+ test('passes', () => {+ process.stdout.write('test stdout');+ process.stderr.write('test stderr');+ });+ test.skip('skipped', () => {});+ test('expected failure', () => {+ test.fail();+ throw new Error('expected failure');+ });+ `,+ }, {}, { PLAYWRIGHT_FORCE_TTY: '1' });+ expect(result.exitCode).toBe(1);+ expect(result.failed).toBe(0);+ expect(result.passed).toBe(2);+ expect(result.output).not.toContain('test stdout');+ expect(result.output).not.toContain('test stderr');+ expect(result.output.match(/Error: global teardown failed/g)).toHaveLength(1);+ expect(result.output).toContain('1 error was not a part of any test');+ expect(result.output).toContain('1 skipped');+ expect(result.output).not.toContain('Running 1 test using');+ expect(result.output).not.toMatch(/› (?:passes|skipped|expected failure)/);+ expect(result.rawOutput).not.toContain('\u001B[1A');+});++test('failures reporter avoids automatic terminal output', async ({ runInlineTest }) => {+ const files = {+ 'playwright.config.ts': `+ export default { reporter: [['json', { outputFile: 'report.json' }]] };+ `,+ 'a.test.ts': passingTest,+ };+ const fallback = await runInlineTest(files, { 'reporter-only-failures': true }, { CI: undefined });+ expect(fallback.exitCode).toBe(0);+ expect(fallback.output).toContain('Running 1 test using');+ expect(fallback.output).toContain('› passes');+ const added = await runInlineTest(files, { 'reporter-only-failures': true, 'add-reporter': 'failures' }, { CI: undefined });+ expect(added.exitCode).toBe(0);+ expect(added.output).toMatch(/^ 1 passed \([^)]+\)\n$/);+});++test('onlyFailures can be configured independently for each reporter', async ({ runInlineTest }, testInfo) => {+ const result = await runInlineTest({+ 'playwright.config.ts': `+ export default {+ reporter: [+ ['failures'],+ ['json', { onlyFailures: true, outputFile: 'failures.json' }],+ ['json', { outputFile: 'complete.json' }],+ ],+ };+ `,+ 'a.test.ts': passingTest + `+ test('fails', () => {+ throw new Error('failure');+ });+ `,+ }, { workers: 1 });+ expect(result.exitCode).toBe(1);+ expect(result.output).not.toContain('› passes');+ expect(result.passed).toBe(1);+ const failures: JSONReport = JSON.parse(fs.readFileSync(testInfo.outputPath('failures.json'), 'utf8'));+ const complete: JSONReport = JSON.parse(fs.readFileSync(testInfo.outputPath('complete.json'), 'utf8'));+ expect(jsonTestSpecs(failures.suites).map(spec => spec.title)).toEqual(['fails']);+ expect(jsonTestSpecs(complete.suites).map(spec => spec.title)).toEqual(['passes', 'fails']);+});++for (const merge of [false, true]) {+ test(`--reporter-only-failures filters final reports and preserves blobs when ${merge ? 'merging' : 'running'}`, async ({ runInlineTest, mergeReports }, testInfo) => {+ const result = await runInlineTest({+ ...testFiles,+ 'playwright.config.ts': `+ export default {+ retries: 1,+ reporter: [+ ['failures'],+ ['json', { outputFile: 'report.json' }],+ ['junit', { outputFile: 'report.xml', includeRetries: false }],+ ['html', { open: 'never' }],+ ['blob', { outputDir: 'complete-blob-report' }],+ ['perfetto', { outputFile: 'perfetto.json' }],+ ['./integrity-reporter.ts', { onlyFailures: false }],+ ],+ };+ `,+ 'integrity-reporter.ts': `+ export default class {+ constructor(options: { onlyFailures?: boolean }) {+ console.log('%%onlyFailures: ' + options.onlyFailures);+ }+ onBegin(config, suite) {+ this.suite = suite;+ console.log('%%begin: ' + suite.allTests().length);+ }+ onTestBegin(test) {+ console.log('%%begin: ' + test.title);+ }+ onTestEnd(test) {+ console.log('%%end: ' + test.title);+ }+ onEnd() {+ console.log('%%end: ' + this.suite.allTests().length);+ }+ }+ `,+ }, merge ? { reporter: 'blob', workers: 1 } : { 'reporter-only-failures': true, 'workers': 1 }, { PLAYWRIGHT_JUNIT_INCLUDE_RETRIES: '0' });+ expect(result.exitCode).toBe(1);+ let output = result.output;+ if (merge) {+ const merged = await mergeReports(testInfo.outputPath('blob-report'), { PLAYWRIGHT_JUNIT_INCLUDE_RETRIES: '0' }, {+ cwd: testInfo.outputPath(),+ additionalArgs: ['--config', testInfo.outputPath('playwright.config.ts'), '--reporter-only-failures'],+ });+ expect(merged.exitCode).toBe(0);+ output = stripAnsi(merged.output);+ }+ expect(output).toContain('2 passed');+ expect(output).toContain('2 failed');+ expect(output).toContain('1 flaky');+ expect(output).toContain('Passed on retry #1');+ expect(output.match(/^ Error: expect\(received\)/gm)).toHaveLength(2);+ expect(output.match(/^ Error: flaky failure/gm)).toHaveLength(1);+ expect(output).toContain('Expected: 2');+ expect(output).toContain('Received: 1');+ expect(output).toMatch(/> +\d+ \| +expect\(1\)\.toBe\(2\);/);+ expect(output).toMatch(/at .*failures\.test\.ts:\d+:\d+/);+ expect(output).not.toContain('Running 6 tests');+ expect(output).not.toContain('passing.test.ts');+ expect(output).toContain('%%begin: 6');+ expect(output).toContain('%%end: 6');+ expect(output).toContain('%%onlyFailures: true');+ expect(fs.existsSync(testInfo.outputPath('passing-ran.txt'))).toBe(true);++ const titles = ['fails', 'flaky', 'unexpectedly passes'];+ const allTitles = [...titles, 'passes', 'skipped', 'expected failure'].sort();+ for (const title of allTitles) {+ expect(output).toContain(`%%begin: ${title}`);+ expect(output).toContain(`%%end: ${title}`);+ }+ for (const retry of [0, 1]) {+ expect(output).toContain(`flaky attempt ${retry}`);+ expect(output).toContain(`flaky stderr ${retry}`);+ }+ const json: JSONReport = JSON.parse(fs.readFileSync(testInfo.outputPath('report.json'), 'utf8'));+ const specs = jsonTestSpecs(json.suites);+ expect(specs.map(spec => spec.title).sort()).toEqual(titles);+ expect(json.stats).toMatchObject({ expected: 0, skipped: 0, unexpected: 2, flaky: 1 });+ for (const spec of specs) {+ const statuses = spec.title === 'flaky' ? ['failed', 'passed'] : spec.title === 'fails' ? ['failed', 'failed'] : ['passed', 'passed'];+ expect(spec.tests[0].results.map(result => result.status)).toEqual(statuses);+ }++ const xml = await xml2js.parseStringPromise(fs.readFileSync(testInfo.outputPath('report.xml'), 'utf8'));+ expect(xml.testsuites.$.tests).toBe('3');+ expect(xml.testsuites.testsuite).toHaveLength(1);+ expect(xml.testsuites.testsuite[0].testcase.map((entry: { $: { name: string } }) => entry.$.name).sort()).toEqual(titles.map(title => `failures › ${title}`));+ const junitTests = xml.testsuites.testsuite[0].testcase;+ expect(junitTests[0].failure).toHaveLength(1);+ expect(junitTests[0].rerunFailure).toHaveLength(1);+ expect(junitTests[1].flakyError).toHaveLength(1);+ expect(junitTests[1].flakyError[0]['system-out'][0]).toContain('flaky attempt 0');+ expect(junitTests[1]['system-out'][0]).toContain('flaky attempt 1');++ const html = fs.readFileSync(testInfo.outputPath('playwright-report', 'index.html'), 'utf8');+ const data = html.match(/<template id="playwrightReportBase64">data:application\/zip;base64,([^<]+)<\/template>/);+ expect(data).not.toBeNull();+ const htmlZip = testInfo.outputPath('html-data.zip');+ fs.writeFileSync(htmlZip, Buffer.from(data![1], 'base64'));+ const htmlDataDir = testInfo.outputPath('html-data');+ await extractZip(htmlZip, { dir: htmlDataDir });+ const htmlReport: HTMLReport = JSON.parse(fs.readFileSync(path.join(htmlDataDir, 'report.json'), 'utf8'));+ expect(htmlReport.files.flatMap(file => file.tests.map(test => test.title)).sort()).toEqual(titles);+ expect(htmlReport.stats).toMatchObject({ total: 3, expected: 0, skipped: 0, unexpected: 2, flaky: 1 });+ const htmlFile: TestFile = JSON.parse(fs.readFileSync(path.join(htmlDataDir, htmlReport.files[0].fileId + '.json'), 'utf8'));+ for (const test of htmlFile.tests) {+ const statuses = test.title === 'flaky' ? ['failed', 'passed'] : test.title === 'fails' ? ['failed', 'failed'] : ['passed', 'passed'];+ expect(test.results.map(result => result.status)).toEqual(statuses);+ }++ if (!merge) {+ const trace: { traceEvents: { cat: string, name: string }[] } = JSON.parse(fs.readFileSync(testInfo.outputPath('perfetto.json'), 'utf8'));+ expect([...new Set(trace.traceEvents.filter(event => event.cat === 'test').map(event => event.name))].sort()).toEqual(allTitles);++ const blobDir = testInfo.outputPath('complete-blob-report');+ const blobFile = fs.readdirSync(blobDir).find(file => file.endsWith('.zip'))!;+ const extractedBlob = testInfo.outputPath('extracted-blob');+ await extractZip(path.join(blobDir, blobFile), { dir: extractedBlob });+ const messages = fs.readFileSync(path.join(extractedBlob, 'report.jsonl'), 'utf8');+ expect(messages).toContain('passing.test.ts');+ for (const title of allTitles)+ expect(messages).toContain(`"title":"${title}"`);+ const resources = fs.readdirSync(path.join(extractedBlob, 'resources'));+ const attachments = resources.filter(resource => resource.endsWith('.txt')).map(resource => fs.readFileSync(path.join(extractedBlob, 'resources', resource), 'utf8'));+ expect(attachments.sort()).toEqual(['failure attachment', 'failure attachment', 'passing attachment']);+ }+ });+}++test('failures reporter prints timeout and successful retry details live without progress', async ({ interactWithTestRunner }, testInfo) => {+ const runner = await interactWithTestRunner({+ 'playwright.config.ts': `+ export default {+ reporter: [['failures', { omitTags: true }]],+ retries: 1,+ };+ `,+ 'a.test.ts': `+ import fs from 'fs';+ import { test } from '@playwright/test';+ test('flaky', { tag: '@tag' }, async ({}, testInfo) => {+ if (testInfo.retry) {+ console.log('RETRY STARTED');+ while (!fs.existsSync('continue'))+ await new Promise(resolve => setTimeout(resolve, 10));+ } else {+ process.stdout.write('partial stdout');+ test.setTimeout(500);+ await new Promise(() => {});+ }+ });+ test('passes', async () => {+ await test.step('passing step', async () => {});+ console.log('LATER TEST STARTED');+ while (!fs.existsSync('finish'))+ await new Promise(resolve => setTimeout(resolve, 10));+ });+ `,+ }, { workers: 1 }, { PLAYWRIGHT_FORCE_TTY: '1', PLAYWRIGHT_FAILURES_OMIT_TAGS: 'false' });++ await runner.waitForOutput('RETRY STARTED');+ const liveOutput = stripAnsi(runner.output);+ expect(liveOutput).toContain('partial stdout\n 1) ');+ expect(liveOutput).toContain('Test timeout of 500ms exceeded.');+ expect(liveOutput.indexOf('Test timeout of 500ms exceeded.')).toBeLessThan(liveOutput.indexOf('RETRY STARTED'));+ fs.writeFileSync(testInfo.outputPath('continue'), '');++ await runner.waitForOutput('LATER TEST STARTED');+ const retryOutput = stripAnsi(runner.output);+ expect(retryOutput).toContain('Passed on retry #1');+ expect(retryOutput.indexOf('Passed on retry #1')).toBeLessThan(retryOutput.indexOf('LATER TEST STARTED'));+ fs.writeFileSync(testInfo.outputPath('finish'), '');++ expect((await runner.exited).exitCode).toBe(0);+ const output = stripAnsi(runner.output);+ expect(output.match(/^ Test timeout of 500ms exceeded\./gm)).toHaveLength(1);+ expect(output.match(/^ Passed on retry #1 /gm)).toHaveLength(1);+ const titleLines = output.split('\n').filter(line => line.startsWith(' 1) '));+ expect(titleLines).toHaveLength(2);+ for (const line of titleLines)+ expect(line).toContain('@tag');+ expect(output).toContain('1 flaky');+ expect(output).toContain('1 passed');+ expect(output).not.toContain('passing step');+ expect(output).not.toMatch(/^\s*(?:✓|ok|✘|x|-)\s+\d+ /m);+ expect(runner.output).not.toContain('\u001B[1A');+ expect(runner.output).not.toContain('\u001B[2K');+});++test('--reporter-only-failures does not filter --list', async ({ runInlineTest }, testInfo) => {+ const result = await runInlineTest({+ 'playwright.config.ts': `+ export default {+ reporter: [['failures'], ['json', { onlyFailures: true, outputFile: 'report.json' }]],+ };+ `,+ 'a.test.ts': passingTest,+ }, { 'list': true, 'reporter-only-failures': true });+ expect(result.exitCode).toBe(0);+ expect(result.output).toContain('Listing tests:');+ expect(result.output).toContain('› passes');+ const report: JSONReport = JSON.parse(fs.readFileSync(testInfo.outputPath('report.json'), 'utf8'));+ expect(jsonTestSpecs(report.suites).map(spec => spec.title)).toEqual(['passes']);+});++function jsonTestSpecs(suites: JSONReportSuite[]): JSONReportSpec[] {+ return suites.flatMap(suite => [+ ...suite.specs,+ ...jsonTestSpecs(suite.suites || []),+ ]);+}diff --git a/utils/generate_types/overrides-test.d.ts b/utils/generate_types/overrides-test.d.tsindex edf8f9aa8b..31e52a68d3 100644--- a/utils/generate_types/overrides-test.d.ts+++ b/utils/generate_types/overrides-test.d.ts@@ -19,13 +19,15 @@ export * from 'playwright-core';export type BlobReporterOptions = { outputDir?: string, fileName?: string };export type DotReporterOptions = { omitTags?: boolean };+export type FailuresReporterOptions = { omitTags?: boolean };export type LineReporterOptions = { omitTags?: boolean };export type ListReporterOptions = { printSteps?: boolean, printFailuresInline?: boolean, omitTags?: boolean };export type GitHubReporterOptions = { omitTags?: boolean };-export type JUnitReporterOptions = { outputFile?: string, stripANSIControlSequences?: boolean, includeProjectInTestName?: boolean, includeRetries?: boolean, omitTags?: boolean };-export type JsonReporterOptions = { outputFile?: string };+export type JUnitReporterOptions = { outputFile?: string, stripANSIControlSequences?: boolean, includeProjectInTestName?: boolean, includeRetries?: boolean, omitTags?: boolean, onlyFailures?: boolean };+export type JsonReporterOptions = { outputFile?: string, onlyFailures?: boolean };export type PerfettoReporterOptions = { outputFile?: string };export type HtmlReporterOptions = {+ onlyFailures?: boolean;outputFolder?: string;open?: 'always' | 'never' | 'on-failure';host?: string;@@ -41,6 +43,7 @@ export type HtmlReporterOptions = {export type ReporterDescription = Readonly<['blob'] | ['blob', BlobReporterOptions] |['dot'] | ['dot', DotReporterOptions] |+ ['failures'] | ['failures', FailuresReporterOptions] |['line'] | ['line', LineReporterOptions] |['list'] | ['list', ListReporterOptions] |['github'] | ['github', GitHubReporterOptions] |@@ -69,7 +72,7 @@ type LiteralUnion<T extends U, U = string> = T | (U & { zz_IGNORE_ME?: never });interface TestConfig<TestArgs = {}, WorkerArgs = {}> {projects?: Project<TestArgs, WorkerArgs>[];- reporter?: LiteralUnion<'list'|'dot'|'line'|'github'|'json'|'junit'|'null'|'html'|'blob'|'perfetto', string> | ReporterDescription[];+ reporter?: LiteralUnion<'list'|'dot'|'failures'|'line'|'github'|'json'|'junit'|'null'|'html'|'blob'|'perfetto', string> | ReporterDescription[];use?: UseOptions<TestArgs, WorkerArgs>;webServer?: TestConfigWebServer | TestConfigWebServer[];}