fix Cucumber retries collapsing into single Allure result#15180
fix Cucumber retries collapsing into single Allure result#15180
Conversation
create-wdio
eslint-plugin-wdio
@wdio/allure-reporter
@wdio/appium-service
@wdio/browser-runner
@wdio/browserstack-service
@wdio/cli
@wdio/concise-reporter
@wdio/config
@wdio/cucumber-framework
@wdio/dot-reporter
@wdio/firefox-profile-service
@wdio/globals
@wdio/jasmine-framework
@wdio/json-reporter
@wdio/junit-reporter
@wdio/lighthouse-service
@wdio/local-runner
@wdio/logger
@wdio/mocha-framework
@wdio/protocols
@wdio/repl
@wdio/reporter
@wdio/runner
@wdio/sauce-service
@wdio/shared-store-service
@wdio/smoke-test-cjs-service
@wdio/smoke-test-reporter
@wdio/smoke-test-service
@wdio/spec-reporter
@wdio/static-server-service
@wdio/sumologic-reporter
@wdio/testingbot-service
@wdio/types
@wdio/utils
@wdio/webdriver-mock-service
@wdio/xvfb
webdriver
webdriverio
commit: |
There was a problem hiding this comment.
Pull request overview
Fixes how Cucumber scenario retries are represented in Allure by ensuring each attempt becomes its own Allure test result (instead of multiple attempts collapsing into a single result).
Changes:
- Implemented
onSuiteRetrylogic to close attempt #1 with aretriedtag and immediately start a fresh Allure test for attempt #2. - Added
onTestStart“scenario collision” handling forscenarioLevelReporterto close orphaned/pending attempts before starting the next attempt. - Added/extended Cucumber retry regression tests for both default mode and
scenarioLevelReportermode.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| packages/wdio-allure-reporter/src/reporter.ts | Adds retry handling to end the first attempt and start a new Allure test; adds scenario-level retry collision detection. |
| packages/wdio-allure-reporter/tests/cucumber.suite.test.ts | Adds regression tests asserting separate Allure results per retry attempt and correct retry tagging/historyId behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const suiteChildren = [...(suite.tests || []), ...(suite.hooks || [])] | ||
| const isSkipped = | ||
| suiteChildren.length > 0 && | ||
| (suite.tests || []).every((t: TestStats) => [AllureStatusEnum.SKIPPED].includes(t.state as AllureStatus)) && | ||
| (suite.hooks || []).every((h: HookStats) => [AllureStatusEnum.PASSED, AllureStatusEnum.SKIPPED].includes(h.state as AllureStatus)) | ||
| const failed = suiteChildren.find((i) => i.state === AllureStatusEnum.FAILED) | ||
| const status = isSkipped | ||
| ? AllureStatusEnum.SKIPPED | ||
| : failed ? getTestStatus(failed) : AllureStatusEnum.FAILED | ||
| const error = failed ? getErrorFromFailedTest(failed) : undefined |
There was a problem hiding this comment.
onSuiteRetry tries to infer the first attempt's status/error from suite.tests/suite.hooks, but in the real WDIO event flow the @wdio/reporter listener calls suiteStat.retry() before invoking onSuiteRetry, which clears these arrays (see packages/wdio-reporter/src/index.ts:102-106 and packages/wdio-reporter/src/stats/suite.ts:64-69). This means suiteChildren will typically be empty here and statusDetails will never be populated. Consider persisting the last-known scenario failure details in the reporter state (e.g. when step/hook ends) or changing the integration point so onSuiteRetry receives the pre-cleared suite stats.
| // Open a fresh Allure test for the retry attempt. | ||
| // Sharing the same historyId is intentional: Allure uses it to group | ||
| // separate result files as retry attempts of the same scenario. | ||
| this._consoleOutput = '' | ||
| this._startTest({ name: suite.title, start: Date.now() }) | ||
| this._currentLeafTitleByCid.set(cid, suite.title) | ||
| const fullTitleForHash = this._mochaFullTitle(cid, suite.title) | ||
| this._emitHistoryIdsFrom(fullTitleForHash) | ||
|
|
||
| const fullName = toFullName(this._pkgByCid.get(cid)!, fullTitleForHash) | ||
| this._pushRuntimeMessage({ type: 'allure:test:info', data: { fullName, fullTitle: fullTitleForHash } }) | ||
|
|
||
| this._emitBaseLabels(cid) | ||
|
|
There was a problem hiding this comment.
The retry attempt initialization here largely duplicates the suite.type === 'scenario' branch in onSuiteStart, but it currently omits the applyTestPlanLabel(...) call and the _decideCucumberSkip/_tpSkipByCid skip handling present in onSuiteStart (around packages/wdio-allure-reporter/src/reporter.ts:509-561). With an active Allure test plan, the retried attempt may miss required labels or be reported even when the scenario should be filtered. Consider refactoring the common scenario-test initialization into a helper that both onSuiteStart and onSuiteRetry call so behavior stays consistent (including test plan labeling/skipping).
| const firstAttemptSuite = cucumberHelper.scenarioEnd({ tests: [cucumberHelper.testFail()], hooks: [] }) | ||
| reporter.onSuiteRetry(firstAttemptSuite as any) |
There was a problem hiding this comment.
This test calls onSuiteRetry with cucumberHelper.scenarioEnd(...), which includes tests/hooks data; however, in the actual WDIO pipeline the suite:retry listener calls suiteStat.retry() before invoking onSuiteRetry, which clears suiteStat.tests/suiteStat.hooks (packages/wdio-reporter/src/index.ts:102-106, packages/wdio-reporter/src/stats/suite.ts:64-69). As written, the test doesn't match the real event object shape and can mask issues in the retry-status/error deduction logic. Consider adjusting the test to pass a SuiteStats instance (or manually clear tests/hooks before calling onSuiteRetry) to reflect production behavior.
| const firstAttemptSuite = cucumberHelper.scenarioEnd({ tests: [cucumberHelper.testFail()], hooks: [] }) | |
| reporter.onSuiteRetry(firstAttemptSuite as any) | |
| // In the WDIO pipeline, suiteStat.retry() clears tests/hooks before onSuiteRetry runs. | |
| const firstAttemptSuite: any = cucumberHelper.scenarioEnd({ tests: [cucumberHelper.testFail()], hooks: [] }) | |
| firstAttemptSuite.tests = [] | |
| firstAttemptSuite.hooks = [] | |
| reporter.onSuiteRetry(firstAttemptSuite) |
Closes #15177
Proposed changes
This commit fixes retry handling by gracefully isolating and terminating the aborted first attempt rather than letting it bleed into the second.
In onSuiteRetry, we now accurately deduce the status of the initial attempt, forcibly terminate it with a "retried" tag, and start a fresh Allure test for the next attempt
In onTestStart, I have implemented scenario collision detection to identify orphaned attempts (caused by scenarioLevelReporter), terminating them natively as
FAILEDbefore initializing Attempt #2.Types of changes
Checklist
Backport Request
//: # (The current
mainbranch is the development branch for WebdriverIO v9. If your change should be released to the current major version of WebdriverIO (v8), please raise another PR with the same changes against thev8branch.)v9and doesn't need to be back-ported#XXXXXFurther comments
Reviewers: @webdriverio/project-committers