Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestNg Cucumber Test Suite is repeated as a whole #164

Open
Tracked by #279
realdadfish opened this issue Dec 19, 2022 · 2 comments
Open
Tracked by #279

TestNg Cucumber Test Suite is repeated as a whole #164

realdadfish opened this issue Dec 19, 2022 · 2 comments

Comments

@realdadfish
Copy link

We're using cucumber-testng-7.8.1 together with testng-7.6.1 and try to setup test retry with this Gradle plugin.

This is one test class for a big set of tests that are largely divided by Cucumber tags:

@CucumberOptions(
    features = [
        "src/test/resources/features/"
    ],
    glue = [
        "our.glue.code",
        "more.shared.glue.code"
    ],
    plugin = [
        "pretty",
        "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm",
        "junit:build/reports/junit/test-results.xml",
        "json:build/reports/cucumber/cucumber-json-report.json"
    ],
    tags = "@dab-gsd and (not @ignore)"
)
@Test
@Listeners(TestLifecycleListenerGsd::class)
class TestRunnerGsd : AbstractTestNGCucumberTests()

Now we figured that when a single test fails not the single test is re-executed, but the whole TestRunnerGsd at once. We've configured no class filters or the like, just enabled the plugin and set maxRetries to 3.

Is retrying Cucumber tests on a per-scenario level supported by this plugin?

@leonard84
Copy link
Member

It does not support TestNG and might not fit your use-case, but if you just want individually retry-able cucumber tests, you might want to take a look at https://github.com/gradle/cucumber-companion.

@pshevche pshevche added the enhancement New feature or request label Apr 2, 2024
@pshevche pshevche removed the enhancement New feature or request label Apr 23, 2024
@pshevche
Copy link
Member

I looked briefly into it. You are right; at the moment, all scenarios in the feature will be retried. This is caused by how TestNG executes and reports individual scenarios: https://github.com/prasanta-biswas/testng-cucumber/blob/master/src/main/java/cucumber/api/testng/AbstractTestNGCucumberTests.java#L25-L28. Basically, it uses scenarios as parameters for a data-driven test. So, TestNG runs a single method executing all scenarios. Currently, the plugin will retry the declared method including all of its iterations (i.e., all scenarios in case of TestNG).

I don't have a good recommendation at the moment apart from splitting large feature files into multiple. We have an issue to revisit Cucumber support on our roadmap, but there is no schedule for it yet: #279. We will check if we can provide a better support for the TestNG runner in that investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants