RetryAnalyzer triggered before @AfterMethod

[vlad8x8]

TestNG Version

7.10.2

Expected behavior

RetryAnalyzer triggered after @AfterMethod
ITestResult.wasRetried() returns false in @AfterMethod before retry

Actual behavior

RetryAnalyzer triggered before @AfterMethod
ITestResult.wasRetried() returns true in @AfterMethod before retry

Is the issue reproducible on runner?

Maven

[github-actions]

Hi, @vlad8x8.
We need more information to reproduce the issue.

Please help share a Minimal, Reproducible Example that can be used to recreate the issue.

In addition to sharing a sample, please also add the following details:

• TestNG version being used.
• JDK version being used.
• How was the test run (IDE/Build Tools)?
• How does your build file (pom.xml | build.gradle | build.gradle.kts) look like ?

It would be better if you could share a sample project that can be directly used to reproduce the problem.
Reply to this issue when all information is provided, thank you.

[vlad8x8]

https://github.com/testng-team/testng/blob/master/testng-core/src/main/java/org/testng/internal/invokers/TestInvoker.java#L548C1-L554C38
if willRetry <doing nothing> testResult.setStatus SKIP, testResult.setWasRetried(true)
what's going on here?

[vlad8x8]

https://github.com/vlad8x8/testng-retry-bug

Output:

---------------------------------------------
Try number: 0
    Is this real retry: false

Test ignored.
After method:
    Is result thinks that it was retried? true
---------------------------------------------
Try number: 1
    Is this real retry: true
After method:
    Is result thinks that it was retried? false

[krmahadevan]

what's going on here?

Not sure what the question is @vlad8x8

On a side note, you seem to have spent time on debugging this issue. Would you like to help raise a pull request to get this fixed? We can help you with getting it merged.

[vlad8x8]

Hi @krmahadevan, sorry for that emotional exclamation. I just don't understand why we setWasRetried before doing any retry.
I hope there was some logic in why it was done this way.
It would be nice if somebody could explain that logic. It would help to fix this code.

[krmahadevan]

@vlad8x8

I just don't understand why we setWasRetried before doing any retry.

Your test was run twice. The first iteration failed, and so was retried and hence it was marked as wasRetried. Since it was retried, this test result status gets overridden and set explicitly to skip.
Now when the test runs once again owing to the retry analyser, it passed and so there was no retry. Hence it was marked as not retried.

Your @AfterMethod gets invoked for all iterations. I think your confusion stems from the fact that you are assuming that the ITestResult object that you get represents the overall test result of a test method which is tied to a retry analyser. That is NOT the case. ITestResult object gets created for every invocation of a method (be it test or configuration).

If you add a listener as below you should see the output explaining to you what is going on.

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

import java.util.Collection;

public class Printer implements ITestListener {

    @Override
    public void onFinish(ITestContext context) {
        show("Skipped", context.getSkippedTests().getAllResults());
        show("Failed", context.getFailedTests().getAllResults());
        show("Passed", context.getPassedTests().getAllResults());
    }

    private void show(String prefix, Collection<ITestResult> results) {
        System.err.println(prefix + ", Size = " + results.size());
        for (ITestResult result : results) {
            System.err.println(result + ", Retried ? " + result.wasRetried());
        }
    }
}

The output would look like below:

---------------------------------------------
Try number: 0
    Is this real retry: false

Test ignored.
After method:
    Is result thinks that it was retried? true
---------------------------------------------
Try number: 1
    Is this real retry: true
After method:
    Is result thinks that it was retried? false
Skipped, Size = 1
[TestResult name=test status=SKIP method=MyTest.test()[pri:0, instance:com.github.issue3154.MyTest@7c469c48] output={null}], Retried ? true
Failed, Size = 0
Passed, Size = 1
[TestResult name=test status=SUCCESS method=MyTest.test()[pri:0, instance:com.github.issue3154.MyTest@7c469c48] output={null}], Retried ? false

===============================================
Default Suite
Total tests run: 2, Passes: 1, Failures: 0, Skips: 0, Retries: 1
===============================================