Regression reported intermittently in Jenkins CI for the Scenario Thememapper LESS builder Robot test

[rpatterson]

BUG/PROBLEM REPORT (OR OTHER COMMON ISSUE)

What I did:

I submitted a plone.restapi PR and requested that the bot run the Jenkins CI jobs .

What I expect to happen:

I expect the CI results to report failures if they're actually related to my changes.

What actually happened:

Jenkins CI reported a failure from a Robot test that is almost certainly unrelated to my PR. I was informed by @wesleybl that it was common for the Robot tests to fail intermittently in CI and to re-run the job. I did re-run the job and indeed the test passed this time.

What version of Plone/ Addons I am using:

I believe that Jenkins CI job is running against Plone 6.0 and Python 3.8 but see that job for further version details.

[rpatterson]

I think an intermittent test is often worse than no test. It frustrates the community of developers and trains them to ignore tests over time. We want our tests to help developers fall in love with testing. This goes double for CI. Does Robot Framework have an equivalent of Python's unittest.TestCase.skipTest(...) that we can use until someone who understands the test can fix the intermittent issue?

[gforcada]

Yes, there is, one needs to add a tag to the robot test, I don't have the tag here at hand, I can try t look it up later

[rpatterson]

Yes, there is, one needs to add a tag to the robot test, I don't have the tag here at hand, I can try t look it up later

Don't suppose you can provide me an example of using that tag? I'd be happy to submit a PR applying it to these offending Robot tests.

[mauritsvanrees]

I can't search properly on my mobile, but I think the tag is 'unstable'. There should be examples in CMFPlone/tests/robots, but you may need to look at 4.3 or 5.0.

[mauritsvanrees]

This test seems stable on 5.2, at least it does not ring a bell for me anymore.
And on 6.0 the theme mapper functionality is gone, so not problems there either.