Seeding-generated test doesn't execute properly (AttributeFilter_ESTest)

[vmassol]

Generated on xwiki-commons-xml using the setup from https://github.com/STAMP-project/botsing/issues/89

Results:

[pderakhshanfar]

This is a flaky test generated by EvoSuite. It is not related to model seeding. Sometimes, EvoSuite generates tests that are not 100% pass when you directly execute them.

I should note that the EvoSuite model seeding issues are not related to Botsing. So, it is better to put these kinds of issues here.

[vmassol]

Note that this is not just one "flaky" test, in practice we got 10 generated "flaky" tests out of 257 generated tests (i.e. 4%). That's a big issue (blocker) for being able to automatically commit generated tests.

[pderakhshanfar]

Again, since EvoSuite executes everything in its own environment, it is possible that when you run them in your environment, you face these flaky tests. EvoSuite developers spent lots of effort to minimize them. However, it is still possible. I personally run them 5 times first. If they pass 100% of the time, I will count them as e usable test for line and mutation coverage.

[vmassol]

However, it is still possible. I personally run them 5 times first. If they pass 100% of the time, I will count them as e usable test for line and mutation coverage.

Couldn't you execute the generated tests inside the botsing/evosuite process and remove the flaky tests? This would also ensure that they compile fine. Since the seeding part runs a dynamic analysis, which I assume is executing the tests, you already have a way to execute tests. That could be a post process part of the tool to remove false positives.

[pderakhshanfar]

The final test generation process ( compiling, minimizing, etc.) is in EvoSuite, and changing that part of code may impact other parts of pre-process. It is much easier to do it as a post-process of EvoSuite/Botsing.

[vmassol]

It is much easier to do it as a post-process of EvoSuite/Botsing.

That's exactly what I'm proposing! :)

Since you already have a pre-process step (the seeding part), it doesn't change fundamentally to also do a postprocess part (it's even more symmetrical! ;)). And in the future wrap all of this in a single, simple to use, process (maven plugin, etc).

Just to make it clear this will allow you to "fix" several issues;

• flakiness - ignore these tests
• optionally capture stdout/stderr and discard tests that generate output when executed (a problem for xwiki)
• verify that no file is generated outside a given directory and discard tests that do generate files
• (add more here)