Issue in pytest-cases 1.8.1 with markers

[browniebroke]

First of all, thanks for this neat library, it has simplified our tests quite a bit for cookiecutter-django.

I'm not sure if I missed a change in 1.8 or if it's an actual bug, but it seems that 1.8.1 is not playing nicely with pytest markers. We use markers to spread markers over multiple workers on our CI:

• One worker runs tests marked with black
• One worker runs tests marked with flake8
• A last worker runs all unmarked tests flake8

We basically have 3 test function which all use a pretty big parametrized fixture.

Until <1.8.1, the black output was running in ~8 mins. With 1.8.1, it takes ~20 mins to run and judging by the output, it looks like a lot more test cases are executed.

I tried to bump to 1.8.0 and the output is consistent with the existing behaviour and execution around ~9mins.

Am I still using the library as it's intended or is there a regression in 1.8.1?

[smarie]

Thanks for the feedback !

Two main things changed indeed

• the pytest optimized order for session- and module-scoped fixtures was disabled thanks to the pytest_collection_modifyitems hook. I already noted in Instead of always disabling the pytest "smart ordering" for session and module scoped fixtures, propose an option #45 that this was too dangerous and should be only done with an explicit commandline option. This should only have an impact about the number of setups/teardowns, not about the number of tests
• the fixtures that have no parameters at all are now parameterized with either a single or two values, depending on whether they are always used or not in tests where fixture unions are used. I would rather bet that this one had the side effects that you mention. In which case this is a bug, I'll have to have a look.

I'll try to fix both asap, probably on monday.
It would help if you are able to provide a smaller prototypic example of the problem if you are able to find it. For example if you can look at the list of test ids before and after, to see if they are different ? Thanks !

[smarie]

I found the issue, it was with the reordering. Let me know if 1.9.0 fixes your problem (it should)

[browniebroke]

Yea, looks like the execution time and load spreading is back to usual figures, thanks for the quick fix!

[smarie]

you're welcome, thanks for the feedback !