Smoke test pytest-rerunfailures

[hugovk]

This adds pytest-rerunfailures to the plugin smoke tests.

It will currently fail, because of the 6.1.0 removals: pytest-dev/pytest-rerunfailures#128, which has affected at least pip (pypa/pip#8925) and matplotlib (matplotlib/matplotlib#18593).

But once it's been fixed, this can be merged and hopefully catch the next removals.

[bluetech]

LGTM, once it's green of course.

[hugovk]

Rebased to trigger a new build because a fixed pytest-rerunfailures 9.1.1 has just been released which supports newest pytest 6.1:

• https://github.com/pytest-dev/pytest-rerunfailures/blob/master/CHANGES.rst#911-2020-09-29

[bluetech]

@hugovk I think it fails now because the pytest version in master is like pytest 6.0.1.dev224+g7386176f4.d20200929 rather than 6.1.0.stuff, so the version checked in rerunfailures doesn't work. This happens because the 6.1.0 tag is only in the 6.1.x branch and setuptools-scm can't find it in master... Dunno how this worked in previous releases...

[hugovk]

Yes, that must be it, rerunfailures has:

PYTEST_GTE_61 = pkg_resources.parse_version(
    pytest.__version__
) >= pkg_resources.parse_version("6.1")

https://github.com/pytest-dev/pytest-rerunfailures/blob/b5e70389ea9bfbbcb94bafb924237887b37bcf78/pytest_rerunfailures.py#L13-L15

And with pytest master:

>>> import pytest
>>> import pkg_resources
>>> 
>>> pytest.__version__
'6.0.1.dev223+gcd67c2a8c'
>>> pkg_resources.parse_version(
...     pytest.__version__
... ) >= pkg_resources.parse_version("6.1")
False
>>> pkg_resources.parse_version(
...     pytest.__version__
... ) >= pkg_resources.parse_version("6.0")
True

It sounds like a good idea in general to have tags on master as well. Would this be possible? And update the release checklist?

(This sort of thing has caused problems with pre-commit recently: PyCQA/isort#1371 + cheshirekow/cmake-format-precommit#3)

[hugovk]

The lack of tag on master also has a side effect that the released 6.1.0 is considered newer than master, pip install -U pytest uninstalls master and "downgrades" to 6.1.0:

⌂76% [hugo:~/github/pytest] master+* ± pip install -e .
Obtaining file:///Users/hugo/github/pytest
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: attrs>=19.2.0 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest==6.0.1.dev223+gcd67c2a8c) (19.3.0)
Requirement already satisfied: iniconfig in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest==6.0.1.dev223+gcd67c2a8c) (1.0.0)
Requirement already satisfied: packaging in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest==6.0.1.dev223+gcd67c2a8c) (20.4)
Requirement already satisfied: py>=1.8.2 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest==6.0.1.dev223+gcd67c2a8c) (1.9.0)
Requirement already satisfied: toml in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest==6.0.1.dev223+gcd67c2a8c) (0.10.1)
Requirement already satisfied: pluggy<1.0,>=0.12 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest==6.0.1.dev223+gcd67c2a8c) (0.13.1)
Requirement already satisfied: six in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from packaging->pytest==6.0.1.dev223+gcd67c2a8c) (1.15.0)
Requirement already satisfied: pyparsing>=2.0.2 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from packaging->pytest==6.0.1.dev223+gcd67c2a8c) (2.4.6)
Installing collected packages: pytest
  Running setup.py develop for pytest
Successfully installed pytest

⌂83% [hugo:~/github/pytest] master+* 21s ± pip install -U pytest
Collecting pytest
  Using cached pytest-6.1.0-py3-none-any.whl (272 kB)
Requirement already satisfied, skipping upgrade: toml in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest) (0.10.1)
Requirement already satisfied, skipping upgrade: attrs>=17.4.0 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest) (19.3.0)
Requirement already satisfied, skipping upgrade: pluggy<1.0,>=0.12 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest) (0.13.1)
Requirement already satisfied, skipping upgrade: py>=1.8.2 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest) (1.9.0)
Requirement already satisfied, skipping upgrade: iniconfig in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest) (1.0.0)
Requirement already satisfied, skipping upgrade: packaging in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from pytest) (20.4)
Requirement already satisfied, skipping upgrade: six in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from packaging->pytest) (1.15.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from packaging->pytest) (2.4.6)
Installing collected packages: pytest
  Attempting uninstall: pytest
    Found existing installation: pytest 6.0.1.dev223+gcd67c2a8c
    Uninstalling pytest-6.0.1.dev223+gcd67c2a8c:
      Successfully uninstalled pytest-6.0.1.dev223+gcd67c2a8c
Successfully installed pytest-6.1.0

[bluetech]

We can merge the 6.1.0 tag but it doesn't seem like a great solution to me. To be consistent we'll need to merge also the patch tags which is something we wanted to get away from. Besides, master is really work for the next version, so having it say 6.1.* is misleading.

I can't think of a good solution other than stopping to use scm-setuptools or somehow tricking it on master... Maybe tag the commit in master after the feature release is branched as 6.2.0a1? In the current state we can tag e0ea00a for example.

Other suggestions welcome

[nicoddemus]

or somehow tricking it on master

We can set the SETUPTOOLS_SCM_PRETEND_VERSION environment variable to trick setuptools scm to use 6.2.0a1 as version. I prefer using that to actual tagging as tagging a repository usually means a new release, which isn't true.

[nicoddemus]

@hugovk feel free to do the honors of merging. 😁

[hugovk]

Thanks!