🔥 Cut comments off dependency lines (#1262)

This enables users to have inline comments in deps lines in `tox.ini`. In particular, this allows having pyup bot integration.

Ref: pyupio/dparse#34
Ref: ansible/molecule#1973

Fixes #1260

docs/changelog/1262.feature.rst

@@ -0,0 +1 @@
+Allow having inline comments in :conf:`deps` — by :user:`webknjaz`

docs/config.rst

@@ -21,9 +21,6 @@ the *ini-style* format under the ``tool.tox.legacy_tox_ini`` key as a multi-line
 Below you find the specification for the *ini-style* format, but you might want to skim some
 :doc:`examples` first and use this page as a reference.
 
-.. _ConfigParser: https://docs.python.org/3/library/configparser.html
-
-
 tox global settings
 -------------------
 
@@ -45,10 +42,10 @@ Global settings are defined under the ``tox`` section as:
     .. versionadded:: 3.2.0
 
     Specify python packages that need to exist alongside the tox installation for the tox build
-    to be able to start. Use this to specify plugin requirements (or the version of ``virtualenv`` -
-    determines the default ``pip``, ``setuptools``, and ``wheel`` versions the tox environments
-    start with). If these dependencies are not specified tox will create :conf:`provision_tox_env`
-    environment so that they are satisfied and delegate all calls to that.
+    to be able to start (must be PEP-508_ compliant). Use this to specify plugin requirements
+    (or the version of ``virtualenv`` - determines the default ``pip``, ``setuptools``, and ``wheel``
+    versions the tox environments start with). If these dependencies are not specified tox will create
+    :conf:`provision_tox_env` environment so that they are satisfied and delegate all calls to that.
 
     .. code-block:: ini
 
@@ -152,10 +149,8 @@ Global settings are defined under the ``tox`` section as:
 
     Activate isolated build environment. tox will use a virtual environment to build
     a source distribution from the source tree. For build tools and arguments use
-    the ``pyproject.toml`` file as specified in
-    `PEP-517 <https://www.python.org/dev/peps/pep-0517/>`_ and
-    `PEP-518 <https://www.python.org/dev/peps/pep-0518/>`_. To specify the virtual
-    environment Python version define use the :conf:`isolated_build_env` config
+    the ``pyproject.toml`` file as specified in `PEP-517`_ and `PEP-518`_. To specify the
+    virtual environment Python version define use the :conf:`isolated_build_env` config
     section.
 
 .. conf:: isolated_build_env ^ string ^ .package
@@ -339,26 +334,32 @@ Complete list of settings that you can put into ``testenv*`` sections:
 
 .. conf:: deps ^ MULTI-LINE-LIST
 
-    Test-specific dependencies - to be installed into the environment prior to project
-    package installation.  Each line defines a dependency, which will be
-    passed to the installer command for processing (see :conf:`indexserver`).
-    Each line specifies a file, a URL or a package name.  You can additionally specify
-    an :conf:`indexserver` to use for installing this dependency
-    but this functionality is deprecated since tox-2.3.
-    All derived dependencies (deps required by the dep) will then be
-    retrieved from the specified indexserver:
+    Environment dependencies - installed into the environment ((see :conf:`install_command`) prior
+    to project after environment creation. One dependency (a file, a URL or a package name) per
+    line. Must be PEP-508_ compliant. All installer commands are executed using the toxinidir_ as the
+    current working directory.
 
     .. code-block:: ini
 
-        [tox]
-        indexserver =
-            myindexserver = https://myindexserver.example.com/simple
-
         [testenv]
-        deps = :myindexserver:pkg
+        deps =
+            pytest
+            pytest-cov >= 3.5
+            pywin32 >=1.0 ; sys_platform == 'win32'
+            octomachinery==0.0.13  # pyup: < 0.1.0 # disable feature updates
+
 
-    (Experimentally introduced in 1.6.1) all installer commands are executed
-    using the ``{toxinidir}`` as the current working directory.
+    .. versionchanged:: 2.3
+
+    Support for index servers is now deprecated, and it's usage discouraged.
+
+    .. versionchanged:: 3.9
+
+    Comment support on the same line as the dependency. When feeding the content to the install
+    tool we'll strip off content (including) from the first comment marker (``#``)
+    preceded by one or more space. For example, if a dependency is
+    ``octomachinery==0.0.13  # pyup: < 0.1.0 # disable feature updates`` it will be turned into
+    just ``octomachinery==0.0.13``.
 
 .. conf:: platform ^ REGEX
 
@@ -559,8 +560,12 @@ having value magic).
 Globally available substitutions
 ++++++++++++++++++++++++++++++++
 
+.. _`toxinidir`:
+
 ``{toxinidir}``
-    the directory where tox.ini is located
+    the directory where ``tox.ini`` is located
+
+.. _`toxworkdir`:
 
 ``{toxworkdir}``
     the directory where virtual environments are created and sub directories
@@ -846,11 +851,11 @@ special case for a combination of factors. Here is how you do it:
 
     [testenv]
     deps =
-        py34-mysql: PyMySQL     ; use if both py34 and mysql are in the env name
-        py27,py36: urllib3      ; use if either py36 or py27 are in the env name
-        py{27,36}-sqlite: mock  ; mocking sqlite in python 2.x & 3.6
-        !py34-sqlite: mock      ; mocking sqlite, except in python 3.4
-        sqlite-!py34: mock      ; (same as the line above)
+        py34-mysql: PyMySQL     # use if both py34 and mysql are in the env name
+        py27,py36: urllib3      # use if either py36 or py27 are in the env name
+        py{27,36}-sqlite: mock  # mocking sqlite in python 2.x & 3.6
+        !py34-sqlite: mock      # mocking sqlite, except in python 3.4
+        sqlite-!py34: mock      # (same as the line above)
 
 Take a look at the first ``deps`` line. It shows how you can special case
 something for a combination of factors, by just hyphenating the combining

docs/example/package.rst

@@ -4,13 +4,12 @@ packaging
 Although one can use tox to develop and test applications one of its most popular
 usage is to help library creators. Libraries need first to be packaged, so then
 they can be installed inside a virtual environment for testing. To help with this
-tox implements `PEP-517 <https://www.python.org/dev/peps/pep-0517/>`_ and
-`PEP-518 <https://www.python.org/dev/peps/pep-0518/>`_. This means that by default
+tox implements PEP-517_ and PEP-518_. This means that by default
 tox will build source distribution out of source trees. Before running test commands
 ``pip`` is used to install the source distribution inside the build environment.
 
-To create a source distribution there are multiple tools out there and with ``PEP-517``
-and ``PEP-518`` you can easily use your favorite one with tox. Historically tox
+To create a source distribution there are multiple tools out there and with PEP-517_ and PEP-518_
+you can easily use your favorite one with tox. Historically tox
 only supported ``setuptools``, and always used the tox host environment to build
 a source distribution from the source tree. This is still the default behavior.
 To opt out of this behaviour you need to set isolated builds to true.
@@ -37,7 +36,7 @@ build requirements.
 
 flit
 ----
-`flit <https://flit.readthedocs.io/en/latest/>`_ requires ``Python 3``, however the generated source
+flit_ requires ``Python 3``, however the generated source
 distribution can be installed under ``python 2``. Furthermore it does not require a ``setup.py``
 file as that information is also added to the ``pyproject.toml`` file.
 
@@ -68,7 +67,7 @@ file as that information is also added to the ``pyproject.toml`` file.
 
 poetry
 ------
-`poetry <https://poetry.eustace.io/>`_ requires ``Python 3``, however the generated source
+poetry_ requires ``Python 3``, however the generated source
 distribution can be installed under ``python 2``. Furthermore it does not require a ``setup.py``
 file as that information is also added to the ``pyproject.toml`` file.
 
@@ -95,3 +94,5 @@ file as that information is also added to the ``pyproject.toml`` file.
    # so unless this is python 3 you can require a given python version for the packaging
    # environment via the basepython key
    basepython = python3
+
+.. include:: ../links.rst

docs/links.rst

@@ -9,6 +9,7 @@
 .. _`nose`: https://pypi.org/project/nose
 .. _`Holger Krekel`: https://twitter.com/hpk42
 .. _`pytest-xdist`: https://pypi.org/project/pytest-xdist
+.. _ConfigParser: https://docs.python.org/3/library/configparser.html
 
 .. _`easy_install`: http://peak.telecommunity.com/DevCenter/EasyInstall
 .. _pip: https://pypi.org/project/pip
@@ -18,7 +19,13 @@
 .. _discover: https://pypi.org/project/discover
 .. _unittest2: https://pypi.org/project/unittest2
 .. _mock: https://pypi.org/project/mock/
+.. _flit: https://flit.readthedocs.io/en/latest/
+.. _poetry: https://poetry.eustace.io/
 .. _pypy: https://pypy.org
 
 .. _`Python Packaging Guide`: https://packaging.python.org/tutorials/packaging-projects/
 .. _`tox.ini`: :doc:configfile
+
+.. _`PEP-508`: https://www.python.org/dev/peps/pep-0508/
+.. _`PEP-517`: https://www.python.org/dev/peps/pep-0517/
+.. _`PEP-518`: https://www.python.org/dev/peps/pep-0518/

setup.cfg

@@ -64,8 +64,9 @@ testing =
     pytest-xdist >= 1.22.2, <2
     pytest-randomly >= 1.2.3, <2
     psutil >= 5.6.1, < 6; python_version != "3.4"
+    flaky >= 3.4.0, < 4
 docs =
-    sphinx >= 1.8.0, < 2
+    sphinx >= 2.0.0, < 3
     towncrier >= 18.5.0
     pygments-github-lexers >= 0.0.5
     sphinxcontrib-autoprogram >= 0.1.5

src/tox/config/__init__.py

@@ -158,6 +158,7 @@ def postprocess(self, testenv_config, value):
                     name_start = "{} ".format(option)
                     if name.startswith(name_start):
                         name = "{}={}".format(option, name[len(option) :].strip())
+            name = self._cut_off_dep_comment(name)
             name = self._replace_forced_dep(name, config)
             deps.append(DepConfig(name, ixserver))
         return deps
@@ -176,6 +177,10 @@ def _replace_forced_dep(self, name, config):
                 return forced_dep
         return name
 
+    @staticmethod
+    def _cut_off_dep_comment(name):
+        return re.sub(r"\s+#.*", "", name).strip()
+
     @classmethod
     def _is_same_dep(cls, dep1, dep2):
         """Definitions are the same if they refer to the same package, even if versions differ."""

tests/integration/test_parallel_interrupt.py

@@ -6,11 +6,13 @@
 from datetime import datetime
 
 import pytest
+from flaky import flaky
 from pathlib2 import Path
 
 from tox.util.main import MAIN_FILE
 
 
+@flaky(max_runs=3)
 @pytest.mark.skipif(
     "sys.platform == 'win32'", reason="triggering SIGINT reliably on Windows is hard"
 )

tests/unit/config/test_config.py

@@ -142,7 +142,9 @@ def test_process_deps(self, newconfig):
             [testenv]
             deps =
                 -r requirements.txt
+                yapf>=0.25.0,<0.27  # pyup: < 0.27 # disable updates
                 --index-url https://pypi.org/simple
+                pywin32 >=1.0 ; sys_platform == '#my-magic-platform' # so what now
                 -fhttps://pypi.org/packages
                 --global-option=foo
                 -v dep1
@@ -151,7 +153,9 @@ def test_process_deps(self, newconfig):
         )  # note that those last two are invalid
         expected_deps = [
             "-rrequirements.txt",
+            "yapf>=0.25.0,<0.27",
             "--index-url=https://pypi.org/simple",
+            "pywin32 >=1.0 ; sys_platform == '#my-magic-platform'",
             "-fhttps://pypi.org/packages",
             "--global-option=foo",
             "-v dep1",

tests/unit/package/test_package_parallel.py

@@ -1,16 +1,13 @@
 import os
-import platform
 import traceback
 
 import py
-import pytest
+from flaky import flaky
 
 from tox.session.commands.run import sequential
 
 
-@pytest.mark.skipif(
-    platform.python_implementation().lower() == "pypy", reason="this is flaky on pypy"
-)
+@flaky(max_runs=3)
 def test_tox_parallel_build_safe(initproj, cmd, mock_venv, monkeypatch):
     initproj(
         "env_var_test",

tox.ini

@@ -124,7 +124,7 @@ source = src/tox
          *\src\tox
 
 [pytest]
-addopts = -ra --showlocals
+addopts = -ra --showlocals --no-success-flaky-report
 rsyncdirs = tests tox
 looponfailroots = tox tests
 testpaths = tests
@@ -136,7 +136,7 @@ include_trailing_comma = True
 force_grid_wrap = 0
 line_length = 99
 known_first_party = tox,tests
-known_third_party = apiclient,docutils,filelock,freezegun,git,httplib2,oauth2client,packaging,pathlib2,pkg_resources,pluggy,py,pytest,setuptools,six,sphinx,toml
+known_third_party = apiclient,docutils,filelock,flaky,freezegun,git,httplib2,oauth2client,packaging,pathlib2,pkg_resources,pluggy,py,pytest,setuptools,six,sphinx,toml
 
 [testenv:release]
 description = do a release, required posarg of the version number