Fix TOML configuration errors (#3388)

.pre-commit-config.yaml

@@ -14,11 +14,6 @@ repos:
     hooks:
       - id: codespell
         additional_dependencies: ["tomli>=2.0.1"]
-  - repo: https://github.com/tox-dev/tox-ini-fmt
-    rev: "1.4.1"
-    hooks:
-      - id: tox-ini-fmt
-        args: ["-p", "fix"]
   - repo: https://github.com/tox-dev/pyproject-fmt
     rev: "2.2.4"
     hooks:

docs/changelog/3386.bugfix.rst

@@ -0,0 +1 @@
+Fix error when using ``requires`` within a TOML configuration file - by :user:`gaborbernat`.

docs/changelog/3387.bugfix.rst

@@ -0,0 +1 @@
+Fix error when using ``deps`` within a TOML configuration file - by :user:`gaborbernat`.

docs/changelog/3388.bugfix.rst

@@ -0,0 +1,7 @@
+Multiple fixes for the TOML configuration by :user:`gaborbernat`.:
+
+- Do not fail when there is an empty command within ``commands``.
+- Allow references for ``set_env`` by accepting list of dictionaries for it.
+- Do not try to be smart about reference unrolling, instead allow the user to control it via the ``extend`` flag,
+  available both for ``posargs`` and ``ref`` replacements.
+- The ``ref`` replacements ``raw`` key has been renamed to ``of``.

docs/config.rst

@@ -1296,6 +1296,39 @@ others to avoid repeating the same values:
 If the target table is one of the tox environments variable substitution will be applied on the replaced value,
 otherwise the text will be inserted as is (e.g., here with extra).
 
+Configuration reference
+~~~~~~~~~~~~~~~~~~~~~~~
+.. versionadded:: 4.21
+
+You can reference other configurations via the ``ref`` replacement. This can either be of type:
+
+
+- ``env``, in this case the configuration is loaded from another tox environment, where string substitution will happen
+  in that environments scope:
+
+    .. code-block:: toml
+
+        [env.src]
+        extras = ["A", "{env_name}"]
+        [env.dest]
+        extras = [{ replace = "ref", env = "src", key = "extras", extend = true }, "B"
+
+  In this case ``dest`` environments ``extras`` will be ``A``, ``src``, ``B``.
+
+- ``raw``, in this case the configuration is loaded as raw, and substitution executed in the current environments scope:
+
+    .. code-block:: toml
+
+        [env.src]
+        extras = ["A", "{env_name}"]
+        [env.dest]
+        extras = [{ replace = "ref", of = ["env", "extras"], extend = true }, "B"]
+
+  In this case ``dest`` environments ``extras`` will be ``A``, ``dest``, ``B``.
+
+The ``extend`` flag controls if after replacement the value should be replaced as is in the host structure (when flag is
+false -- by default) or be extended into. This flag only operates when the host is a list.
+
 Positional argument reference
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. versionadded:: 4.21
@@ -1305,9 +1338,12 @@ You can reference positional arguments via the ``posargs`` replacement:
 .. code-block:: toml
 
     [env.A]
-    commands = [["python", { replace = "posargs", default = ["a", "b"] } ]]
+    commands = [["python", { replace = "posargs", default = ["a", "b"], extend = true } ]]
 
 If the positional arguments are not set commands will become ``python a b``, otherwise will be ``python posarg-set``.
+The ``extend`` option instructs tox to unroll the positional arguments within the host structure. Without it the result
+would become ``["python", ["a", "b"]`` which would be invalid.
+
 Note that:
 
 .. code-block:: toml
@@ -1318,48 +1354,60 @@ Note that:
 Differs in sense that the positional arguments will be set as a single argument, while in the original example they are
 passed through as separate.
 
-Environment variable reference
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-.. versionadded:: 4.21
-
-You can reference environment variables via the ``env`` replacement:
+Empty commands groups will be ignored:
 
 .. code-block:: toml
 
     [env.A]
-    set_env.COVERAGE_FILE = { replace = "env", name = "COVERAGE_FILE", default = "ok" }
+    commands = [[], ["pytest]]
 
-If the environment variable is set the the ``COVERAGE_FILE`` will become that, otherwise will default to ``ok``.
+will only invoke pytest. This is especially useful together with posargs allowing you to opt out of running a set of
+commands:
 
-Other configuration reference
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+  .. code-block:: toml
+
+    [env.A]
+    commands = [
+        { replace = "posargs", default = ["python", "patch.py"]},
+        ["pytest"]
+    ]
+
+When running ``tox run -e A`` it will invoke ``python patch.py`` followed by pytest. When running ``tox run -e A --`` it
+will invoke only pytest.
+
+
+Environment variable reference
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. versionadded:: 4.21
 
 You can reference environment variables via the ``env`` replacement:
 
 .. code-block:: toml
 
-    [env_run_base]
-    extras = ["A", "{env_name}"]
-    [env.ab]
-    extras = [{ replace = "ref", raw = ["env_run_base", "extras"] }, "B"]
+    [env.A]
+    set_env.COVERAGE_FILE = { replace = "env", name = "COVERAGE_FILE", default = "ok" }
 
-In this case the ``extras`` for ``ab`` will be ``A``, ``B`` and ``ab``.
+If the environment variable is set the the ``COVERAGE_FILE`` will become that, otherwise will default to ``ok``.
 
-Reference replacement rules
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+References within set_env
+~~~~~~~~~~~~~~~~~~~~~~~~~
+.. versionadded:: 4.21.1
 
-When the replacement happens within a list and the returned value is also of type list the content will be extending the
-list rather than replacing it. For example:
+When you want to inherit ``set_env`` from another environment you can use the feature that if you pass a list of
+dictionaries to ``set_env`` they will be merged together, for example:
 
 .. code-block:: toml
 
-    [env_run_base]
-    extras = ["A"]
-    [env.ab]
-    extras = [{ replace = "ref", raw = ["env_run_base", "extras"] }, "B"]
+        [tool.tox.env_run_base]
+        set_env = { A = "1", B = "2"}
+
+        [tool.tox.env.magic]
+        set_env = [
+            { replace = "ref", of = ["tool", "tox", "env_run_base", "set_env"]},
+            { C = "3", D = "4"},
+        ]
 
-In this case the ``extras`` will be ``'A', 'B'`` rather than ``['A'], 'B'``.  Otherwise the replacement is in-place.
+Here the ``magic`` tox environment will have both ``A``, ``B``, ``C`` and ``D`` environments set.
 
 INI only
 --------

docs/user_guide.rst

@@ -33,18 +33,18 @@ these. The canonical file for this is either a ``tox.toml`` or ``tox.ini`` file.
                 "pytest>=8",
                 "pytest-sugar"
             ]
-            commands = [["pytest", { replace = "posargs", default = ["tests"] }]]
+            commands = [["pytest", { replace = "posargs", default = ["tests"], extend = true }]]
 
             [env.lint]
             description = "run linters"
             skip_install = true
             deps = ["black"]
-            commands = [["black", { replace = "posargs", default = ["."]} ]]
+            commands = [["black", { replace = "posargs", default = ["."], extend = true} ]]
 
             [env.type]
             description = "run type checks"
             deps = ["mypy"]
-            commands = [["mypy", { replace = "posargs", default = ["src", "tests"]} ]]
+            commands = [["mypy", { replace = "posargs", default = ["src", "tests"], extend = true} ]]
 
 
     .. tab:: INI
@@ -133,18 +133,18 @@ When ``<env_name>`` is the name of a specific environment, test environment conf
             "pytest>=8",
             "pytest-sugar"
         ]
-        commands = [["pytest", { replace = "posargs", default = ["tests"] }]]
+        commands = [["pytest", { replace = "posargs", default = ["tests"], extend = true }]]
 
         [env.lint]
         description = "run linters"
         skip_install = true
         deps = ["black"]
-        commands = [["black", { replace = "posargs", default = ["."]} ]]
+        commands = [["black", { replace = "posargs", default = ["."], extend = true} ]]
 
         [env.type]
         description = "run type checks"
         deps = ["mypy"]
-        commands = [["mypy", { replace = "posargs", default = ["src", "tests"]} ]]
+        commands = [["mypy", { replace = "posargs", default = ["src", "tests"], extend = true} ]]
 
 .. tab:: INI
 
@@ -217,7 +217,7 @@ Basic example
             "pytest>=7",
             "pytest-sugar",
         ]
-        commands = [[ "pytest", "tests", { replace = "posargs"} ]]
+        commands = [[ "pytest", "tests", { replace = "posargs", extend = true} ]]
 
 .. tab:: INI
 

src/tox/config/loader/convert.py

@@ -58,6 +58,8 @@ def _to_typing(self, raw: T, of_type: type[V], factory: Factory[V]) -> V:  # noq
         if origin in {list, List}:
             entry_type = of_type.__args__[0]  # type: ignore[attr-defined]
             result = [self.to(i, entry_type, factory) for i in self.to_list(raw, entry_type)]
+            if isclass(entry_type) and issubclass(entry_type, Command):
+                result = [i for i in result if i is not None]
         elif origin in {set, Set}:
             entry_type = of_type.__args__[0]  # type: ignore[attr-defined]
             result = {self.to(i, entry_type, factory) for i in self.to_set(raw, entry_type)}
@@ -160,7 +162,7 @@ def to_path(value: T) -> Path:
 
     @staticmethod
     @abstractmethod
-    def to_command(value: T) -> Command:
+    def to_command(value: T) -> Command | None:
         """
         Convert to a command to execute.
 

src/tox/config/loader/memory.py

@@ -51,7 +51,7 @@ def to_path(value: Any) -> Path:
         return Path(value)
 
     @staticmethod
-    def to_command(value: Any) -> Command:
+    def to_command(value: Any) -> Command | None:
         if isinstance(value, Command):
             return value
         if isinstance(value, str):

src/tox/config/loader/str_convert.py

@@ -72,7 +72,7 @@ def _win32_process_path_backslash(value: str, escape: str, special_chars: str) -
         return "".join(result)
 
     @staticmethod
-    def to_command(value: str) -> Command:
+    def to_command(value: str) -> Command | None:
         """
         At this point, ``value`` has already been substituted out, and all punctuation / escapes are final.
 

src/tox/config/loader/toml/__init__.py

@@ -96,8 +96,10 @@ def to_path(value: TomlTypes) -> Path:
         return Path(TomlLoader.to_str(value))
 
     @staticmethod
-    def to_command(value: TomlTypes) -> Command:
-        return Command(args=cast(List[str], value))  # validated during load in _ensure_type_correct
+    def to_command(value: TomlTypes) -> Command | None:
+        if value:
+            return Command(args=cast(List[str], value))  # validated during load in _ensure_type_correct
+        return None
 
     @staticmethod
     def to_env_list(value: TomlTypes) -> EnvList:

src/tox/config/loader/toml/_replace.py

@@ -38,8 +38,8 @@ def __call__(self, value: TomlTypes, depth: int = 0) -> TomlTypes:  # noqa: C901
             res_list: list[TomlTypes] = []
             for val in value:  # apply replacement for every entry
                 got = self(val, depth)
-                if isinstance(val, dict) and val.get("replace") in {"posargs", "ref"} and isinstance(got, (list, set)):
-                    res_list.extend(got)
+                if isinstance(val, dict) and val.get("replace") and val.get("extend"):
+                    res_list.extend(cast(List[Any], got))
                 else:
                     res_list.append(got)
             value = res_list
@@ -63,18 +63,23 @@ def __call__(self, value: TomlTypes, depth: int = 0) -> TomlTypes:  # noqa: C901
                         self.args,
                     )
                 if replace_type == "ref":  # pragma: no branch
-                    if of := value.get("raw"):
-                        validated_of = cast(List[str], validate(of, List[str]))
-                        return self.loader.load_raw_from_root(self.loader.section.SEP.join(validated_of))
-                    if self.conf is not None:  # pragma: no branch # noqa: SIM102
-                        if (env := value.get("env")) and (key := value.get("key")):  # pragma: no branch
-                            return cast(TomlTypes, self.conf.get_env(cast(str, env))[cast(str, key)])
+                    return self._replace_ref(value, depth)
+
             res_dict: dict[str, TomlTypes] = {}
             for key, val in value.items():  # apply replacement for every entry
                 res_dict[key] = self(val, depth)
             value = res_dict
         return value
 
+    def _replace_ref(self, value: dict[str, TomlTypes], depth: int) -> TomlTypes:
+        if self.conf is not None and (env := value.get("env")) and (key := value.get("key")):
+            return cast(TomlTypes, self.conf.get_env(cast(str, env))[cast(str, key)])
+        if of := value.get("of"):
+            validated_of = cast(List[str], validate(of, List[str]))
+            loaded = self.loader.load_raw_from_root(self.loader.section.SEP.join(validated_of))
+            return self(loaded, depth)
+        return value
+
 
 _REFERENCE_PATTERN = re.compile(
     r"""

src/tox/config/loader/toml/_validate.py

@@ -64,7 +64,16 @@ def validate(val: TomlTypes, of_type: type[T]) -> TypeGuard[T]:  # noqa: C901, P
         if val not in choice:
             msg = f"{val!r} is not one of literal {','.join(repr(i) for i in choice)}"
     elif not isinstance(val, of_type):
-        msg = f"{val!r} is not of type {of_type.__name__!r}"
+        if issubclass(of_type, (bool, str, int)):
+            fail = not isinstance(val, of_type)
+        else:
+            try:  # check if it can be converted
+                of_type(val)  # type: ignore[call-arg]
+                fail = False
+            except Exception:  # noqa: BLE001
+                fail = True
+        if fail:
+            msg = f"{val!r} is not of type {of_type.__name__!r}"
     if msg:
         raise TypeError(msg)
     return cast(T, val)  # type: ignore[return-value] # logic too complicated for mypy

src/tox/config/set_env.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+from functools import reduce
 from pathlib import Path
 from typing import Callable, Iterator, Mapping
 
@@ -10,7 +11,9 @@
 
 
 class SetEnv:
-    def __init__(self, raw: str | dict[str, str], name: str, env_name: str | None, root: Path) -> None:
+    def __init__(  # noqa: C901
+        self, raw: str | dict[str, str] | list[dict[str, str]], name: str, env_name: str | None, root: Path
+    ) -> None:
         self.changed = False
         self._materialized: dict[str, str] = {}  # env vars we already loaded
         self._raw: dict[str, str] = {}  # could still need replacement
@@ -23,6 +26,9 @@ def __init__(self, raw: str | dict[str, str], name: str, env_name: str | None, r
         if isinstance(raw, dict):
             self._raw = raw
             return
+        if isinstance(raw, list):
+            self._raw = reduce(lambda a, b: {**a, **b}, raw)
+            return
         for line in raw.splitlines():  # noqa: PLR1702
             if line.strip():
                 if line.startswith("file|"):

src/tox/config/sets.py

@@ -235,6 +235,13 @@ def set_env_factory(raw: object) -> SetEnv:
             if not (
                 isinstance(raw, str)
                 or (isinstance(raw, dict) and all(isinstance(k, str) and isinstance(v, str) for k, v in raw.items()))
+                or (
+                    isinstance(raw, list)
+                    and all(
+                        isinstance(e, dict) and all(isinstance(k, str) and isinstance(v, str) for k, v in e.items())
+                        for e in raw
+                    )
+                )
             ):
                 raise TypeError(raw)
             return SetEnv(raw, self.name, self.env_name, root)

src/tox/provision.py

@@ -94,7 +94,7 @@ def add_tox_requires_min_version(reqs: list[Requirement]) -> list[Requirement]:
         base=[],  # disable inheritance for provision environments
         package="skip",  # no packaging for this please
         # use our own dependency specification
-        deps=PythonDeps("\n".join(str(r) for r in requires), root=state.conf.core["tox_root"]),
+        deps=PythonDeps(requires, root=state.conf.core["tox_root"]),
         pass_env=["*"],  # do not filter environment variables, will be handled by provisioned tox
         recreate=state.conf.options.recreate and not state.conf.options.no_recreate_provision,
     )

src/tox/pytest.py

@@ -259,7 +259,10 @@ def chdir(self, to: Path | None = None) -> Iterator[None]:
         finally:
             os.chdir(cur_dir)
 
-    def run(self, *args: str, from_cwd: Path | None = None) -> ToxRunOutcome:
+    def run(self, *args: str, from_cwd: Path | None = None, raise_on_config_fail: bool = True) -> ToxRunOutcome:
+        if raise_on_config_fail and args and args[0] in {"c", "config"}:
+            self.monkeypatch.setenv("_TOX_SHOW_CONFIG_RAISE", "1")
+
         with self.chdir(from_cwd):
             state = None
             self._capfd.readouterr()  # start with a clean state - drain