Test regressions with black-24.2.0

[mgorny]

After upgrading to black-24.2.0 (24.1.1 seems fine), I'm getting the following test failures on top of 307e87d:

$ python -m pytest -vv
========================================================= test session starts =========================================================
platform linux -- Python 3.11.8, pytest-8.0.0, pluggy-1.4.0 -- /tmp/python-lsp-black/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/python-lsp-black
collected 21 items                                                                                                                    

tests/test_plugin.py::test_pylsp_format_document PASSED                                                                         [  4%]
tests/test_plugin.py::test_pyls_format_pyi_document PASSED                                                                      [  9%]
tests/test_plugin.py::test_pylsp_format_document_unchanged PASSED                                                               [ 14%]
tests/test_plugin.py::test_pyls_format_pyi_document_unchanged PASSED                                                            [ 19%]
tests/test_plugin.py::test_pylsp_format_document_syntax_error PASSED                                                            [ 23%]
tests/test_plugin.py::test_pylsp_format_document_with_config PASSED                                                             [ 28%]
tests/test_plugin.py::test_pylsp_format_range[0-0-a = "hello"\n] PASSED                                                         [ 33%]
tests/test_plugin.py::test_pylsp_format_range[1-1-b = [\n    "a",\n    "very",\n    "very",\n    "very",\n    "very",\n    "very",\n    "very",\n    "very",\n    "very",\n    "long",\n    "line",\n]\n] PASSED [ 38%]
tests/test_plugin.py::test_pylsp_format_range[2-2-c = 42\n] PASSED                                                              [ 42%]
tests/test_plugin.py::test_pylsp_format_range[0-2-a = "hello"\nb = [\n    "a",\n    "very",\n    "very",\n    "very",\n    "very",\n    "very",\n    "very",\n    "very",\n    "very",\n    "long",\n    "line",\n]\nc = 42\n] PASSED [ 47%]
tests/test_plugin.py::test_pylsp_format_range_unchanged PASSED                                                                  [ 52%]
tests/test_plugin.py::test_pylsp_format_range_syntax_error PASSED                                                               [ 57%]
tests/test_plugin.py::test_load_config PASSED                                                                                   [ 61%]
tests/test_plugin.py::test_load_config_target_version PASSED                                                                    [ 66%]
tests/test_plugin.py::test_load_config_defaults FAILED                                                                          [ 71%]
tests/test_plugin.py::test_load_config_with_skip_options FAILED                                                                 [ 76%]
tests/test_plugin.py::test_entry_point PASSED                                                                                   [ 80%]
tests/test_plugin.py::test_pylsp_format_crlf_document PASSED                                                                    [ 85%]
tests/test_plugin.py::test_pylsp_format_line_length PASSED                                                                      [ 90%]
tests/test_plugin.py::test_cache_config PASSED                                                                                  [ 95%]
tests/test_plugin.py::test_pylsp_settings PASSED                                                                                [100%]

============================================================== FAILURES ===============================================================
______________________________________________________ test_load_config_defaults ______________________________________________________

config = {'fast': False, 'line_length': 88, 'preview': False, 'pyi': False, ...}

    def test_load_config_defaults(config):
        config = load_config(str(fixtures_dir / "example.py"), config)
    
>       assert config == {
            "line_length": 88,
            "target_version": set(),
            "pyi": False,
            "fast": False,
            "skip_magic_trailing_comma": False,
            "skip_string_normalization": False,
            "preview": False,
        }
E       AssertionError: assert {'line_length': 88, 'fast': False, 'pyi': False, 'skip_string_normalization': False, 'skip_magic_trailing_comma': False, 'target_version': {<TargetVersion.PY39: 9>, <TargetVersion.PY38: 8>, <TargetVersion.PY311: 11>, <TargetVersion.PY310: 10>}, 'preview': False} == {'line_length': 88, 'target_version': set(), 'pyi': False, 'fast': False, 'skip_magic_trailing_comma': False, 'skip_string_normalization': False, 'preview': False}
E         
E         Common items:
E         {'fast': False,
E          'line_length': 88,
E          'preview': False,
E          'pyi': False,
E          'skip_magic_trailing_comma': False,
E          'skip_string_normalization': False}
E         Differing items:
E         {'target_version': {<TargetVersion.PY39: 9>, <TargetVersion.PY38: 8>, <TargetVersion.PY311: 11>, <TargetVersion.PY310: 10>}} != {'target_version': set()}
E         
E         Full diff:
E           {
E               'fast': False,
E               'line_length': 88,
E               'preview': False,
E               'pyi': False,
E               'skip_magic_trailing_comma': False,
E               'skip_string_normalization': False,
E         -     'target_version': set(),
E         ?                       ^^^^^^
E         +     'target_version': {
E         ?                       ^
E         +         <TargetVersion.PY38: 8>,
E         +         <TargetVersion.PY39: 9>,
E         +         <TargetVersion.PY310: 10>,
E         +         <TargetVersion.PY311: 11>,
E         +     },
E           }

tests/test_plugin.py:282: AssertionError
_________________________________________________ test_load_config_with_skip_options __________________________________________________

config_with_skip_options = <pylsp.config.config.Config object at 0x7f52a3b257d0>

    def test_load_config_with_skip_options(config_with_skip_options):
        config = load_config(
            str(fixtures_dir / "skip_options" / "example.py"), config_with_skip_options
        )
    
>       assert config == {
            "line_length": 88,
            "target_version": set(),
            "pyi": False,
            "fast": False,
            "skip_magic_trailing_comma": True,
            "skip_string_normalization": True,
            "preview": False,
        }
E       AssertionError: assert {'line_length': 88, 'fast': False, 'pyi': False, 'skip_string_normalization': True, 'skip_magic_trailing_comma': True, 'target_version': {<TargetVersion.PY39: 9>, <TargetVersion.PY38: 8>, <TargetVersion.PY311: 11>, <TargetVersion.PY310: 10>}, 'preview': False} == {'line_length': 88, 'target_version': set(), 'pyi': False, 'fast': False, 'skip_magic_trailing_comma': True, 'skip_string_normalization': True, 'preview': False}
E         
E         Common items:
E         {'fast': False,
E          'line_length': 88,
E          'preview': False,
E          'pyi': False,
E          'skip_magic_trailing_comma': True,
E          'skip_string_normalization': True}
E         Differing items:
E         {'target_version': {<TargetVersion.PY39: 9>, <TargetVersion.PY38: 8>, <TargetVersion.PY311: 11>, <TargetVersion.PY310: 10>}} != {'target_version': set()}
E         
E         Full diff:
E           {
E               'fast': False,
E               'line_length': 88,
E               'preview': False,
E               'pyi': False,
E               'skip_magic_trailing_comma': True,
E               'skip_string_normalization': True,
E         -     'target_version': set(),
E         ?                       ^^^^^^
E         +     'target_version': {
E         ?                       ^
E         +         <TargetVersion.PY38: 8>,
E         +         <TargetVersion.PY39: 9>,
E         +         <TargetVersion.PY310: 10>,
E         +         <TargetVersion.PY311: 11>,
E         +     },
E           }

tests/test_plugin.py:298: AssertionError
========================================================== warnings summary ===========================================================
tests/test_plugin.py:8
  /tmp/python-lsp-black/tests/test_plugin.py:8: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    import pkg_resources

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= short test summary info =======================================================
FAILED tests/test_plugin.py::test_load_config_defaults - AssertionError: assert {'line_length': 88, 'fast': False, 'pyi': False, 'skip_string_normalization': False, 'skip_magic_trailing_c...
FAILED tests/test_plugin.py::test_load_config_with_skip_options - AssertionError: assert {'line_length': 88, 'fast': False, 'pyi': False, 'skip_string_normalization': True, 'skip_magic_trailing_co...
=============================================== 2 failed, 19 passed, 1 warning in 0.58s ===============================================

[josch]

I found the same problem in Debian when trying to build this and reported it here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1064266

[josch]

If I understand the code in pylsp_black/plugin.py correctly, then black.find_project_root in _load_config() is supposed to find the project root of tests/fixtures/example.py which routes to the pyproject.toml of python-lsp-black itself. Then further on in that function it queries:

file_config = pyproject_toml.get("tool", {}).get("black", {})

And the value of target-version in tool.black is:

[tool.black]
target-version = ['py38', 'py39', 'py310', 'py311']

So I don't know why this test expected target-version to be the empty set. It should've been listing 38, 39, 310 and 311 in the first place. My theory, that the culprit is this commit:

psf/black@2623269

Which fixes this issue: psf/black#2863

The explanation is, that in the past, black was returning an empty tool.black from ./tests/fixtures/pyproject.toml but now it returns data from the toplevel pyproject.toml.

What do you think?

[ccordoba12]

Sounds good to me. But then fixing this would require increasing Black to 24.2.0? Or do you prefer to make the test result conditional on its version?