C extensions printing on import break pylint diagnostics

[piotr-machura]

Importing pygame by default displays a greeting:

pygame 2.1.2 (SDL 2.0.18, Python 3.10.1)
Hello from the pygame community. https://www.pygame.org/contribute.html

which seems to confuse the pylint output parser. The greeting can be surpressed with

export PYGAME_HIDE_SUPPORT_PROMPT="hide"

which then fixes the issue.

Steps to reproduce

Create a test directory with a minimal .pylintrc file:

[MASTER]
extension-pkg-whitelist=pygame

and a test window.py file (notice that it still needs import sys in order to run):

import pygame
from pygame.locals import QUIT

pygame.this_is_surely_not_a_pygame_method()

# Some boilerplate to check if pygame works correctly
if __name__ == '__main__':
    pygame.init()
    pygame.display.set_mode((400, 300))
    pygame.display.set_caption('Hello World!')
    while True: # main game loop
        for event in pygame.event.get():
            if event.type == QUIT:
                pygame.quit()
                sys.exit()
        pygame.display.update()

When using pylint directly with pylint window.py we get the expected result, as well as pygame's greeting:

pygame 2.1.2 (SDL 2.0.18, Python 3.10.1)
Hello from the pygame community. https://www.pygame.org/contribute.html
************* Module window
window.py:5:0: E1101: Module 'pygame' has no 'this_is_not_a_method' member (no-member)
window.py:16:16: E0602: Undefined variable 'sys' (undefined-variable)

Pylint correctly picks up the invalid method and lack of import sys. Seems like everything works as expected.

Now, when used from pylsp connected to neovim 0.6.1 these diagnostics can't be obtained (:lua vim.diagnostic.get(0) and :lua vim.lsp.diagnostic.get(0) return an empty table) and the following errors are encountered:

[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'Exception in thread Thread-1:\nTraceback (most recent call last):\n  File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    self.run()\n  File "/usr/lib/python3.10/threading.py", line 1371, in run\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    self.function(*self.args, **self.kwargs)\n  File "/usr/lib/python3.10/site-packages/pylsp/_utils.py", line 33, in run\n    return func(*args, **kwargs)\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	"  File \"/usr/lib/python3.10/site-packages/pylsp/python_lsp.py\", line 299, in lint\n    flatten(self._hook('pylsp_lint', doc_uri, is_saved=is_saved))\n  File \"/usr/lib/python3.10/site-packages/pylsp/python_lsp.py\", line 156, in _hook\n    return hook_handlers(config=self.config, workspace=workspace, document=doc, **kwargs)\n  File \"/usr/lib/python3.10/site-packages/pluggy/hooks.py\", line 286, in __call__\n"
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    return self._hookexec(self, self.get_hookimpls(), kwargs)\n  File "/usr/lib/python3.10/site-packages/pluggy/manager.py", line 93, in _hookexec\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    return self._inner_hookexec(hook, methods, kwargs)\n  File "/usr/lib/python3.10/site-packages/pluggy/manager.py", line 337, in traced_hookexec\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    return outcome.get_result()\n  File "/usr/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    raise ex[1].with_traceback(ex[2])\n  File "/usr/lib/python3.10/site-packages/pluggy/callers.py", line 52, in from_call\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    result = func()\n  File "/usr/lib/python3.10/site-packages/pluggy/manager.py", line 335, in <lambda>\n    outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))\n  File "/usr/lib/python3.10/site-packages/pluggy/manager.py", line 84, in <lambda>\n    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(\n  File "/usr/lib/python3.10/site-packages/pluggy/callers.py", line 208, in _multicall\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    return outcome.get_result()\n  File "/usr/lib/python3.10/site-packages/pluggy/callers.py", line 80, in get_result\n    raise ex[1].with_traceback(ex[2])\n  File "/usr/lib/python3.10/site-packages/pluggy/callers.py", line 187, in _multicall\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    res = hook_impl.function(*args)\n  File "/usr/lib/python3.10/site-packages/pylsp/plugins/pylint_lint.py", line 182, in pylsp_lint\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	'    return PylintLinter.lint(document, is_saved, flags=flags)\n  File "/usr/lib/python3.10/site-packages/pylsp/plugins/pylint_lint.py", line 110, in lint\n'
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	"    for diag in json.loads(json_out):\n"
[ERROR][2022-01-03 02:27:24] .../vim/lsp/rpc.lua:420	"rpc"	"pylsp"	"stderr"	"ValueError: Expected object or value\n"

Now if we run export PYGAME_HIDE_SUPPORT_PROMPT="hide" and try again the greeting gets disabled, the above rpc errors dissapear and everything works just fine. This happens beacuse pygame's greeting gets dumped somewhere in between the JSON output, confusing the json.loads:.

$ pylint window.py -f json
pygame 2.1.2 (SDL 2.0.18, Python 3.10.1)
Hello from the pygame community. https://www.pygame.org/contribute.html
[
    {
        "type": "convention",
        "module": "window",
        "obj": "",
        "line": 18,
        "column": 0,
        "endLine": null,
        "endColumn": null,
        "path": "window.py",
        "symbol": "trailing-newlines",
        "message": "Trailing newlines",
        "message-id": "C0305"
    },
    {
        "type": "error",
        "module": "window",
        "obj": "",
        "line": 5,
        "column": 0,
        "endLine": 5,
        "endColumn": 27,
        "path": "window.py",
        "symbol": "no-member",
        "message": "Module 'pygame' has no 'this_is_not_a_method' member",
        "message-id": "E1101"
    },
    {
        "type": "error",
        "module": "window",
        "obj": "open_window",
        "line": 16,
        "column": 16,
        "endLine": 16,
        "endColumn": 19,
        "path": "window.py",
        "symbol": "undefined-variable",
        "message": "Undefined variable 'sys'",
        "message-id": "E0602"
    }
]

Expected behavior

The greeting should not interfere with pylint's JSON output. I don't know if something can be done here (a hacky "remove everything until first [ before parsing" or "add PYGAME_HIDE_SUPPORT_PROMPT="hide" to environment before calling pylint" should work I guess), but I believe that is really a problem pylint should fix (but they haven't for over a year - see pylint-dev/pylint#3518).

Setup details

$ pylsp --version
v1.3.2

$ pylint --version
pylint 2.12.2
astroid 2.9.1
Python 3.10.1 (main, Dec 18 2021, 23:53:45) [GCC 11.1.0]

$ nvim --version
NVIM v0.6.1
Build type: Release
LuaJIT 2.0.5
Compiled by builduser

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

Run :checkhealth for more info

Used with the following lspconfig (on commit hash #486d51c):

require('lspconfig').pylsp.setup({
  cmd = { 'pylsp' },
  settings = {
    pylsp = {
      plugins = {
        pylint = { enabled = true },
        pydocstyle = { enabled = true, convention = 'google' },
        mccabe = { enabled = false },
      },
    },
  },
})

[ccordoba12]

Hey @piotr-machura, thanks for reporting. Please open a pull request in the Pylint plugin:

https://github.com/python-lsp/python-lsp-server/blob/develop/pylsp/plugins/pylint_lint.py

to set that environment variable on top of it, explaining in a comment the reason why it's necessary.

Thanks!