Lack of stacktrace during multi-processing crashes makes debugging harder

[owillebo]

pylint 2.7.x crashes when doing a 7 job run with about 480 files.
This does not happen with pylint 2.6.x

Steps to reproduce

Run;

pylint -j7 [with about 480 files]

Current behavior

Crash with following stack trace;

Traceback (most recent call last):
  File "c:\program files\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\program files\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python39\Scripts\pylint.exe\__main__.py", line 7, in <module>
  File "c:\program files\python39\lib\site-packages\pylint\__init__.py", line 22, in run_pylint
    PylintRun(sys.argv[1:])
  File "c:\program files\python39\lib\site-packages\pylint\lint\run.py", line 358, in __init__
    linter.check(args)
  File "c:\program files\python39\lib\site-packages\pylint\lint\pylinter.py", line 866, in check
    check_parallel(
  File "c:\program files\python39\lib\site-packages\pylint\lint\parallel.py", line 138, in check_parallel
    for (
  File "c:\program files\python39\lib\multiprocessing\pool.py", line 870, in next
    raise value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x00000206534E67F0>'. Reason: 'PicklingError("Can't pickle <enum 'Context'>: attribute lookup Context on astroid failed")'

Expected behavior

No crash

pylint --version output

Crashing pylint version;
pylint 2.7.2
astroid 2.5.1
Python 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]

Not crashing pylint version;
pylint 2.6.2
astroid 2.4.2
Python 3.9.2 (tags/v3.9.2:1a79785, Feb 19 2021, 13:44:55) [MSC v.1928 64 bit (AMD64)]

[owillebo]

I found the root cause, see #4215.

When linting the code mentioned in #4215 with more than 1 job you get the MaybeEncodingError exception, i.e.

With bug.py holding;

  if len(dont_care[0]):
      pass

You get;

                C:\Temp>pylint -j2 bug.py
                Traceback (most recent call last):
                  File "c:\program files\python39\lib\runpy.py", line 197, in _run_module_as_main
                    return _run_code(code, main_globals, None,
                  File "c:\program files\python39\lib\runpy.py", line 87, in _run_code
                    exec(code, run_globals)
                  File "C:\Program Files\Python39\Scripts\pylint.exe\__main__.py", line 7, in <module>
                  File "c:\program files\python39\lib\site-packages\pylint\__init__.py", line 22, in run_pylint
                    PylintRun(sys.argv[1:])
                  File "c:\program files\python39\lib\site-packages\pylint\lint\run.py", line 358, in __init__
                    linter.check(args)
                  File "c:\program files\python39\lib\site-packages\pylint\lint\pylinter.py", line 866, in check
                    check_parallel(
                  File "c:\program files\python39\lib\site-packages\pylint\lint\parallel.py", line 138, in check_parallel
                    for (
                  File "c:\program files\python39\lib\multiprocessing\pool.py", line 870, in next
                    raise value
                multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x0000022F9BECB310>'. Reason: 'PicklingError("Can't pickle <enum 'Context'>: attribute lookup Context on astroid failed")'

and;

            C:\Temp>pylint -j1 bug.py
            ************* Module bug
            bug.py:1:0: C0114: Missing module docstring (missing-module-docstring)
            Traceback (most recent call last):
              File "c:\program files\python39\lib\runpy.py", line 197, in _run_module_as_main
                return _run_code(code, main_globals, None,
              File "c:\program files\python39\lib\runpy.py", line 87, in _run_code
                exec(code, run_globals)
              File "C:\Program Files\Python39\Scripts\pylint.exe\__main__.py", line 7, in <module>
              File "c:\program files\python39\lib\site-packages\pylint\__init__.py", line 22, in run_pylint
                PylintRun(sys.argv[1:])
              File "c:\program files\python39\lib\site-packages\pylint\lint\run.py", line 358, in __init__
                linter.check(args)
              File "c:\program files\python39\lib\site-packages\pylint\lint\pylinter.py", line 862, in check
                self._check_files(
              File "c:\program files\python39\lib\site-packages\pylint\lint\pylinter.py", line 896, in _check_files
                self._check_file(get_ast, check_astroid_module, name, filepath, modname)
              File "c:\program files\python39\lib\site-packages\pylint\lint\pylinter.py", line 922, in _check_file
                check_astroid_module(ast_node)
              File "c:\program files\python39\lib\site-packages\pylint\lint\pylinter.py", line 1054, in check_astroid_module
                retval = self._check_astroid_module(
              File "c:\program files\python39\lib\site-packages\pylint\lint\pylinter.py", line 1099, in _check_astroid_module
                walker.walk(ast_node)
              File "c:\program files\python39\lib\site-packages\pylint\utils\ast_walker.py", line 75, in walk
                self.walk(child)
              File "c:\program files\python39\lib\site-packages\pylint\utils\ast_walker.py", line 75, in walk
                self.walk(child)
              File "c:\program files\python39\lib\site-packages\pylint\utils\ast_walker.py", line 72, in walk
                callback(astroid)
              File "c:\program files\python39\lib\site-packages\pylint\checkers\refactoring\len_checker.py", line 76, in visit_call
                instance = next(len_arg.infer())
              File "c:\program files\python39\lib\site-packages\astroid\node_classes.py", line 366, in infer
                yield from self._infer(context, **kwargs)
              File "c:\program files\python39\lib\site-packages\astroid\decorators.py", line 135, in raise_if_nothing_inferred
                yield next(generator)
              File "c:\program files\python39\lib\site-packages\astroid\decorators.py", line 99, in wrapped
                res = next(generator)
              File "c:\program files\python39\lib\site-packages\astroid\inference.py", line 368, in infer_subscript
                for value in self.value.infer(context):
              File "c:\program files\python39\lib\site-packages\astroid\node_classes.py", line 380, in infer
                for i, result in enumerate(generator):
              File "c:\program files\python39\lib\site-packages\astroid\decorators.py", line 135, in raise_if_nothing_inferred
                yield next(generator)
              File "c:\program files\python39\lib\site-packages\astroid\decorators.py", line 96, in wrapped
                generator = _func(node, context, **kwargs)
              File "c:\program files\python39\lib\site-packages\astroid\inference.py", line 201, in infer_name
                raise exceptions.NameInferenceError(
            # astroid.exceptions.NameInferenceError: 'dont_care' not found in <Module.bug l.0 at 0x210bac07430>.

[Pierre-Sassoulas]

Thank you for the investigation. So if #4215 is fixed this one is too ?

[owillebo]

Unless you don't want this issue covering up other's, i.e this issue covered up #4215 <#4215> making it harder to address.

…

On Mon, Mar 8, 2021, 17:57 Pierre Sassoulas ***@***.***> wrote: Thank you for the investigation. So if #4215 <#4215> is fixed this one is too ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4165 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXQ4KQDE2YNMQ3O6ESJHWTTCT6YHANCNFSM4YMQYZCQ> .

[Pierre-Sassoulas]

OK, I think I understood, the crash was coming from another problem but this issue is about the difficulty to find the real problem and not the crash itself.

[owillebo]

Exactly. Thank you.

…

On Mon, Mar 8, 2021, 18:32 Pierre Sassoulas ***@***.***> wrote: OK, I think I understood, the crash was coming from another problem but this issue is about the difficulty to find the real problem and not the crash itself. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4165 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXQ4KWZRQEVTZZPE2A6LZ3TCUC2BANCNFSM4YMQYZCQ> .

[DanielNoord]

@Pierre-Sassoulas I think you fixed this recently, right?

[Pierre-Sassoulas]

Probably fixed in #5987, thank you @DanielNoord