Bug pylint 2721 #664

hippo91 · 2019-04-21T09:01:07Z

Steps

For new features or bug fixes, add a ChangeLog entry describing what your PR does.
Write a good description on what the PR does.

Description

This PR is deep rework of the numpy brain.
The original brain has been split into numerous one. Each numpy brain module corresponds to
a numpyone. There are:

brain_numpy_core_fromnumeric => numpy.core.fromnumeric
brain_numpy_core_function_base => numpy.core.function_base
brain_numpy_core_multiarray => numpy.core.multiarray
brain_numpy_core_numeric => numpy.core.numeric
brain_core_numerictypes => numpy.core.numerictypes
brain_core_umath => numpy.core.umath
brain_numpy_random_mtrand => numpy.random.mtrand

Last but not least, the numpy.ndarray class has its corresponding astroid's brain (brain_numpy_ndarray).

Each astroid's brain has its own associated unit test.

The major improvement, besides a clearer organisation, is the fact that each method of the ndarray class should now be correctly inferred. Especially those returning ndarray objects.

It has been made possible by ensuring that the return type of those methods is inferred as ndarray and nothing else. To achieve this, the inference_tip function has been used.

When a numpy callable may return two different kinds of objects, for example bool or ndarray, the choice to prefer ndarray has been made in the corresponding astroid brain method.

There is still work to do, especially correcting the return types of the brain_numpy_random_mtrand and brain_numpy_core_numerictypes modules. The last one should also be reworked to correct method signature of the generic class.
But this PR should strongly improve the numpy support for pylint.

Type of Changes

	Type
✓	🐛 Bug fix
✓	✨ New feature
✓	🔨 Refactoring

Related Issue

Closes pylint-dev/pylint#2747
Closes pylint-dev/pylint#2721
Closes pylint-dev/pylint#2326
Closes pylint-dev/pylint#2021

Related to pylint-dev/pylint#2767
Related to pylint-dev/pylint#2694
Related to pylint-dev/pylint#2784
Related to pylint-dev/pylint#2759

… zeros_like function (pass) and filter tuple for type inferred for a call to zeros_like

…rom types inferred by call to full_like and ones_like

…space functions

…arrays

…ay from numpy_core_numerictypes_transform function to infer_numpy_ndarray which is used a inference_tip. Add a _looks_like_numpy_ndarray function. Deletion of the register_transform for linspace function because it is correctly inferred with the inference_tip function for ndarray

…ndarray' and not 'numpy.core.numerictypes.ndarray'

…ure of setitem method. Add of empty_like, ones_like function

…ey are redondant with the one test only ndarray are inferred

…s for ndarray

…py_* module has its own unittest module

hippo91 · 2019-04-26T14:02:50Z

This PR should also prevent infinite recursion for bug pylint-dev/pylint#2865

wshanks · 2019-04-26T17:21:02Z

For this file:

import numpy as np
a = np.append([0], 0)
a = -a

I still see:

/tmp/test.py:3:4: E1130: bad operand type for unary -: tuple (invalid-unary-operand-type)
/tmp/test.py:3:4: E1130: bad operand type for unary -: list (invalid-unary-operand-type)

Should that have been fixed by this PR?

hippo91 · 2019-04-29T11:24:52Z

@willsALMANJ sorry for the delay.
The append function is not taken into account through astroid numpy brains.
I don't understand yet why this problem appears. I'll try to have a look ASAP.
Can you, please, open a specific issue for that?
Thanks again for your feedback.

PCManticore

👏 Wow, this is tremendous, thank you so much @hippo91 ! Overall it looks great, but left a couple of comments to be addressed before shipping this, the major one being the use of inference in transform filters. Just looking forward to have a bunch of numpy issues closed at once in pylint's tracker! Thank you again for the great work! 🎉

PCManticore · 2019-05-05T10:18:55Z

.gitignore

@@ -13,3 +13,6 @@ astroid.egg-info/
 .cache/
 .eggs/
 .pytest_cache/
+.python-version


@hippo91 These should go into your global gitignore, as they are specific to tools you use on your dev environment rather than specific to tools used by the project itself.

Yes ! Off-course.

PCManticore · 2019-05-05T10:19:11Z

ChangeLog

+    - ``numpy.core.umath`` 
+    - ``numpy.random.mtrand``
+
+  Close PyCQA/pylint2865


That's a nice set of closed issues!

PCManticore · 2019-05-05T10:23:18Z

astroid/brain/brain_numpy_utils.py

+    return (
+        isinstance(node, astroid.Attribute)
+        and node.attrname == member_name
+        and node.expr.inferred()[0].name.startswith("numpy")


@hippo91 Unfortunately using inference in the transform filters could have certain recursion error side effects, given that brain is not fully ready by the time the transform filters are called. Is there any way to prevent having this inference call here?

@PCManticore , i changed the way the attribute is meant to be a numpy member.
I also introduced docstrings and type annotations.

PCManticore · 2019-05-05T10:24:31Z

astroid/tests/unittest_brain_numpy_core_fromnumeric.py

+
+    def subTest(self, msg=None, **params):
+        try:
+            # For python versions above 3.5 this should be ok


❓ Isn't subTest available from 3.4? If so, we could use it today without the need of subtestMock.

Well done @PCManticore . You are totally right this mock is useless now.
I deleted it.

PCManticore · 2019-05-05T10:25:08Z

astroid/tests/unittest_brain_numpy_core_function_base.py

+from astroid import builder
+
+
+class SubTestWrapper(unittest.TestCase):


Even if we can't subTest right now (check my previous question on this topic), we should move these definitions into a common test file and use them from there.

All occurrences of this mock have been deleted.

…into bug_pylint_2721

hippo91 · 2019-05-25T20:13:00Z

@PCManticore ,

Thanks for you review and kind words.
I made the changes you suggested.
The Travis CI failed due to the pylint step and i don't really understand why.
I used pylint with an old version of pylint's master branch (approximately 2 months old) and everything went fine. However after updating to last version i got the same problem as in Travis.
Have you any clue about it?

PCManticore · 2019-06-01T07:30:13Z

Hi @hippo91 This should work now if you try again, there was a bug in pylint's master.

hippo91 · 2019-06-01T09:16:59Z

Thanks @PCManticore !
It seems that i'v got a problem with python3.8. I will investigate this ASAP.

PCManticore · 2019-06-01T09:41:26Z

@hippo91 Seems to me that the Python 3.8 tests started to fail for both pylint and astroid. We should fix it separate of this PR though.

PCManticore · 2019-06-02T08:56:51Z

Thank you again @hippo91 for all the hard work! 👏

hippo91 · 2019-06-02T09:04:24Z

You are welcome @PCManticore. Thanks for merging it.

hippo91 added 30 commits February 26, 2019 15:08

Goes back to pass instead of return any for randint function. Declare…

b0ad5a1

… zeros_like function (pass) and filter tuple for type inferred for a call to zeros_like

Removing declaration of zeros_like (pass). Filtering tuple and list f…

919f938

…rom types inferred by call to full_like and ones_like

Add transforms to define return type of numpy.zeros_like et numpy.lin…

0e142fe

…space functions

Add a test to ensure that numpy functions calls are infered as numpy …

0f7d210

…arrays

Merge branch 'master' into bug_pylint_2721

9714e3b

Merge branch 'master' into bug_pylint_2721

35baf16

Correcting inference test. linspace and zeros_like are inferred as '.…

83aa30f

…ndarray' and not 'numpy.core.numerictypes.ndarray'

Merge branch 'master' into bug_pylint_2721

3e7aef9

Add of __getitem__ and __setitem__ methods in ndarray. Correct signat…

d7c0373

…ure of setitem method. Add of empty_like, ones_like function

Split the brain_numpy modules in smaller ones for clarity.

d6f1fe1

Test the unicity of inferred value for function returning ndarray.

81b70ed

Remove test_numpy_function_calls_not_inferred_as_* methods because th…

c5687bc

…ey are redondant with the one test only ndarray are inferred

Finish the split of the numpy brain. Implements all the dunder method…

c5d9205

…s for ndarray

Merge branch 'master' into bug_pylint_2721

b7e42aa

Finalizing description of numpy.ndarray

19f9f27

Correctiong missing default value

86ffb90

Deleting test about __inv__ method which doesn't exists in ndarray

be16e43

Split the unittest_brain_numpy module in parts so that each brain_num…

181c83e

…py_* module has its own unittest module

Changes uninferable return toward correct ones

6be0cda

Split the unittest_brain_numpy

ecb4fbd

Check that functions return a ndarray

0e0edeb

Split the unittest_brain_numpy module

2aba1f9

Reformat

2308d93

Add logspace et geomspace functions

27a9d33

Typo.

ee0db7f

Add a unittest dedicated to the numpy_core_fromnumeric brain

8718254

Merge branch 'master' into bug_pylint_2721

5869ac5

Correct prototypes of 3 dunder methods.

0401d81

Refactor in order to test the methods of ndarray

cee9012

hippo91 added 3 commits April 20, 2019 19:34

Factorize functions inference system

585e118

Refactor to make the introduction of new functions easier

7d5ae0d

Refomat according to black

4ab338e

wshanks mentioned this pull request Apr 25, 2019

pylint no longer detects numpy ufunc methods pylint-dev/pylint#2885

Closed

hippo91 added 2 commits April 26, 2019 15:42

Corrects the way a numpy member is identified

ee3d323

Add the references to pylint bugs corrected

4ad2fd1

wshanks mentioned this pull request Apr 30, 2019

Add numpy brain tips for numpy.append #666

Closed

PCManticore reviewed May 5, 2019

View reviewed changes

hippo91 added 5 commits May 25, 2019 09:45

Merge branch 'master' into bug_pylint_2721

5646dd8

Merge branch 'bug_pylint_2721' of https://github.com/hippo91/astroid …

0a81846

…into bug_pylint_2721

Deleting personal setup

442dcae

Avoid use of inference engine before it is fully set up. Add docstrings

c2bb4f6

Deleting useless subTest mock for python version prior to 3.4

f8d12d2

Merge branch 'master' into bug_pylint_2721

40a7ee9

Merge branch 'master' into bug_pylint_2721

f818d1a

PCManticore merged commit 33065ec into pylint-dev:master Jun 2, 2019

SimonBiggs mentioned this pull request Jun 2, 2019

Can remove numpy pylint ignores soon! pymedphys/pymedphys#371

Closed

hippo91 mentioned this pull request Jun 12, 2019

False positive invalid-unary-operand-type for return value of np.diff() pylint-dev/pylint#2954

Closed

This was referenced Aug 31, 2019

Maximum recursion depth exceed numpy, chained call pylint-dev/pylint#2865

Closed

False positive: "Does not support item assignment" error with np.zeros_like pylint-dev/pylint#2747

Closed

hippo91 deleted the bug_pylint_2721 branch June 20, 2020 09:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug pylint 2721 #664

Bug pylint 2721 #664

hippo91 commented Apr 21, 2019

hippo91 commented Apr 26, 2019

wshanks commented Apr 26, 2019 •

edited

Loading

hippo91 commented Apr 29, 2019

PCManticore left a comment

PCManticore May 5, 2019

hippo91 May 25, 2019

PCManticore May 5, 2019

PCManticore May 5, 2019

hippo91 May 25, 2019

PCManticore May 5, 2019

hippo91 May 25, 2019

PCManticore May 5, 2019

hippo91 May 25, 2019

hippo91 commented May 25, 2019

PCManticore commented Jun 1, 2019

hippo91 commented Jun 1, 2019

PCManticore commented Jun 1, 2019

PCManticore commented Jun 2, 2019

hippo91 commented Jun 2, 2019

		from astroid import builder


		class SubTestWrapper(unittest.TestCase):

Bug pylint 2721 #664

Bug pylint 2721 #664

Conversation

hippo91 commented Apr 21, 2019

Steps

Description

Type of Changes

Related Issue

hippo91 commented Apr 26, 2019

wshanks commented Apr 26, 2019 • edited Loading

hippo91 commented Apr 29, 2019

PCManticore left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hippo91 commented May 25, 2019

PCManticore commented Jun 1, 2019

hippo91 commented Jun 1, 2019

PCManticore commented Jun 1, 2019

PCManticore commented Jun 2, 2019

hippo91 commented Jun 2, 2019

wshanks commented Apr 26, 2019 •

edited

Loading