Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add future argument to all NodeNG.statement() calls #1235

Merged
merged 12 commits into from
Nov 24, 2021
Merged
2 changes: 1 addition & 1 deletion astroid/brain/brain_dataclasses.py
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@ def _looks_like_dataclass_field_call(node: Call, check_scope: bool = True) -> bo
If check_scope is False, skips checking the statement and body.
"""
if check_scope:
stmt = node.statement()
stmt = node.statement(future=True)
scope = stmt.scope()
if not (
isinstance(stmt, AnnAssign)
Expand Down
2 changes: 1 addition & 1 deletion astroid/brain/brain_namedtuple_enum.py
Original file line number Diff line number Diff line change
Expand Up @@ -365,7 +365,7 @@ def infer_enum_class(node):
if any(not isinstance(value, nodes.AssignName) for value in values):
continue

stmt = values[0].statement()
stmt = values[0].statement(future=True)
if isinstance(stmt, nodes.Assign):
if isinstance(stmt.targets[0], nodes.Tuple):
targets = stmt.targets[0].itered()
Expand Down
4 changes: 2 additions & 2 deletions astroid/mixins.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ class FilterStmtsMixin:

def _get_filtered_stmts(self, _, node, _stmts, mystmt):
"""method used in _filter_stmts to get statements and trigger break"""
if self.statement() is mystmt:
if self.statement(future=True) is mystmt:
# original node's statement is the assignment, only keep
# current node (gen exp, list comp)
return [node], True
Expand All @@ -64,7 +64,7 @@ def _get_filtered_stmts(self, lookup_node, node, _stmts, mystmt):
"""method used in filter_stmts"""
if self is mystmt:
return _stmts, True
if self.statement() is mystmt:
if self.statement(future=True) is mystmt:
# original node's statement is the assignment, only keep
# current node (gen exp, list comp)
return [node], True
Expand Down
19 changes: 14 additions & 5 deletions astroid/nodes/node_classes.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
InferenceError,
NoDefault,
ParentMissingError,
StatementMissing,
)
from astroid.manager import AstroidManager
from astroid.nodes.const import OP_PRECEDENCE
Expand Down Expand Up @@ -387,7 +388,7 @@ def ilookup(self, name):
return _infer_stmts(stmts, context, frame)

def _get_filtered_node_statements(self, nodes):
statements = [(node, node.statement()) for node in nodes]
statements = [(node, node.statement(future=True)) for node in nodes]
# Next we check if we have ExceptHandlers that are parent
# of the underlying variable, in which case the last one survives
if len(statements) > 1 and all(
Expand Down Expand Up @@ -437,10 +438,18 @@ def _filter_stmts(self, stmts, frame, offset):
#
# def test(b=1):
# ...

if self.statement() is myframe and myframe.parent:
if (
self.parent
and self.statement(future=True) is myframe
and myframe.parent
):
myframe = myframe.parent.frame()
mystmt = self.statement()

# nodes.Module don't have a parent attribute
try:
mystmt = self.statement(future=True)
except StatementMissing:
mystmt = self
Copy link
Collaborator Author

@DanielNoord DanielNoord Nov 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cdce8p These are the lines I referred to. Both calls to self.statement() will raise a StatementMissing exception when called on a nodes.Module. By adding a check for self.parent, which nodes.Module don't have, and a try ... except we can catch this.
However, similar to the actual change in #1217 I wonder if mystmt should actually become a nodes.Module in the first place (as it would still be with this change). You probably have a better idea if this is the way to go about fixing this.

Note that nodes.Module can't be imported due to circular imports so we can't do isinstance() here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had some time to think about it. Did I make made a mistake by suggesting to raise an error if statement() is called on Module?

Some points to consider

  • statement() is often used together with frame() and thus doesn't quite represent an actual ast.Statement. It's rather used to compare if the statement() of a node is also equal to the frame(). Thus returning Module made sense in that context.
  • During a normal pylint run you didn't encounter an AttributeError as a parent had always been defined or it was a Module which returned itself. To update pylint, we would need to add exception handling in at least some / most (?) cases.
  • Just from looking at the changes here, I'm not sure it's an actual improvement.

@DanielNoord What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DanielNoord Sorry to ping you again so soon. If we follow though and revert the change, I think if would be good to do so before pylint 2.12.0 is released. Reverting it would also address #1239.

/CC: @Pierre-Sassoulas

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we need to revert that we might as well do it quick as the deprecation warning exists in 2.8.5 that has been released, it's better if it's reverted fast. I did not follow this close enough to have an opinion about it, so I trust your decision on this. Re-releasing astroid with a revert is cheap.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend doing that. Would like to know what @DanielNoord thinks first as he also spend considerable time with the original change. One or two days more doesn't really matter too much, just the issue with the 2.12 release.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't gotten the time to look at this sufficiently. Should be able to do so tomorrow.

My initial feeling is that we should probably try and find a way to allow the precious behaviour but not in statement. statement returning Modules is completely unexpected based on the methods name and that confusion is what caused this in the first place. I think it is valuable to avoid such unlogical behaviour.

I haven't looked into why we need to return modules exactly in the first place, @cdce8p already mentioned it but haven't fully read the initial comment. But if this is indeed a valid use case, perhaps we need a module_and_statement()?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do agree. The name statement isn't ideal.
However as for changing it, I don't think it's worth it. We don't gain anything and just break code in the process. If there were a clear advantage, maybe. But even then there is a point to be made that the function should stay the same and we should instead add a new one that only returns actual Statement nodes. That would be backwards compatible.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this PR my response is WIP. But;

I have looked at _filter_stmts, which is the only function causing problems in astroid with future=True.
https://github.com/PyCQA/astroid/blob/30711bc012592844a902d1d5912b3b2b6f2dce0f/astroid/nodes/node_classes.py#L401

mystmt is used on some occasions, which I will discuss to show what happens when mystmt is in fact a nodes.Module.
https://github.com/PyCQA/astroid/blob/30711bc012592844a902d1d5912b3b2b6f2dce0f/astroid/nodes/node_classes.py#L448

nodes.Module.fromlineno is always 0. Therefore, this check will never be True.

https://github.com/PyCQA/astroid/blob/30711bc012592844a902d1d5912b3b2b6f2dce0f/astroid/nodes/node_classes.py#L465

stmt will never be a nodes.Module since _filter_stmts is only called with self.locals[name]. This will never include a nodes.Module so this check will never be True.

https://github.com/PyCQA/astroid/blob/30711bc012592844a902d1d5912b3b2b6f2dce0f/astroid/nodes/node_classes.py#L476

_get_filtered_stmts is not defined on nodes.Module. All three definitions of _get_filtered_stmts rely on comparing the self of _get_filtered_stmts to mystmt or calling self.statement() and comparing to mystmt. Since self is never nodes.Module this will never return True and done will never be True.

https://github.com/PyCQA/astroid/blob/30711bc012592844a902d1d5912b3b2b6f2dce0f/astroid/nodes/node_classes.py#L569

Since nodes.Module doesn't have a parent this will never be True.

So, to me it seems that the return of self for nodes.Module is done to avoid the AttributeError but is not meaningful in any way. Initialising mystmt as None and only reassigning if self.parent as I do in the commit I just added seems fine for astroid.

I tested these changes locally with pylint and all tests pass.


Then there is the issue of whether the behaviour of future is compatible with pylint. I am going to investigate this now and see whether we really need statement() to return nodes.Module. I agree that adding try ... except everywhere would not be good, but seeing as the change to astroid is relatively simple I think we might get away with it.

Copy link
Collaborator Author

@DanielNoord DanielNoord Nov 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think pylint-dev/pylint#5310 shows that the effect of no longer returning nodes.Module for pylint is minimal as well.
That PR adds one isinstance(nodes.Module) (related to nodes.Module.fromlineno), one if XXX.parent and changes one except AttributeError to except StatementMissing. I tested that PR against the latest commit of astroid in this branch and all tests passed.


Based on this I think we do not need to revert the earlier change.

We never need statement() to return nodes.Module, both in pylint and in astroid. There is no if statement in any of the projects that evaluates to True whenever the method does return a nodes.Module. I think the return self was added to avoid creating crashes when AttributeError was raised or might have served a purpose it no longer does.

# line filtering if we are in the same frame
#
# take care node may be missing lineno information (this is the case for
Expand Down Expand Up @@ -1816,7 +1825,7 @@ def _get_filtered_stmts(self, lookup_node, node, stmts, mystmt):
if isinstance(lookup_node, (Const, Name)):
return [lookup_node], True

elif self.statement() is mystmt:
elif self.statement(future=True) is mystmt:
# original node's statement is the assignment, only keeps
# current node (gen exp, list comp)

Expand Down
2 changes: 1 addition & 1 deletion astroid/nodes/scoped_nodes.py
Original file line number Diff line number Diff line change
Expand Up @@ -2587,7 +2587,7 @@ def getattr(self, name, context=None, class_context=True):
# Look for AnnAssigns, which are not attributes in the purest sense.
for value in values:
if isinstance(value, node_classes.AssignName):
stmt = value.statement()
stmt = value.statement(future=True)
if isinstance(stmt, node_classes.AnnAssign) and stmt.value is None:
raise AttributeInferenceError(
target=self, attribute=name, context=context
Expand Down
2 changes: 1 addition & 1 deletion astroid/protocols.py
Original file line number Diff line number Diff line change
Expand Up @@ -637,7 +637,7 @@ def _determine_starred_iteration_lookups(starred, target, lookups):
lookups.append((index, len(element.itered())))
_determine_starred_iteration_lookups(starred, element, lookups)

stmt = self.statement()
stmt = self.statement(future=True)
if not isinstance(stmt, (nodes.Assign, nodes.For)):
raise InferenceError(
"Statement {stmt!r} enclosing {node!r} " "must be an Assign or For node.",
Expand Down
4 changes: 3 additions & 1 deletion tests/unittest_builder.py
Original file line number Diff line number Diff line change
Expand Up @@ -614,7 +614,9 @@ def test_module_base_props(self) -> None:
self.assertEqual(module.pure_python, 1)
self.assertEqual(module.package, 0)
self.assertFalse(module.is_statement)
self.assertEqual(module.statement(), module)
with pytest.warns(DeprecationWarning) as records:
self.assertEqual(module.statement(), module)
assert len(records) == 1
with pytest.warns(DeprecationWarning) as records:
module.statement()
assert len(records) == 1
DanielNoord marked this conversation as resolved.
Show resolved Hide resolved
Expand Down