Fix parsing of long lines when `missing-final-newline` is enabled #5786

DanielNoord · 2022-02-10T08:17:06Z

Add yourself to CONTRIBUTORS if you are a new contributor.
Add a ChangeLog entry describing what your PR does.
If it's a new feature, or an important bug fix, add a What's New entry in
doc/whatsnew/<current release.rst>.
Write a good description on what the PR does.

Type of Changes

	Type
✓	🐛 Bug fix

Description

Closes #5724.

The stuff at the beginning of this regex pattern isn't actually needed for us to extract our disable comments. This massively speeds up the parsing and unstucks us from the code in the issue.
I haven't added a test case as it is a bit difficult to test "don't look for things we don't need"...

DanielNoord · 2022-02-10T08:40:38Z

The first pattern does not allow:
# a comment # pylint: disable=this.
This can be fixed with the second pattern, which disallows:
# a comment # something in between pylint: disable=this.

However, do we really want to support something in between? Is there any use case for adding words in between the # and pylint?

DanielNoord · 2022-02-10T08:50:35Z

Seems like we're using the second pattern ourselves in:
# -*- pylint: disable=no-init,too-few-public-methods, useless-object-inheritance.

I think it makes sense to disallow this, but this would probably need to be deprecated. That would also mean that the performance fix can only go in 3.x.

Do we want to disallow any non-whitespace characters between the # and pylint for disable comments with the knowledge that this would boost the performance of the regex pattern that looks for these?
If yes, do we want to make this change in 3.x or in 2.x?

Pierre-Sassoulas · 2022-02-10T09:09:47Z

ChangeLog

@@ -129,6 +129,10 @@ Release date: TBA

  Closes #5569

+* Fix parsing of long lines when ``missing-final-newline`` is enabled.


Suggested change

* Fix parsing of long lines when ``missing-final-newline`` is enabled.

* Optimize parsing of long lines when ``missing-final-newline`` is enabled.

Pierre-Sassoulas · 2022-02-10T09:09:55Z

doc/whatsnew/2.13.rst

@@ -180,6 +180,10 @@ Other Changes

  Closes #5177, #5212

+* Fix parsing of long lines when ``missing-final-newline`` is enabled.


Suggested change

* Fix parsing of long lines when ``missing-final-newline`` is enabled.

* Optimize parsing of long lines when ``missing-final-newline`` is enabled.

Pierre-Sassoulas · 2022-02-10T09:26:13Z

pylint/utils/pragma_parser.py

    \#                 # Beginning of comment
-    .*?                # Anything (as little as possible)
+    \s*?               # Any whitespaces (as little as possible)


This is a breaking change, because every change is a breaking change (heavy sigh), but it's probably worth it.

We could do:

Suggested change

\s*? # Any whitespaces (as little as possible)

.*? # Anything (as little as possible)

Until we release 3.0, but then we need a way to make the deprecation known. So should we warn the user if we match a pylint comment with .*? but not with \s*? ? it would actually decrease performance in 2.X, possibly by a lot [citation required].

Another solution is to consider that it was an unintended feature and just consider this a performance fix. How many users are going to be pissed off by this decision... ? I wish I knew. We already know that the user affected by the catastrophic backtracking aren't happy 😄

I don't think there is a real performance change if we make .*? a capturing group and check if it contains anything other than spaces whenever we use this regex pattern. That should be relatively straightforward.

Wouldn't it solve the breaking change issue ?

I don't think so. The breaking change would be to remove .* completely and replace it with nothing or with \s*. To deprecate this we can use a capturing group and emit a warning when we match anything other than spaces.

But why would we be doing the breaking change if there is no performance issues if we keep the current behavior ?

skirpichev · 2022-02-11T05:20:32Z

I would expect, that the backtracking issue might be solved without incompatible changes.

DanielNoord · 2022-02-11T07:54:44Z

I would expect, that the backtracking issue might be solved without incompatible changes.

Please feel free to see if you can find a pattern that satisfies the current tests. I thought it should be quite easy as well, but stumbled upon some roadblocks..

skirpichev · 2022-02-11T12:40:21Z

This works for me:

--- pragma_parser.py.old        2022-02-11 15:35:07.864685538 +0300
+++ pragma_parser.py    2022-02-11 15:35:14.156481194 +0300
@@ -9,11 +9,8 @@
 # so that an option can be continued with the reasons
 # why it is active or disabled.
 OPTION_RGX = r"""
-    \s*                # Any number of whithespace
-    \#?                # One or zero hash
-    .*                 # Anything (as much as possible)
-    (\s*               # Beginning of first matched group and any number of whitespaces
-    \#                 # Beginning of comment
+    (?:\s*|\s*\#.*(?=\#.*?\bpylint:))
+    (\#                 # Beginning of comment
     .*?                # Anything (as little as possible)
     \bpylint:          # pylint word and column
     \s*                # Any number of whitespaces

DanielNoord · 2022-02-11T12:44:42Z

This works for me:

--- pragma_parser.py.old        2022-02-11 15:35:07.864685538 +0300
+++ pragma_parser.py    2022-02-11 15:35:14.156481194 +0300
@@ -9,11 +9,8 @@
 # so that an option can be continued with the reasons
 # why it is active or disabled.
 OPTION_RGX = r"""
-    \s*                # Any number of whithespace
-    \#?                # One or zero hash
-    .*                 # Anything (as much as possible)
-    (\s*               # Beginning of first matched group and any number of whitespaces
-    \#                 # Beginning of comment
+    (?:\s*|\s*\#.*(?=\#.*?\bpylint:))
+    (\#                 # Beginning of comment
     .*?                # Anything (as little as possible)
     \bpylint:          # pylint word and column
     \s*                # Any number of whitespaces

Have you tested this against tests/functional/l/line_too_long.py? Locally that fails with this change for me.

skirpichev · 2022-02-11T14:53:46Z

Hmm, I see. Probably, we should have another alternative to match comments, i.e. ^\s*\#.*. With that change - tests pass for me.

DanielNoord · 2022-02-11T15:06:25Z

diff --git a/pylint/utils/pragma_parser.py b/pylint/utils/pragma_parser.py
index 5ef4ef481..482e295ef 100644
--- a/pylint/utils/pragma_parser.py
+++ b/pylint/utils/pragma_parser.py
@@ -12,8 +12,8 @@ OPTION_RGX = r"""
-    \s*                # Any number of whitespace
-    \#?                # One or zero hash
-    .*                 # Anything (as much as possible)
-    (\s*               # Beginning of first matched group and any number of whitespaces
-    \#                 # Beginning of comment
+    (?:^\s*\#.*|\s*|\s*\#.*(?=\#.*?\bpylint:))
+    (\#                 # Beginning of comment
     .*?                # Anything (as little as possible)
     \bpylint:          # pylint word and column
     \s*                # Any number of whitespaces

Nice job! I think that actually works. If you want you can create a PR yourself @skirpichev, you are the one that came up with it!

skirpichev · 2022-02-11T15:27:42Z

If you want you can create a PR yourself

No, I think you did almost everything.

BTW, probably different conditions in the (?:) group require comments. And there should be a regression test, isn't? I don't know how to do this for the pylint. (I use pytest-timeout for the diofant for similar performance issues.)

Pierre-Sassoulas · 2022-02-12T19:29:13Z

(I use pytest-timeout for the diofant for similar performance issues.)

We're using.. nothing I guess. We have a global timeout for the tests in CI, and a benchmark using pytest-benchmark that we do not look at (so it's basically a time waster in our CI right now, that we hope to use later). Is pytest-timeout is something that would solve this issue ?

skirpichev · 2022-02-13T03:38:34Z

Is pytest-timeout is something that would solve this issue?

Yes, that's how I address such issues (local timeouts for tests). It's a little tricky, because you need to find out a timeout, that triggers problem on different machines (e.g. locally for you and on the Github Actions CI).

DanielNoord · 2022-02-13T09:59:21Z

We have a similar system in place, but don't use pytest-timeout. I have used it before, so I can probably come with something for this once as well!

DanielNoord · 2022-03-16T09:16:20Z

Superseded by #5925, thanks to @skirpichev 😄

…ine' (#5925) * Fix parsing of long lines when ``missing-final-newline`` is enabled * Adapt fa31b6b to be backward-compatible Fixes #5724 Also address comments from the PR #5786 Co-authored-by: Daniël van Noord <13665637+DanielNoord@users.noreply.github.com> Co-authored-by: Pierre Sassoulas <pierre.sassoulas@gmail.com>

Fix parsing of long lines when missing-final-newline is enabled

4224a04

DanielNoord added Enhancement ✨ Improvement to a component performance labels Feb 10, 2022

Update

591bf38

DanielNoord marked this pull request as draft February 10, 2022 08:50

Pierre-Sassoulas reviewed Feb 10, 2022

View reviewed changes

skirpichev mentioned this pull request Mar 16, 2022

Optimize handling of long lines for checkers like 'missing-final-newline' #5925

Merged

4 tasks

DanielNoord closed this Mar 16, 2022

DanielNoord deleted the missing-final branch March 16, 2022 09:16

Pierre-Sassoulas mentioned this pull request Apr 16, 2023

Add support for noqa: ERROR MESSAGE #2493

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix parsing of long lines when `missing-final-newline` is enabled #5786

Fix parsing of long lines when `missing-final-newline` is enabled #5786

DanielNoord commented Feb 10, 2022

DanielNoord commented Feb 10, 2022

DanielNoord commented Feb 10, 2022

Pierre-Sassoulas Feb 10, 2022

Pierre-Sassoulas Feb 10, 2022

Pierre-Sassoulas Feb 10, 2022 •

edited

Loading

DanielNoord Feb 10, 2022

Pierre-Sassoulas Feb 11, 2022

DanielNoord Feb 11, 2022

Pierre-Sassoulas Feb 12, 2022

skirpichev commented Feb 11, 2022

DanielNoord commented Feb 11, 2022

skirpichev commented Feb 11, 2022

DanielNoord commented Feb 11, 2022

skirpichev commented Feb 11, 2022

DanielNoord commented Feb 11, 2022 •

edited

Loading

skirpichev commented Feb 11, 2022

Pierre-Sassoulas commented Feb 12, 2022

skirpichev commented Feb 13, 2022

DanielNoord commented Feb 13, 2022

DanielNoord commented Mar 16, 2022

		@@ -129,6 +129,10 @@ Release date: TBA

		Closes #5569

		* Fix parsing of long lines when ``missing-final-newline`` is enabled.

	* Fix parsing of long lines when ``missing-final-newline`` is enabled.
	* Optimize parsing of long lines when ``missing-final-newline`` is enabled.

		@@ -180,6 +180,10 @@ Other Changes

		Closes #5177, #5212

		* Fix parsing of long lines when ``missing-final-newline`` is enabled.

	\s*? # Any whitespaces (as little as possible)
	.*? # Anything (as little as possible)

Fix parsing of long lines when missing-final-newline is enabled #5786

Fix parsing of long lines when missing-final-newline is enabled #5786

Conversation

DanielNoord commented Feb 10, 2022

Type of Changes

Description

DanielNoord commented Feb 10, 2022

DanielNoord commented Feb 10, 2022

Pierre-Sassoulas Feb 10, 2022

Choose a reason for hiding this comment

Pierre-Sassoulas Feb 10, 2022

Choose a reason for hiding this comment

Pierre-Sassoulas Feb 10, 2022 • edited Loading

Choose a reason for hiding this comment

DanielNoord Feb 10, 2022

Choose a reason for hiding this comment

Pierre-Sassoulas Feb 11, 2022

Choose a reason for hiding this comment

DanielNoord Feb 11, 2022

Choose a reason for hiding this comment

Pierre-Sassoulas Feb 12, 2022

Choose a reason for hiding this comment

skirpichev commented Feb 11, 2022

DanielNoord commented Feb 11, 2022

skirpichev commented Feb 11, 2022

DanielNoord commented Feb 11, 2022

skirpichev commented Feb 11, 2022

DanielNoord commented Feb 11, 2022 • edited Loading

skirpichev commented Feb 11, 2022

Pierre-Sassoulas commented Feb 12, 2022

skirpichev commented Feb 13, 2022

DanielNoord commented Feb 13, 2022

DanielNoord commented Mar 16, 2022

Fix parsing of long lines when `missing-final-newline` is enabled #5786

Fix parsing of long lines when `missing-final-newline` is enabled #5786

Pierre-Sassoulas Feb 10, 2022 •

edited

Loading

DanielNoord commented Feb 11, 2022 •

edited

Loading