Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pylint to CI and fix all Python #3052

Merged
merged 1 commit into from
May 11, 2023

Conversation

declension
Copy link
Contributor

@declension declension commented Apr 19, 2023

Short description of changes

  • Add and run pylint in CI
  • Split other linters into own jobs, was getting a bit all-or-nothing in that job
  • Add a sane Pylint config tweaked from default, but less onerous
  • Define the least-delta max-line-length that PEP-008 allows, 99 chars, and reflow a few lines. Looks nice anyway IMO
  • Fix all the various errors, plus a few typos etc too
  • Rename one module (script) that wasn't python-friendly (snake_case) naming
  • Initialise some object attributes in the constructor, not randomly elsewhere (soft warning in IDE)

ℹ️ This actually found a few bugs / gotchas I think (woohoo it was worth it) - so it'd be good if someone could sanity check those changes here, especially:

  • In get_release_contributors realname was never defined so that code path would throw exceptions
  • Some of the regex char classes were actually control characters (missing the raw string prefix), so fixed.

CHANGELOG: Autobuild: Modernise / fix some Python and add linting to CI

Context: Fixes an issue?
Fixes: #2546

Does this change need documentation? What needs to be documented and how?

Probably not?

Status of this Pull Request

What is missing until this pull request can be merged?

Checklist

  • I've verified that this Pull Request follows the general code principles
  • I tested my code and it does what I want
  • My code follows the style guide
  • I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
  • I've filled all the content above

Linting results on current main

As a reference, these are the issues that currently come up on main (at 20148a7)

❯ find tools/ -name '*.py' -print -exec pylint {} +
tools/generate_json_rpc_docs.py
tools/get_release_contributors.py
tools/qt5-to-qt6-country-code-table.py
************* Module generate_json_rpc_docs
tools/generate_json_rpc_docs.py:60:12: W0719: Raising too general exception: Exception (broad-exception-raised)
tools/generate_json_rpc_docs.py:52:12: W0201: Attribute 'current_tag' defined outside __init__ (attribute-defined-outside-init)
tools/generate_json_rpc_docs.py:54:12: W0201: Attribute 'current_tag' defined outside __init__ (attribute-defined-outside-init)
tools/generate_json_rpc_docs.py:57:12: W0201: Attribute 'current_tag' defined outside __init__ (attribute-defined-outside-init)
tools/generate_json_rpc_docs.py:121:0: R0903: Too few public methods (1/2) (too-few-public-methods)
tools/generate_json_rpc_docs.py:153:0: C0103: Constant name "current_item" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/generate_json_rpc_docs.py:157:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
tools/generate_json_rpc_docs.py:184:16: C0103: Constant name "current_item" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/generate_json_rpc_docs.py:188:0: C0103: Constant name "preamble" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/generate_json_rpc_docs.py:253:5: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
************* Module get_release_contributors
tools/get_release_contributors.py:12:0: C0301: Line too long (154/100) (line-too-long)
tools/get_release_contributors.py:38:0: C0301: Line too long (133/100) (line-too-long)
tools/get_release_contributors.py:196:0: C0301: Line too long (108/100) (line-too-long)
tools/get_release_contributors.py:211:0: C0301: Line too long (160/100) (line-too-long)
tools/get_release_contributors.py:227:0: C0301: Line too long (164/100) (line-too-long)
tools/get_release_contributors.py:248:0: C0301: Line too long (117/100) (line-too-long)
tools/get_release_contributors.py:250:0: C0301: Line too long (119/100) (line-too-long)
tools/get_release_contributors.py:254:0: C0301: Line too long (143/100) (line-too-long)
tools/get_release_contributors.py:92:22: W1401: Anomalous backslash in string: '\A'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
tools/get_release_contributors.py:92:38: W1401: Anomalous backslash in string: '\Z'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
tools/get_release_contributors.py:219:48: W1401: Anomalous backslash in string: '\s'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
tools/get_release_contributors.py:219:52: W1401: Anomalous backslash in string: '\S'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
tools/get_release_contributors.py:219:67: W1401: Anomalous backslash in string: '\s'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
tools/get_release_contributors.py:34:4: W0107: Unnecessary pass statement (unnecessary-pass)
tools/get_release_contributors.py:59:4: C0116: Missing function or method docstring (missing-function-docstring)
tools/get_release_contributors.py:62:4: C0116: Missing function or method docstring (missing-function-docstring)
tools/get_release_contributors.py:92:8: C0103: Variable name "m" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:95:8: W1203: Use lazy % formatting in logging functions (logging-fstring-interpolation)
tools/get_release_contributors.py:98:33: W0622: Redefining built-in 'hash' (redefined-builtin)
tools/get_release_contributors.py:103:8: C0103: Variable name "r" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:113:37: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
tools/get_release_contributors.py:115:0: W0621: Redefining name 'args' from outer scope (line 259) (redefined-outer-name)
tools/get_release_contributors.py:120:39: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
tools/get_release_contributors.py:121:8: C0103: Variable name "r" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:121:12: W3101: Missing timeout argument for method 'requests.get' can cause your program to hang indefinitely (missing-timeout)
tools/get_release_contributors.py:124:4: C0116: Missing function or method docstring (missing-function-docstring)
tools/get_release_contributors.py:125:8: C0103: Variable name "m" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:130:8: C0103: Variable name "m" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:133:8: C0103: Variable name "r" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:135:12: W1203: Use lazy % formatting in logging functions (logging-fstring-interpolation)
tools/get_release_contributors.py:140:12: C0103: Variable name "u" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:144:8: W1203: Use lazy % formatting in logging functions (logging-fstring-interpolation)
tools/get_release_contributors.py:151:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
tools/get_release_contributors.py:151:37: C0103: Variable name "f" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:158:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
tools/get_release_contributors.py:158:37: C0103: Variable name "f" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:60:8: W0201: Attribute 'repo' defined outside __init__ (attribute-defined-outside-init)
tools/get_release_contributors.py:63:8: W0201: Attribute 'token' defined outside __init__ (attribute-defined-outside-init)
tools/get_release_contributors.py:163:0: C0103: Constant name "authors_cache" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/get_release_contributors.py:169:0: C0116: Missing function or method docstring (missing-function-docstring)
tools/get_release_contributors.py:169:16: C0103: Argument name "to" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:183:0: C0116: Missing function or method docstring (missing-function-docstring)
tools/get_release_contributors.py:183:35: C0103: Argument name "to" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:187:0: C0116: Missing function or method docstring (missing-function-docstring)
tools/get_release_contributors.py:187:33: C0103: Argument name "to" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:191:0: C0116: Missing function or method docstring (missing-function-docstring)
tools/get_release_contributors.py:191:38: C0103: Argument name "to" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:195:0: C0116: Missing function or method docstring (missing-function-docstring)
tools/get_release_contributors.py:195:55: C0103: Argument name "to" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:198:10: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
tools/get_release_contributors.py:201:47: C0103: Argument name "to" doesn't conform to snake_case naming style (invalid-name)
tools/get_release_contributors.py:216:8: W0622: Redefining built-in 'hash' (redefined-builtin)
tools/get_release_contributors.py:211:111: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
tools/get_release_contributors.py:221:12: W1203: Use lazy % formatting in logging functions (logging-fstring-interpolation)
tools/get_release_contributors.py:227:96: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
tools/get_release_contributors.py:240:29: E0602: Undefined variable 'realname' (undefined-variable)
tools/get_release_contributors.py:263:8: C0103: Constant name "level" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/get_release_contributors.py:265:8: C0103: Constant name "level" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/get_release_contributors.py:267:8: C0103: Constant name "level" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/get_release_contributors.py:20:0: C0411: standard import "import argparse" should be placed before "import requests" (wrong-import-order)
tools/get_release_contributors.py:21:0: C0411: standard import "import logging" should be placed before "import requests" (wrong-import-order)
tools/get_release_contributors.py:22:0: C0411: standard import "import os" should be placed before "import requests" (wrong-import-order)
tools/get_release_contributors.py:23:0: C0411: standard import "import re" should be placed before "import requests" (wrong-import-order)
tools/get_release_contributors.py:24:0: C0411: standard import "import subprocess" should be placed before "import requests" (wrong-import-order)
************* Module qt5-to-qt6-country-code-table
tools/qt5-to-qt6-country-code-table.py:43:0: C0325: Unnecessary parens after 'raise' keyword (superfluous-parens)
tools/qt5-to-qt6-country-code-table.py:1:0: C0103: Module name "qt5-to-qt6-country-code-table" doesn't conform to snake_case naming style (invalid-name)
tools/qt5-to-qt6-country-code-table.py:13:0: C0103: Constant name "qt5_include" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/qt5-to-qt6-country-code-table.py:14:0: C0103: Constant name "qt6_include" doesn't conform to UPPER_CASE naming style (invalid-name)
tools/qt5-to-qt6-country-code-table.py:17:0: C0116: Missing function or method docstring (missing-function-docstring)
tools/qt5-to-qt6-country-code-table.py:32:8: W0621: Redefining name 'name' from outer scope (line 64) (redefined-outer-name)
tools/qt5-to-qt6-country-code-table.py:20:9: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
tools/qt5-to-qt6-country-code-table.py:20:25: C0103: Variable name "f" doesn't conform to snake_case naming style (invalid-name)
tools/qt5-to-qt6-country-code-table.py:21:8: C0103: Variable name "s" doesn't conform to snake_case naming style (invalid-name)
tools/qt5-to-qt6-country-code-table.py:31:12: W0719: Raising too general exception: Exception (broad-exception-raised)
tools/qt5-to-qt6-country-code-table.py:31:28: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
tools/qt5-to-qt6-country-code-table.py:43:12: E0702: Raising str while only classes or instances are allowed (raising-bad-type)
tools/qt5-to-qt6-country-code-table.py:47:0: C0116: Missing function or method docstring (missing-function-docstring)
tools/qt5-to-qt6-country-code-table.py:47:16: W0621: Redefining name 'name' from outer scope (line 64) (redefined-outer-name)
tools/qt5-to-qt6-country-code-table.py:49:18: R1728: Consider using a generator instead 'max(int(x) for x in table.keys())' (consider-using-generator)
tools/qt5-to-qt6-country-code-table.py:50:11: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
tools/qt5-to-qt6-country-code-table.py:52:15: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)

------------------------------------------------------------------
Your code has been rated at 6.76/10 (previous run: 6.76/10, +0.00)

@ann0see
Copy link
Member

ann0see commented Apr 20, 2023

Thanks for that!

Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a Python expert, but If I find time, I'll try to review 😀

.editorconfig Outdated Show resolved Hide resolved
.pylintrc Outdated Show resolved Hide resolved
tools/generate_json_rpc_docs.py Outdated Show resolved Hide resolved
tools/get_release_contributors.py Show resolved Hide resolved
tools/get_release_contributors.py Show resolved Hide resolved
tools/get_release_contributors.py Show resolved Hide resolved
tools/qt5_to_qt6_country_code_table.py Outdated Show resolved Hide resolved
tools/qt5_to_qt6_country_code_table.py Show resolved Hide resolved
@ann0see
Copy link
Member

ann0see commented Apr 20, 2023

I'm not sure how much time I can spend in the next days on the review but I'll try to fit it in some time. In the meantime you could maybe look if the CONTRIBUTING.md file needs changes. The maintainers should also decide where to add your name - if you want to be displayed as contributor.

Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I think this is the first review round from my side. A few (rather quite a lot ;-) minor comments).
I think in general it's good - but we need some testing before it can be merged.

.github/autobuild/get_build_vars.py Outdated Show resolved Hide resolved
.github/workflows/coding-style-check.yml Outdated Show resolved Hide resolved
.github/workflows/coding-style-check.yml Show resolved Hide resolved
.github/workflows/coding-style-check.yml Outdated Show resolved Hide resolved
.pylintrc Show resolved Hide resolved
tools/get_release_contributors.py Outdated Show resolved Hide resolved
tools/get_release_contributors.py Outdated Show resolved Hide resolved
tools/get_release_contributors.py Outdated Show resolved Hide resolved
tools/qt5_to_qt6_country_code_table.py Show resolved Hide resolved
tools/qt5_to_qt6_country_code_table.py Show resolved Hide resolved
@ann0see ann0see self-requested a review April 24, 2023 20:38
@declension
Copy link
Contributor Author

I think that's everything I can see @ann0see / @pljones .

Should I change txvContributors in util.cpp then or not - wasn't sure...

@ann0see
Copy link
Member

ann0see commented Apr 26, 2023

Should I change txvContributors in util.cpp then or not - wasn't sure...

I think so? But probably once this PR is approved and very close to a merge. We'd then regroup your commits/squash them and have the contributor addition as separate commit.

Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.github/autobuild/get_build_vars.py Show resolved Hide resolved
tools/generate_json_rpc_docs.py Show resolved Hide resolved
@ann0see
Copy link
Member

ann0see commented Apr 29, 2023

Next step: Check if build behaves as expected

@ann0see ann0see added this to the Release 3.10.0 milestone Apr 30, 2023
@ann0see ann0see self-requested a review April 30, 2023 18:58
@ann0see
Copy link
Member

ann0see commented Apr 30, 2023

@declension I think this is very close. Please also have a look at the hidden comments. They aren't much.

Especially: #3052 (comment)

@ann0see ann0see removed their request for review April 30, 2023 19:00
Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still a small comment, but I believe this is ready afterwards.

@pljones pljones added the refactoring Non-behavioural changes, Code cleanup label May 5, 2023
@ann0see
Copy link
Member

ann0see commented May 7, 2023

Ok. I'm still not fully sure about the column length. It's a bit unfortunate that we need to have different ones in Python and C files - as far as I understand. It's certainly not blocking.

I'll read through the code once again, but I think that after the commits have been squashed into one, we can merge this.

Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Please squash this into one commit if possible.

@declension
Copy link
Contributor Author

Ok. Please squash this into one commit if possible.

Is this to the maintainer or to me?

I normally leave this to maintainers since it's more about their choice than the contributors, but if you want me to force push something I can I guess

@ann0see
Copy link
Member

ann0see commented May 8, 2023

If you squash it authorship is preserved in a better way. So I'd prefer you doing it.

@ann0see ann0see force-pushed the add-python-linting branch 2 times, most recently from 8063772 to b28e22e Compare May 11, 2023 21:13
* Ended up bigger than I'd hoped, but I think these are nearly all good changes...

* Add pylint in CI
* Split other linters into own jobs, was getting a bit all-or-nothing in that job
* Add a sane config tweaked from default, but less onerous
* Define the least-delta max-line-length that PEP-008 allows, 99 chars, and reflow a few lines. Looks nice anyway IMO
* Fix all the various errors
* Fix a few typos etc too
* Rename one module (script) that wasn't  python-friendly (snake_case) naming
* ...and references to it
* Initialise some object attributes in the constructor, not randomly elsewhere (soft warning in IDE)

This actually found a few bugs, and gotchas (woohoo) so it'd be good if someone could sanity check those changes here:
  * In get_release_contributors `realname` was never defined so that code path would throw exceptions
 * Some of the regex char classes were actually control characters (missing the raw string prefix), so fixed.

This closes jamulussoftware#2546

Actually remove all escaping of literal braces

Simple testing proves this isn't needed here:

$ cat /usr/include/qt/QtCore/qlocale.h | python -c 'import re; import sys; s = sys.stdin.read(); print(re.search(r"enum Country[^\n]+{([^}]+)}", s).group(1))' | md5sum
46625fcf574732033f70b2f10ce25bc4  -

vs
$ cat /usr/include/qt/QtCore/qlocale.h | python -c 'import re; import sys; s = sys.stdin.read(); print(re.search(r"enum Country[^\n]+\{([^}]+)\}", s).group(1))' | md5sum
46625fcf574732033f70b2f10ce25bc4  -
Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I've now squashed it. Will merge soon

@ann0see ann0see merged commit ad50a9d into jamulussoftware:main May 11, 2023
@ann0see
Copy link
Member

ann0see commented May 11, 2023

Congrats - and sorry for the huge delay. But development on Jamulus is slow...

@ann0see
Copy link
Member

ann0see commented May 12, 2023

Codacy on my repo still found an error https://app.codacy.com/gh/ann0see/jamulus/pullRequest?prid=11754631

I believe it's not that serious

@declension
Copy link
Contributor Author

Codacy on my repo still found an error https://app.codacy.com/gh/ann0see/jamulus/pullRequest?prid=11754631

I believe it's not that serious

Yes that looks safe to me - as it says there are no shell injection possibilities here (no shell at all), and I can't currently see any way that passing an --author=$BAD can be a problem, but maybe there are. git is pretty battle-tested anyway

@declension
Copy link
Contributor Author

Thanks for merging, and all your reviews @ann0see!

@pljones pljones added the tooling Changes to the automated build system label Jun 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
refactoring Non-behavioural changes, Code cleanup tooling Changes to the automated build system
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

CI: Run Python syntax check
3 participants