Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate numpy member transforms to reduce function calls #2550

Merged

Conversation

correctmost
Copy link
Contributor

Type of Changes

Type
βœ“ πŸ”¨ Refactoring

Description

pylint-dev/pylint#9783 notes that numpy predicates are called millions of times when linting large codebases. I was able to reduce these calls by ~5.9 million when linting yt-dlp by consolidating the transforms.

Stats

I profiled yt-dlp @ ef36d517f9b0 with the import-error rule.

Before

# Total: 62145284 function calls

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  4560600    0.529    0.000    0.529    0.000 astroid/brain/brain_numpy_utils.py:64(name_looks_like_numpy_member)
  1875144    0.212    0.000    0.213    0.000 astroid/brain/brain_numpy_utils.py:72(attribute_looks_like_numpy_member)
Command Mean [s] Min [s] Max [s] Relative
pylint --recursive=y . 33.301 Β± 0.262 32.927 33.765 1.00

After

# Total: 56197047 function calls

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   228030    0.060    0.000    0.060    0.000 astroid/brain/brain_numpy_utils.py:79(member_name_looks_like_numpy_member)
   234393    0.045    0.000    0.046    0.000 astroid/brain/brain_numpy_utils.py:88(attribute_name_looks_like_numpy_member)
Command Mean [s] Min [s] Max [s] Relative
pylint --recursive=y . 32.800 Β± 0.345 32.352 33.517 1.00

Copy link

codecov bot commented Sep 8, 2024

Codecov Report

All modified and coverable lines are covered by tests βœ…

Project coverage is 92.94%. Comparing base (887668b) to head (99186c7).
Report is 38 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2550      +/-   ##
==========================================
- Coverage   92.97%   92.94%   -0.04%     
==========================================
  Files          93       93              
  Lines       11081    11086       +5     
==========================================
+ Hits        10303    10304       +1     
- Misses        778      782       +4     
Flag Coverage Ξ”
linux 92.82% <100.00%> (-0.04%) ⬇️
pypy 92.94% <100.00%> (-0.04%) ⬇️
windows 92.92% <100.00%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Ξ”
astroid/brain/brain_numpy_core_function_base.py 100.00% <100.00%> (ΓΈ)
astroid/brain/brain_numpy_core_multiarray.py 100.00% <100.00%> (ΓΈ)
astroid/brain/brain_numpy_core_numeric.py 100.00% <100.00%> (ΓΈ)
astroid/brain/brain_numpy_utils.py 88.57% <100.00%> (-11.43%) ⬇️

Copy link
Member

@Pierre-Sassoulas Pierre-Sassoulas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is yt-dlp using numpy ? I wonder if some brains should be user activated. (Via an astroid-numpy or pylint-numpy plugin).

Also we might want to deprecate now or even remove now, I doubt those are APIs used by lots of person. Removing in 4.0 seems reasonable (waiting for another input on that, we might just remove now).

@correctmost
Copy link
Contributor Author

yt-dlp doesn't seem to use numpy.

I was hesitant to remove the unused functions from brain_numpy_utils.py because of backward compatibility concerns. Should there be a tracking ticket in the 4.0.0 milestone for API changes?

I'm wondering if some of the API clean-ups needed for typing can happen "in place" like the changes in this PR. The code (and tests) would be messier in the short term, but it might make it easier to make progress without having to maintain multiple branches. The idea needs more investigation, though :).

@jacobtylerwalls
Copy link
Member

I'll discuss this on discord, but I'd like to advance the main branches of pylint and astroid to 4.0-dev later this month.

@DanielNoord
Copy link
Collaborator

@jacobtylerwalls I think this happened for pylint right? Can we merge this then?

@jacobtylerwalls
Copy link
Member

Sure thing. I'll follow with a PR do the same in astroid so that @correctmost can resolve the todos by doing the removals.

@jacobtylerwalls jacobtylerwalls merged commit c7ea1e9 into pylint-dev:main Sep 30, 2024
21 checks passed
@correctmost correctmost deleted the cm/reduce-numpy-transforms branch October 2, 2024 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants