mypy 0.982 run 15 minutes (previous version needs only one minute) #13821
Comments
|
Such things are hard to diagnose. You can try bisecting it to a specific mypy commit, then it will be much easier. |
|
I definitely will try today. Also I will try to extract problematic part of our code and share it on GH |
|
3ae19a2 this takes more than 25 minutes I run this in the background and killed after 25 minutes, it takes longer than previous because I use my PC. |
|
I looked at the commit, and a few things seemed potentially suspicious: changes in |
|
replaced almost all |
|
That would indicate that the |
|
In my case: True we can stick to 0.971, newest version of mypy is not crucial. However, it can affect many other projects CI/CD, especially calculation-dependent (our backend is one big calculator with hundreds calculator classes and some REST API) |
|
Yeah, this seems pretty bad and we should try to fix the regression. It would be very helpful if you could you try to narrow this down a bit -- it's possible that there is some specific call or calls to |
|
@Behoston you could try adding |
|
Hello, We are experiencing similar issues with Colour: We went from ~30mins to ~1h 20mins on CI when updating our dependencies, one of them was mypy==0.961 --> mypy==0.982. Cheers, Thomas |
|
Our project also went from ~5min to ~25min going from 0.971 to 0.981/2. |
|
It would be useful if you could post the details of your projects, so we can reproduce the slow down. As is mentioned above, it's very hard to diagnose these things in the abstract. If your project is not open source, it would be very helpful if you could make a shareable repro of the bad behaviour. For example, try some of the strategies mentioned above:
|
|
We can probably just revert the change to sum, it's a somewhat weird change to have: #13961 In all cases, would love to have small repros of the performance regression |
|
https://github.com/dostanesie-pl/mypy-bug @hauntsaninja I published part of our code. |
|
Unfortunately I cannot publish our code. Maybe that has nothing to do with the cause for this, some of our modules have increased by a factor of 10, while others stay the same. |
|
Works much better on version |
|
We're still seeing substantial slowdowns on version |
|
In my case time is acceptable (at least locally, I need to fix some code so
not tested on Ci yet), so I cannot help with 0.990
pt., 11 lis 2022, 14:02 użytkownik James Marchant ***@***.***>
napisał:
… We're still seeing substantial slowdowns on version 0.990; is there an
easy way to see which lines are slow (have already used --timing-stats to
narrow down the files)
—
Reply to this email directly, view it on GitHub
<#13821 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB3WCSOZ3NLFXLQB75EGAODWHY7U3ANCNFSM6AAAAAAQ56WG3I>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
0.990 did not really change anything for us, still 1h20+ check times here. |
|
Have bisected the commits between 0.971 and 0.981 to identify which one causes the slowdown for us. For us it seems to be d27bff62f71a8b914b1df239467148e81d2e88a2:
|
|
Hm, that PR is almost a pure refactor. Actually I have something to suspect. Potentially you have some literal type with a lot of values or some large enum. There is a function |
|
I ran
|
|
For comparison here's the
|
|
@JMMarchant thanks for the data. Unfortunately, I still can't figure out what is going on. Would it be possible to profile on the last good and first bad commit? Just to exclude other possible changes to the call graph (there were quite a lot in recent couple releases). Also could you please post your mypy config? |
[mypy]
mypy_path = $MYPY_CONFIG_FILE_DIR/stubs
files =
src/,
dev-scripts/,
docs/,
scripts/,
setup.py,
tests/
; Error message config
show_error_context = True
show_column_numbers = True
show_error_codes = True
pretty = True
; Untyped definitions and calls
disallow_untyped_defs = True
disallow_incomplete_defs = True
check_untyped_defs = True
disallow_untyped_decorators = True
; None and optional
no_implicit_optional = True
; Warnings
warn_redundant_casts = True
warn_unused_ignores = True
warn_no_return = True
warn_return_any = True
warn_unreachable = True
; These are all the third-party libraries we have generated partial or full stubs
; for. We are a bit more permissive for what is allowed in these stubs/how detailed
; they need to be. They can all be found in the stubs/ folder.
; - albumentations
; - envyaml
; - fire
; - GPUtil
; - grpc
; - intermine
; - lightgbm
; - marshmallow_enum
; - marshmallow_union
; - matplotlib
; - methodtools
; - opacus
; - pandasql
; - pdoc
; - pyarrow
; - pydicom
; - pytorch_tabnet
; - scipy
; - skimage
; - sklearn
; - snsql
; - sphobjinv
; - sqlparse
; - sqlvalidator
; - statsmodels
; - torchvision
; - wirerope
[mypy-albumentations.*,envyaml.*,fire.*,GPUtil.*,grpc.*,intermine.*,lightgbm.*,marshmallow_enum.*,marshmallow_union.*,matplotlib.*,methodtools.*,opacus.*,pandasql.*,pdoc.*,pyarrow.*,pydicom.*,pytorch_tabnet.*,scipy.*,skimage.*,sklearn.*,snsql.*,sphobjinv.*,sqlparse.*,sqlvalidator.*,statsmodels.*,torchvision.*,wirerope.*]
allow_incomplete_defs = True
allow_untyped_defs = True
disallow_any_unimported = True
[mypy-docs.*]
disallow_any_expr = True
; Suppress less useful errors for tests
[mypy-tests.*]
; Tests with multiple asserts may cause code to be marked as unreachable even
; though it's fine. See https://github.com/python/mypy/issues/9457
warn_unreachable = False
; Suppress errors for untyped third party test libraries:
; - chromedriver_autoinstaller
; - jupytext
; - nbclient
; - nbformat
; - pytest_flask
; - requests_toolbelt
; - testbook
[mypy-chromedriver_autoinstaller.*,jupytext.*,nbclient.*,nbformat.*,pytest_flask.*,requests_toolbelt.*,testbook.*]
ignore_missing_imports = True |
This should help with the investigation of tricky performance regressions like #13821. I tried to implement in such a way that it will give minimal impact when not used (since I am touching a hot method).
|
@JMMarchant After looking a bit more at these, it seems like something may be slow with union handling and/or overloads. I just merged a PR that adds an optimization to overloads, could you please:
|
|
I am also seeing significant performance regression between 0.971 and master: 5min -> 20min for a mypyc-compiled run on my codebase. The per-file time stats point to a file that has a large function with many if/else branches depending on values of a very large Enum (~13min for a single file). However the per-line time stats seem to be inconsistent with the per-file stats, and while the same file does show up with some slow lines, the numbers don't add up to the same amount as the per-file time stats. The largest lines on the per-line stats point to overloaded methods taking the enum value as a parameter (specifically I will try to reduce this file into a more practical reproducer and profile it in the next few days. |
|
Times are expected to not add up exactly because line stats only count time spent in expressions (and only for type checking). There may be some additional time spent analyzing enclosing statement, but it is hard to attribute a line number to this and such time is usually small. |
I did expect a small difference, but this is not even close: per-file timing: 790953050us There's a >600x mismatch! |
|
Profiling results are interesting! There are ~20 if statement branches to visit, and each branch spends ~2min (pure-python run) in the This also explains the mismatch in timings, as the per-line timings are only collected in Relevant speedscope output for a single if statement branch: @ilevkivskyi I think the merging of subtype visitors is likely to blame for that. I have a hunch that some of the optimizations in the proper subtype check got lost along the way. For instance This sort of issue/regression has come up a number of times. It seems to me it might be worth revisiting the way Enum narrowing is done. Maybe instead of always exploding them into |
|
Yep, when you mentioned the mismatch, binder was my first guess (and it indeed uses proper subtypes). I am not sure what is the best way to go here. One option is to restore the previous logic for proper subtypes for unions. Although it would be inconsistent (the whole point of the merge was that proper vs non-proper subtype checks should go through the same code paths except when an At least I can try to find a minimal PR that would fix the regression. I will post here when I will have something. |
|
@huguesb @JMMarchant (and others). Could you please try #14276 to see if it fixes performance for you? |
Enums are exploded into Union of Literal when narrowed. Conditional branches on enum values can result in multiple distinct narrowing of the same enum which are later subject to subtype checks (most notably via `is_same_type`, when exiting frame context in the binder). Such checks would have quadratic complexity: `O(N*M)` where `N` and `M` are the number of entries in each narrowed enum variable, and led to drastic slowdown if any of the enums involved has a large number of valuees. Implemement a linear-time fast path where literals are quickly filtered, with a fallback to the slow path for more complex values. Fixes python#13821
I have not tried it but I doubt it will help much since the profiling clearly shows the bulk of the time being spent in the generator expr above, at line 302. I have a different fix in mind: #14277 |
Enums are exploded into Union of Literal when narrowed. Conditional branches on enum values can result in multiple distinct narrowing of the same enum which are later subject to subtype checks (most notably via `is_same_type`, when exiting frame context in the binder). Such checks would have quadratic complexity: `O(N*M)` where `N` and `M` are the number of entries in each narrowed enum variable, and led to drastic slowdown if any of the enums involved has a large number of valuees. Implemement a linear-time fast path where literals are quickly filtered, with a fallback to the slow path for more complex values. In our codebase there is one method with a chain of a dozen if statements operating on instances of an enum with a hundreds of values. Prior to the regression it was typechecked in less than 1s. After the regression it takes over 13min to typecheck. This patch fully fixes the regression for us. Fixes python#13821
Enums are exploded into Union of Literal when narrowed. Conditional branches on enum values can result in multiple distinct narrowing of the same enum which are later subject to subtype checks (most notably via `is_same_type`, when exiting frame context in the binder). Such checks would have quadratic complexity: `O(N*M)` where `N` and `M` are the number of entries in each narrowed enum variable, and led to drastic slowdown if any of the enums involved has a large number of valuees. Implemement a linear-time fast path where literals are quickly filtered, with a fallback to the slow path for more complex values. In our codebase there is one method with a chain of a dozen if statements operating on instances of an enum with a hundreds of values. Prior to the regression it was typechecked in less than 1s. After the regression it takes over 13min to typecheck. This patch fully fixes the regression for us. Fixes python#13821
Enums are exploded into Union of Literal when narrowed. Conditional branches on enum values can result in multiple distinct narrowing of the same enum which are later subject to subtype checks (most notably via `is_same_type`, when exiting frame context in the binder). Such checks would have quadratic complexity: `O(N*M)` where `N` and `M` are the number of entries in each narrowed enum variable, and led to drastic slowdown if any of the enums involved has a large number of valuees. Implemement a linear-time fast path where literals are quickly filtered, with a fallback to the slow path for more complex values. In our codebase there is one method with a chain of a dozen if statements operating on instances of an enum with a hundreds of values. Prior to the regression it was typechecked in less than 1s. After the regression it takes over 13min to typecheck. This patch fully fixes the regression for us. Fixes python#13821
Enums are exploded into Union of Literal when narrowed. Conditional branches on enum values can result in multiple distinct narrowing of the same enum which are later subject to subtype checks (most notably via `is_same_type`, when exiting frame context in the binder). Such checks would have quadratic complexity: `O(N*M)` where `N` and `M` are the number of entries in each narrowed enum variable, and led to drastic slowdown if any of the enums involved has a large number of values. Implement a linear-time fast path where literals are quickly filtered, with a fallback to the slow path for more complex values. In our codebase there is one method with a chain of a dozen `if` statements operating on instances of an enum with a hundreds of values. Prior to the regression it was typechecked in less than 1s. After the regression it takes over 13min to typecheck. This patch fully fixes the regression for us. Fixes #13821.
Bug Report
My project is not open-source but I can give you (mypy maintainers) access to repository on gitlab.
After updating mypy to newest version it takes 15 minutes to complete on CI and 8 minutes on my PC. When I rollback to previous release it will works fine (about 15 seconds on local machine and about 45 on CI).
I have base class for about 100 subclasses and mypy hangs always in same place.
I tried deleting mypy cache dir locally, on CI we build new docker image each run so there is no mypy cache.
To Reproduce
I really don't know :(
Expected Behavior
Mypy should run about one minute for version
0.971.Actual Behavior
On CI it takes more than 15 minutes, on local machine 8 minutes.
Your Environment
mypy.ini(and other config files):Full mypy log
The text was updated successfully, but these errors were encountered: