Fix Pruning callback and add a few features

[carmocca]

What does this PR do?

Many test fixes
Many refactors

TODO:

• Make lottery a Callable + tests
• Support iterative pruning + tests
• Save pruned model on checkpoint
• Check docs look good
• Sparsity history and tracking
• Allow resampling
• Full model statistics, more verbose level
• Support loading non-permanent pruned checkpoint
• Second chance? edit: left for a future PR (maybe?)
• CHANGELOG edit: No need since pruning hasn't been released yet

Before submitting

• Was this discussed/approved via a GitHub issue?: Discussed offline
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified
• Check that target branch and milestone match!

Did you have fun?

Make sure you had fun coding 🙃

[pep8speaks]

Hello @carmocca! Thanks for updating this PR.

• In the file pytorch_lightning/callbacks/pruning.py:

Line 394:13: W503 line break before binary operator
Line 395:13: W503 line break before binary operator

Comment last updated at 2021-02-10 14:42:43 UTC

[codecov]

Codecov Report

Merging #5825 (0f3d113) into release/1.2-dev (774e9bd) will decrease coverage by 0%.
The diff coverage is 96%.

@@               Coverage Diff                @@
##           release/1.2-dev   #5825    +/-   ##
================================================
- Coverage               83%     83%    -0%     
================================================
  Files                  181     181            
  Lines                12929   12796   -133     
================================================
- Hits                 10771   10612   -159     
- Misses                2158    2184    +26     

[Borda]

it would be much easier to have each of the 5 features as a separate PR

[carmocca]

it would be much easier to have each of the 5 features as a separate PR

Consider this as a Pruning make-over more than adding individual features. The TODO above was just for me to track pending things

[Borda]

it would be much easier to have each of the 5 features as a separate PR

Consider this as a Pruning make-over more than adding individual features. The TODO above was just for me to track pending things

yes but it now because there is some code already, so remove all in one PR and then do this, now the reading with seeing past state is hard...

[tchaton]

Hey @carmocca, incredible work there !

[carmocca]

Line 394:13: W503 line break before binary operator
Line 395:13: W503 line break before binary operator

This is a false positive. This rule should be ignored:

https://github.com/PyTorchLightning/pytorch-lightning/blob/388eeea52d3abe5e8c08197ed06dc9ca9eeefe1c/setup.cfg#L83