Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Have pip cache purge and pip cache remove handle empty directories. #9058

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Have pip cache purge and pip cache remove handle empty directories. #9058

wants to merge 1 commit into from

Conversation

duckinator
Copy link
Contributor

@duckinator duckinator commented Oct 26, 2020 •
edited
Loading

EDIT (May 6 2022): Editing to add an updated todo list.

TODO:

  • Check for directories without .whl files, instead of empty directories — see the comment by @sbidoul below
  • Add tests (at least for subdirs_with_no_files(path) or whatever this gets changed to).

These are the changes discussed in #7372:

  1. Make pip cache remove prune empty directories.
  2. Make pip cache purge delete everything in the http and wheels caches (not just files).
  3. Make pip cache purge remove selfchecks.json.

This PR does not account for having pip cache purge remove things in the pip/selfchecks/ directory when appropriate, which was also discussed in #7372.

@pradyunsg
Copy link
Member

Heads up: The MacOS workers on Azure Pipelines are failing right now -- #9030.

@duckinator
Copy link
Contributor Author

duckinator commented Oct 26, 2020 •
edited
Loading

Thanks for the heads up. I also just discovered that os.scandir() didn't arrive until Python 3.5, so will need to use something else.

@pradyunsg
Copy link
Member

You could wait for a couple of weeks, and we'll drop Python 2 support from master. :)

@duckinator
Copy link
Contributor Author

Making it work without using os.scandir() is apparently pretty annoying.

Do y'all support Python 3 versions before 3.5? If you don't then, screw it, I'll just wait for you to drop Python 2 support lmao

@pradyunsg
Copy link
Member

Do y'all support Python 3 versions before 3.5?

Nope. Python 3.5 and Python 2 get dropped after 20.3. :)

@duckinator
Copy link
Contributor Author

Okay, in that case let's leave this for after Py2 is dropped. 👍

@duckinator
Copy link
Contributor Author

duckinator commented Dec 4, 2020 •
edited
Loading

Rebased off of master (ab7ff0a).

(It seems Py2 is still in the list of tests; just wanted to keep the PR from getting stale.)

@duckinator
Copy link
Contributor Author

duckinator commented Feb 3, 2021 •
edited
Loading

Rebased off master (d108e49).

I'm not sure what the reason for the failing CI tasks is. I do see this error in the failed tasks, however:

ERROR: InvocationError for command /Users/runner/work/pip/pip/.tox/lint/bin/pre-commit run --all-files --show-diff-on-failure --hook-stage=manual (exited with code 1)

@duckinator
Copy link
Contributor Author

duckinator commented Feb 3, 2021 •
edited
Loading

As a note, anything requiring more than rebasing this PR will probably have to wait a month or two at least. Pretty busy with trying to get a house and such. 😅

@uranusjr
Copy link
Member

uranusjr commented Feb 3, 2021

This seems to be the specific error:

flake8...................................................................Failed
- hook id: flake8
- exit code: 1

src/pip/_internal/commands/cache.py:190:17: E127 continuation line over-indented for visual indent

@duckinator
Copy link
Contributor Author

ah, thank you @uranusjr. Had trouble finding that in the CI output. 😅

I fixed that problem (or at least, that's what tox -e lint says locally), but it still ends with this and has exit code 1, for some reason:

isort....................................................................Passed
mypy.....................................................................Passed                            
use logger.warning(......................................................Passed
check for eval().........................................................Passed
rst ``code`` is two backticks............................................Passed
NEWS fragment........................................(no files to check)Skipped
check-manifest...........................................................Passed
ERROR: InvocationError for command /home/puppy/dev/python/duckinator/pip/.tox/lint/bin/pre-commit run --all-files --sho
w-diff-on-failure --hook-stage=manual (exited with code 1)
_______________________________________________________ summary _______________________________________________________
ERROR:   lint: commands failed

Not sure what's up with that.

@sbidoul
Copy link
Member

sbidoul commented Feb 3, 2021

@duckinator the flake8 error seems to be still present.

@BrownTruck
Copy link
Contributor

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

@BrownTruck BrownTruck added the needs rebase or merge PR has conflicts with current master label Jun 11, 2021
@pradyunsg
Copy link
Member

pradyunsg commented Sep 18, 2021 •
edited
Loading

Hiya @duckinator -- would you be interested in updating this PR? :)

@pradyunsg pradyunsg added the S: awaiting response Waiting for a response/more information label Sep 18, 2021
@duckinator
Copy link
Contributor Author

@pradyunsg I'll rebase this sometime in the next few days. If it's not done by Wednesday (September 22nd) feel free to ping me here again. 👍

@duckinator
Copy link
Contributor Author

Running a bit behind on things this week, but still hoping to get to it in the next few days. Sorry for the delay!

@pradyunsg pradyunsg added this to the 21.3 milestone Sep 22, 2021
@pradyunsg
Copy link
Member

No worries and thanks for the update! Please don’t feel pressured to update this ASAP. :)

I’m gonna put this into the release milestone to remind myself to keep an eye out for updates here — it’s totally fine if this gets pushed down the road.

@pypa-bot pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 24, 2021
@duckinator duckinator force-pushed the issue-7372 branch 2 times, most recently from c75cae3 to 11db9a5 Compare September 24, 2021 06:34
@duckinator
Copy link
Contributor Author

Rebased, and simplified(!) because some of the changes in the last ~year let me simplify my code. ^.^

@uranusjr
Copy link
Member

Gentle ping

@duckinator
Copy link
Contributor Author

@uranusjr hey, thanks for the nudge. Got a bit overwhelmed with other things, and this fell through the cracks.

@duckinator
Copy link
Contributor Author

Rebased off main (as of c394bf6). Currently working on adding types to everything I added.

@duckinator
Copy link
Contributor Author

duckinator commented Apr 12, 2022 •
edited
Loading

Added type information to all the functions I added in src/pip/_internal/utils/filesystem.py. Last remaining thing is tests for pip._internals.utils.filesystem.subdirs_with_no_files(path_str).

EDIT: I missed the return types, whoops.

@duckinator duckinator force-pushed the issue-7372 branch 2 times, most recently from e600b8b to b8a5a79 Compare April 12, 2022 22:38
@duckinator duckinator force-pushed the issue-7372 branch 2 times, most recently from d5b9ca0 to f417270 Compare April 19, 2022 04:10
@duckinator
Copy link
Contributor Author

duckinator commented Apr 19, 2022 •
edited
Loading

Incorporated @uranusjr's suggestions and rebased off c247ddc.

EDIT: I thought @uranusjr's changes would work verbatim and just went for it because this system has no Python dev environment. CI has informed me I was wrong. Undoing that for now. 😅

Still need to add tests for subdirs_with_no_files(path_str).

@sbidoul
Copy link
Member

sbidoul commented May 3, 2022

To implement #5037, I am going to add some metadata files in the wheel cache entries directories. So instead of checking for empty directories, we may want to check for directories with no .whl remaining files.

@duckinator
Copy link
Contributor Author

@sbidoul thanks for the heads-up. I added an updated TODO in the first comment (so it's not buried), which includes accounting for that. 👍

@duckinator

This comment was marked as outdated.

Sign in to view
@duckinator
Copy link
Contributor Author

Rebased off main (81f6a9f).

@uranusjr
Copy link
Member

Need to update tests to match the new output.

@sbidoul
Copy link
Member

sbidoul commented Apr 10, 2023

Since 22.2 introduced origin.json in cache entries, and with #11897, this PR and it's first remaining TODO becomes more important:

Check for directories without .whl files, instead of empty directories

@duckinator
Copy link
Contributor Author

Rebased off main (3682309) and updated tests to match the new output format.

@duckinator
[commands/cache] make pip cache purge remove everything from http +…
12d7676 
… wheels caches; make `pip cache remove` prune empty directories.
@duckinator
Copy link
Contributor Author

Rebased off main (e98cc5c).

@notatallshaw
Copy link
Member

Can you add a news entry please: https://pip.pypa.io/en/latest/development/contributing/#news-entries

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pradyunsg pradyunsg pradyunsg left review comments

@sbidoul sbidoul sbidoul left review comments

@uranusjr uranusjr uranusjr left review comments

Assignees
No one assigned
Labels
C: cache Dealing with cache and files in it type: enhancement Improvements to functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@duckinator @pradyunsg @uranusjr @sbidoul @BrownTruck @notatallshaw @pypa-bot