-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pre-existing build directory fix #8283
Pre-existing build directory fix #8283
Conversation
03d25f1
to
9024011
Compare
Uh there’s a test ensuring |
I'm really uncomfortable about "just" adding a disambiguating string here. I'd rather we properly understood why we have two build directories with the same name first. I think it's probably a legitimate fix (in the new resolver I guess it's possible that we're looking at two sdists for the same project at the same time, just with different versions) but I think we should be clearer about what's going on before applying this. |
OK tl;dr; version: We don't guarantee the name used for the build directory, so I'd be fine with adding a UUID. Maybe if the directory with just the project name doesn't exist, use that so that in the normal case there's no change in behaviour. We can fix the tests that check for the error to instead confirm that the existing directory doesn't get overwritten (i.e, instead of asserting that pip fails, assert that pip succeeds and that the pre-created directory remains present and unchanged). Some random unrelated thoughts on the failing tests:
The remainder of this is rambling notes I made while investigating. They took some time to discover, so I haven't deleted them. But they are probably of marginal value if we all agree with the suggestion above. I did some investigation. If The code in this area has changed. For unnamed requirements we use I suspect the whole of the mechanism that lets the user specify a build directory and try to diagnose build issues is only of use to a tiny proportion of users, and anyone with the expertise to do such a diagnosis is probably equally capable of manually building a wheel using the backend directly, so I think this is a prime candidate for deprecation on the basis of YAGNI. |
It's an OLD change, that had lots of tests marked as network. #2354
Sounds good to me! I thought I'd written a comment pointing toward this as well. I think I either dreamt that up or (more likely) forgot to post it because I was annoyed while trying to type down an explaination of why network requests during dependency resolution are a bad idea. :) |
This only happens in test_req.py. It's the only test module that imports from Namely, we "only" need to update |
I am tempted to always use a UUID, at least in the new resolver, since I kind of feel it may improve pip’s concurrency situation (the “is there an existing directory” check can be a potential race condition). I think we can add a flag to One additional observation: Is this call path also used for PEP 517 packages? Because the logic to check whether there’s a previous build directory is setuptools-dependant: pip/src/pip/_internal/operations/prepare.py Lines 415 to 432 in 9024011
|
That's a fair point. On that basis I'm fine with using a UUID always.
Um, quite probably...? I'll take a look at this after lunch (I'm busy this morning on other stuff). This might need a wider review. |
@pradyunsg see #8333 |
9024011
to
4ca684f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm guessing you chose the argument name parallel_builds
for this because you expect it to also be useful for the work to add multithreading to pip? I don't have a problem with the name, but if it's not linked to the multithreading stuff, we're going to end up with another case of the same name being used for two different ideas. So I'm not asking for a change (I don't know of a better name) just flagging the point up.
Some of the tests have started failing (the Travis job has expected passes that now fail. These need addressing before we merge. (I know Travis is "not required", but my view is that's only if it's green but not reporting its status correctly).
Correct. I am not very satisfied with the name either, but have nothing better to offer 🙁
It turns out there are tests checking whether the build directly ends with the package name, so we appending a UUID breaks them. Arrgh why are things so difficult. |
137468a
to
7b2c64c
Compare
I pushed the following changes:
|
These are the only cases where backtracking can happen. This approach also accounts for VCS requirements relying on the same ensure function to do cloning :/
The new resolver uses UUID to allow parallel build directories, so this error will no longer occur.
7b2c64c
to
daf454b
Compare
This looks like us picking up a whole bunch of technical debt. I'm on board for us doing this, since we have already run out of contract hours in some senses, for the new resolver work, and cleaning up the mess that is pip's build directory handling is too big for us right now. I did want to flag this, just in case someone from the future comes and wonders why the heck we did something like this. |
I'd characterise it as paying interest on a long-standing technical debt. The build directory stuff has been a mess that's been around for ages, and the new resolver work can't be expected to fix all the problems in pip's code base. If we have to add nasty workarounds because of that existing issue, that's a shame, but inevitable. A genuinely useful additional funding opportunity would be precisely to reduce technical debt in pip's codebase, without the constraints of a specific project deliverable like the new resolver :-) |
@pradyunsg Gentle ping on this, I think it's just waiting for your approval. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
This would be awesome, but does anything in the world work like this? :o |
Well-bounded refactoring without deliverables is generally accepted as a legitimate project in commercially-run companies with reasonable project management. It is not easily funded for pip, but that’s a problem in the OSS structure in general, not the idea of refactoring as a fundable project. |
Looks like this failed the new lint checks. @pradyunsg I seem to remember seeing you were involved in that PR - do you know what the agreed process is for this? Was the idea that PRs that failed would need a follow-up PR to fix the new warnings? |
Yea, a follow up pr fixing them would be best. |
I see two approaches of fixing the current lint failures caused by the offending line in follow-up PRs at pip/src/pip/_internal/cli/cmdoptions.py Line 838 in 7bf78f0
|
Let's go with 1. ^.^ |
Cool, I have made the required changes in #8405 , we can go ahead and merge it to fix lint failures. |
Fix for #8282. I’m not yet sure we can safely append the hex. It does make the test pass though.
The code here depends on th branch for #8282; it needs to be merged first.