-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
exclude '.tox', '.nox' from being copied during 'pip install .' #6770
Conversation
…l . to speed installation up
cc @theacodes |
Omg yes please. This will make nox much faster for common use cases and will also make tox faster for others. |
We did this before and had to revert it because it broke people's |
Legacy is fun. :)
Could we put this behavior behind a flag? Would that be horribly offensive?
…On Mon, Jul 22, 2019 at 7:38 PM Donald Stufft ***@***.***> wrote:
We did this before and had to revert it because it broke people's setup.py
files, some setup.py files make assumptions about what's on disk in
certain phases of development and we cant just start YOLO excluding files.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6770?email_source=notifications&email_token=AAB5I4YKMIUFWYFNJ5D4RGDQAZVLPA5CNFSM4IF7FWF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2RXULA#issuecomment-514030124>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAB5I432RKXMLIL7B5PXP43QAZVLPANCNFSM4IF7FWFQ>
.
|
@dstufft, what about a .pipignore solution? ideally we should just install what's included in the source dist. I guess if someone wants the whole git repo in their source dist its their right and they should opt in to that. That kind of patch is a bit beyond me though, I never even looked at the code for pip before. |
@dstufft, also I would personally be more than happy with just .tox and .nox. |
For running the Linters locally: tox -e lint-py3 https://pip.pypa.io/en/stable/development/getting-started/#running-linters |
The ideal solution for this that we're looking at is to build a sdist from the current directory, and then build a wheel using that - this would avoid copying anything but the essential files needed for a build. |
that would be the ideal solution. |
I'm okay with this -- but if someone comes around to asking for more, I'm gonna point them at #2195 and tell them that is how we have to solve this. I have 0 interest in managing a blocklist that has to be kept up to date. |
this may potentially break setuptool_scm worse than pip usually does |
VCS directories would be continue to be copied. That's not the current state of the PR but that's definitely what we're going to do, if we do this. |
@RonnyPfannschmidt The original proposal not to copy VCS directories (which definitely would break setuptools_scm) or the revised one just to skip |
thanks for the call out, i missed the update about the scope reduction |
Well... We had #4900 and even there, I think it was clear that going dir -> sdist -> installed route is a lot better. |
I agree that it makes more sense to use the sdist. |
Okay, with the scope reductions I think we can move forward with this? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs tests and a skim through the documentation to put a sentence about the skipped directories in the right place.
The tests should probably also verify that VCS directories continue to get copied.
can you point me to a test that does something similar? |
it does not seem like shutil.copytree supports having an ignore glob that only effects the top level. |
The ignore function receives the "current" directory being iterated over and the list of contents. If the current directory is not equal to the top-level source directory then the list of contents can be returned without any changes. E.g.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few minor comments.
Could we also put a note about this new behavior somewhere in the docs? I would add it to or make a new section after pip_install/#build-system-interface. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixing in a diff shortly.
about the doc update, I am not sure where this should go. it seems that explaining this will require explaining that pip copies the entire directory to a temp location prior to installing, which is already quiet a handful to dump on the user in the usage manual. |
I'd be inclined to put it in the docs on the build system interface, as @cjerdonek suggested. It could be an additional paragraph (or subsection) explaining how a local source is prepared for building, explaining that it's copied to a temporary directory (excluding I don't think it needs to be an in-depth discussion, just hit the points I mention above and that should do (if anyone feels like going into greater detail, they can always submit a follow-up PR). |
After looking at the docs, I feel that the proper place for this is the pip_install reference page. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2 minor items, otherwise LGTM.
tests/unit/test_download.py
Outdated
def test_unpack_file_url_excludes_expected_dirs(tmpdir, exclude_dir): | ||
src_dir = tmpdir / 'src' | ||
dst_dir = tmpdir / 'dst' | ||
src_included_file = Path.joinpath(src_dir, 'file.txt') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, my previous comment was not very good - I meant using the joinpath
method on src_dir
which is now a Path
, like src_dir.joinpath('file.txt')
. Likewise with joinpath
and touch
below.
docs/html/reference/pip_install.rst
Outdated
|
||
$ pip install path/to/SomeProject | ||
|
||
During the installation, pip will copy the entire project directory to a temporary location and install from there. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would say regular installation
just so people don't get the idea that it may apply to editable installs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
@pradyunsg, I feel like addressed your change request. |
@pradyunsg This LGTM, but I don't want to merge with your review request outstanding. Can you confirm it's been dealt with (I think it has) and I'll merge (or you can)? @omry Thanks for your contribution :-) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Happy to let someone else click the merge button -- I really need to sleep.
Done. Thanks again @omry! |
Thanks! |
In October. We have a 3 month release cadence. |
Thanks for making this happen, everyone. We'll have some happy tox and Nox
users in October. 💜
…On Mon, Aug 5, 2019, 9:16 PM Pradyun Gedam ***@***.***> wrote:
In October. We have a 3 month release cadence.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6770?email_source=notifications&email_token=AAB5I42R2SMIO7CC4PVDCQLQDD3LLA5CNFSM4IF7FWF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3TYXRA#issuecomment-518491076>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAB5I4Z3FVV5XWBESMGHBZDQDD3LLANCNFSM4IF7FWFQ>
.
|
'pip install .' is very slow in the presence of large directories in the source tree.
Specifial test automation directries (.tox, .nox).
This diff excludes the common culprits from the copy to a temporary directory, speeding up pip install .
significantly in such cases.
For my own repo, this gets the pip install . speed from 1 minute and 30 seconds to 2 seconds.
EDIT:
I changed this PR to address only .tox and .nox due to concerns about not copying scm directories breaking build systems.
Future improvements to this should be in the form of changing the build to build a sdist from the current directory, and then build a wheel using that.
(see comment below from @pfmoore).