Mutually exclusive --target, --user and --prefix #4557

techtonik · 2017-06-17T15:25:40Z

Rebased #4111.

pradyunsg · 2017-06-21T05:22:40Z

Thanks for this! ^.^

@techtonik You have a failing test, could you look into it?

techtonik · 2017-06-22T16:37:22Z

I can't read the test right away. Not clear for me what is it doing. Some methods were deprecated and I don't really have the time to study them.

dpkp · 2017-06-24T20:33:47Z

fix for test failure:

diff --git a/tests/functional/test_install_reqs.py b/tests/functional/test_install_reqs.py
index 93592b2..1fedb20 100644
--- a/tests/functional/test_install_reqs.py
+++ b/tests/functional/test_install_reqs.py
@@ -262,6 +262,7 @@ def test_wheel_target_with_prefix_in_pydistutils_cfg(script, data,
                                                      virtualenv):
     # Make sure wheel is available in the virtualenv
     script.pip('install', 'wheel')
+    script.pip('download', 'setuptools', 'wheel', '-d', data.packages)
     virtualenv.system_site_packages = True
     homedir = script.environ["HOME"]
     script.scratch_path.join("bin").mkdir()

pradyunsg · 2017-06-27T14:48:43Z

fix for test failure:

@techtonik This indeed does fix the test. Could you add this to your PR?

In fact, you don't need to install wheel for this test.

Bo98 · 2017-08-15T11:19:27Z

The tests have since changed, so common_wheels will be the way to go here, which will resolve the failing test.

See b709b1e#diff-0ef5cb8465369ba3cca1484fcfb75e7fL226

Could you update the PR @techtonik?

pradyunsg · 2017-08-26T06:55:01Z

tests/functional/test_install_reqs.py

+def test_wheel_target_with_prefix_in_pydistutils_cfg(
+        script, data, virtualenv, common_wheels):
+    # pip needs `wheel` to build `requiresupper` wheel before installing
+    # (or to install it?)


Just to build it.

Thanks. Fixed the comment.

pypa-bot · 2017-08-26T08:26:21Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

pradyunsg · 2017-08-26T08:56:06Z

news/4111.bugfix

@@ -0,0 +1 @@
+Fix ``pip install --target`` by making ``--target``, ``--user``, and ``--prefix`` mutually exclusive.


I would phrase this as:

pip install `--target` is now mutually exclusive with `--user` and `--prefix`.

techtonik · 2017-08-26T17:51:33Z

Travis outage?

pradyunsg · 2017-08-26T17:59:21Z

The pypy job is flaky because of an upstream pytest bug. #4687 should help with that. No idea about the py34 job though.

I've restarted both of them.

techtonik · 2017-08-26T21:01:06Z

Looks like it doesn't help. Still some problems on Travis with websockets, probably unrelated but still.. https://www.traviscistatus.com/

BrownTruck · 2017-08-31T22:30:04Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

Also override defaults set somewhere (e.g. .pydistutils.cfg) when command line switches are given.

Replace ``--no-use-wheel`` with ``--no-binary=:all:``

Replace --no-use-wheel with --no-binary=:all: and remove assertion of deprecation message.

https://travis-ci.org/pypa/pip/jobs/244001886#L244

Looks like `--target` doesn't use packages installed in parent environment, so there is no sense to install `wheel` in parent https://travis-ci.org/pypa/pip/jobs/268573000#L547

It is not installing wheel. Either `pip` need to build wheel, which is impossible without `wheel` package, or it can not install wheels without `wheel` https://travis-ci.org/pypa/pip/jobs/268577126#L547

"Can not perform a '--user' install. User site-packages are not visible in this virtualenv." https://travis-ci.org/pypa/pip/jobs/268627056#L609

techtonik · 2017-12-31T11:23:12Z

I am out of mana. (

dpkp · 2018-02-06T21:33:41Z

I think this is an important fix and would love to help push across the finish line. I believe this fixes the merge conflict:

diff --cc src/pip/_internal/locations.py
index 5a20c92a,1043e57a..00000000
--- a/src/pip/_internal/locations.py
+++ b/src/pip/_internal/locations.py
@@@ -155,10 -155,12 +155,12 @@@ def distutils_scheme(dist_name, user=Fa
      # NOTE: setting user or home has the side-effect of creating the home dir
      # or user base for installations during finalize_options()
      # ideally, we'd prefer a scheme class that has no side-effects.
 -    assert not (user and prefix), "user={0} prefix={1}".format(user, prefix)
 -    assert not (home and prefix), "home={0} prefix={1}".format(home, prefix)
 -    assert not (home and user), "home={0} user={1}".format(home, user)
 +    assert not (user and prefix), "user={} prefix={}".format(user, prefix)
-     i.user = user or i.user
-     if user:
++    assert not (home and prefix), "home={} prefix={}".format(home, prefix)
++    assert not (home and user), "home={} user={}".format(home, user)
+     if user or home:
          i.prefix = ""
+     i.user = user or i.user
      i.prefix = prefix or i.prefix
      i.home = home or i.home
      i.root = root or i.root

pradyunsg · 2018-08-13T18:06:31Z

I think I did that right.

@techtonik Could you take a look to see if the merge is indeed correct?

techtonik · 2018-09-03T08:54:18Z

I will try to rebase, but if anybody can explain the role ~/.pydistutils.cfg and how it is used for tests - that will help greatly. I don't want to leave "I don't know what I am doing" for commit message. :D

RonnyPfannschmidt · 2018-09-03T08:56:49Z

tests/functional/test_install_reqs.py

@@ -10,6 +10,12 @@
 from tests.lib.local_repos import local_checkout


+if os.name == 'posix':


this is a toxic destructive test - if ever any user with a preconfigured configfile runs it, they will face a lost configuration, and potentially hours or days of figuring why something they knew worked is suddenly broken

I just copied that from the test below to avoid code duplication in many places. I have no idea how this config file is supposed to be used.

The code in current master does the same.

pip/tests/functional/test_install_reqs.py

Lines 232 to 236 in 404838a

if os.name == 'posix':

user_filename = ".pydistutils.cfg"

else:

user_filename = "pydistutils.cfg"

user_cfg = os.path.join(os.path.expanduser('~'), user_filename)

So yes, it points to user HOME, but I thought that test framework should mock that, no?

@RonnyPfannschmidt so any ideas how to change that?

This is fine since we have an isolate fixture to isolate the tests from the actual system, which is used when virtualenv fixture is used.

AFAIK, by the time we isolate, it's too late, since this is done during collection.

Ah right. The execution during collection can be deferred using a function then, with the appropriate comment on why it's a function.

pradyunsg · 2018-09-18T14:21:03Z

anybody can explain the role ~/.pydistutils.cfg

It's basically a configuration file to pass options to distutils (i.e. setup.py considers those values, which can be overidden by CLI options to setup.py that pip does not pass) -- https://docs.python.org/3/install/index.html#distutils-configuration-files

lorencarvalho · 2018-09-18T14:26:52Z

In case anyone is curious how pydistutils.cfg (and it's various permutations) gets discovered by distutils, the pertinent bits are here:
https://github.com/python/cpython/blob/8cf4b34b3665b8bb39ea7111e6b5c3410899d3e4/Lib/distutils/dist.py#L333-L363

of note is the line:

    The file in the user's home directory can be disabled with the
   --no-user-cfg option.

which actually only applies to source distributions, doesn't apply to wheels (since this code was written before wheels existed)

pradyunsg · 2018-09-30T19:43:06Z

Sorry for the lack of a heads-up.

I'm dropping this PR from the 18.1 milestone since there seem to be changes still needed here; albeit minor ones.

BrownTruck · 2018-10-30T04:45:07Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

jaraco · 2018-11-12T13:49:28Z

This PR is a really important one. Pip is currently broken for a very basic use-case (#4106), among others. I should think this would be a blocker, but instead, now two contributors have been burned out trying to fix it. Perhaps one of the project's maintainers would consider adopting this PR and push it over the line, enacting whatever changes are necessary to make it stable and satisfactory?

pfmoore · 2018-11-12T14:17:55Z

@jaraco CI needs fixing - it appears to simply be a lint fix, so that shouldn't be hard. After that, it's more a matter of fixing review comments and getting someone to care enough.

There seems to be some debate over the tests, which I don't really understand but I think needs addressing before this can be merged (or at least @pradyunsg and @RonnyPfannschmidt need to confirm that they are now happy with the test.

As to moving this forward, I stated my own position above - I have no issues with this, but I'm not sufficiently convinced that this change is important to be willing to merge it without being sure that the various concerns others have raised have been addressed. @techtonik hasn't fixed the issue that was pointed out, he's asked for someone else to suggest a fix, but that hasn't happened. IMO, unless someone cares enough about this PR to (a) explain its importance better, and (b) follow up on review comments. then it'll probably remain unmerged.

#4390 (which AIUI was the trigger for this) is simply saying we should fail more gracefully if --target is specified with other options that don't make sense in conjunction with it. So I'm not clear how pip can be "broken" because of this, it's not like there's any behaviour that should work which doesn't, simply our failure mode for some unsupported cases is suboptimal.

#4106, to which you refer, seems to be about not being able to use --target if you have an incompatible option (--user, presumably, although it's not confirmed in the issue discussion) set in global config. This doesn't fix that, nor is there any agreed way forward which would fix that (other than "don't set things in your global config that are incompatible with the options you want to use"). A resolution would be nice, for the people affected by it, but it's likely to need a pretty comprehensive overhaul of the option system, which wasn't really designed with people setting fundamental behaviour-altering flags (like --user) in global config, in mind.

Note - I'm entirely happy if @techtonik (or anyone else) says they haven't got the interest to push this forward. I'm not demanding anything here, quite the opposite, I'm saying that no-one feels the problem is important enough to be worth getting stressed about. (Or more accurately, no-one feels this solution is that important, and no-one has yet stepped up to contribute on the more fundamental issue).

jaraco · 2018-11-12T14:31:17Z

This [pr] doesn't fix #4106.

In that case, since #4106 is the main issue that I'm concerned about, perhaps that issue should be re-opened and unbundled from #4390.

I'll provide more detail in #4106 to describe how this isn't about --user and it's implicated in a very vanilla install.

Bo98 · 2018-11-12T20:03:50Z

One thing this looks like it might fix is:
http://stackoverflow.com/questions/24257803/distutilsoptionerror-must-supply-either-home-or-prefix-exec-prefix-not-both for Homebrew Python.

jaraco · 2018-11-12T21:26:07Z

@Bo98: There are several workarounds in that SO post, none of which are really robust as each requires mutating the user's file system in ways that aren't reliably available. For instance, writing to ./setup.cfg or ~/.distutils.cfg (aka ~/distutils.cfg on Windows) could overwrite existing values in those files. Plus, these workarounds are not properly coupled with the issue (they're user-global or project-local, whereas the cause is proximate to the Python environment). None of those is a proper fix.

lorencarvalho · 2018-11-12T21:26:53Z

The rudimentary workaround mentioned above (and formerly suggested by the Homebrew maintainers) works but is not a good long term solution due to the fact that modifying a global configuration file is fraught with issues. For example, if pip were to modify that file (even temporarily) other concurrently running processes could have their configuration unexpectedly changed out from underneath them. A good example of this can be found in linkedin/shiv#49

Bo98 · 2018-11-12T23:05:35Z

@jaraco Exactly - I meant that this PR has the proper fix. @pfmoore wanted an example of where this solution is needed.

I was more referring to the problem rather than the workarounds.

pradyunsg · 2019-05-23T04:35:31Z

Thanks for this PR and the discussion around it!

I'm closing this PR since it's bitrotten. Please file a new PR if there's still interest in this.

pradyunsg added the type: enhancement Improvements to functionality label Jun 27, 2017

pradyunsg reviewed Aug 26, 2017

View reviewed changes

pypa-bot added needs rebase or merge PR has conflicts with current master and removed needs rebase or merge PR has conflicts with current master labels Aug 26, 2017

pradyunsg reviewed Aug 26, 2017

View reviewed changes

BrownTruck added the needs rebase or merge PR has conflicts with current master label Aug 31, 2017

halmi and others added 12 commits September 14, 2017 10:25

Make --target, --user, and --prefix mutually exclusive.

5232fbd

Also override defaults set somewhere (e.g. .pydistutils.cfg) when command line switches are given.

Add news entry.

0b1d546

Fix CI test errors.

8c01518

Replace ``--no-use-wheel`` with ``--no-binary=:all:``

Fix more unit tests

5cba275

Replace --no-use-wheel with --no-binary=:all: and remove assertion of deprecation message.

Fix too long line.

cf8e1a1

Fixing build failure with common_wheels

6c30bdd

https://travis-ci.org/pypa/pip/jobs/244001886#L244

Fixing test that still fails

fd8aa04

Looks like `--target` doesn't use packages installed in parent environment, so there is no sense to install `wheel` in parent https://travis-ci.org/pypa/pip/jobs/268573000#L547

Still fixing wheel failure

73df058

It is not installing wheel. Either `pip` need to build wheel, which is impossible without `wheel` package, or it can not install wheels without `wheel` https://travis-ci.org/pypa/pip/jobs/268577126#L547

Cleanup

c3bd8ac

Remove more unnecessary code

a6c1638

These lines were necessary

5333b78

"Can not perform a '--user' install. User site-packages are not visible in this virtualenv." https://travis-ci.org/pypa/pip/jobs/268627056#L609

Rephrase news comment

4d970ab

techtonik force-pushed the pr/4111 branch from f68aa44 to 4d970ab Compare September 14, 2017 07:26

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 14, 2017

BrownTruck added the needs rebase or merge PR has conflicts with current master label Dec 16, 2017

This was referenced May 5, 2018

distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both linkedin/shiv#16

Closed

Write a temporary ~/.pydistutils.cfg to override any system distutils configuration linkedin/shiv#27

Merged

pradyunsg added this to the 18.1 milestone Aug 5, 2018

Merge branch 'master' into pr/4111

8e58aca

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Aug 13, 2018

rouge8 mentioned this pull request Aug 14, 2018

Writing ~/.pydistutils.cfg affects parallel builds. linkedin/shiv#49

Closed

RonnyPfannschmidt reviewed Sep 3, 2018

View reviewed changes

pradyunsg removed this from the 18.1 milestone Sep 30, 2018

pradyunsg mentioned this pull request Sep 30, 2018

Release 18.1 #5782

Closed

BrownTruck added the needs rebase or merge PR has conflicts with current master label Oct 30, 2018

pradyunsg closed this May 23, 2019

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 22, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 22, 2019

		@@ -0,0 +1 @@
		Fix ``pip install --target`` by making ``--target``, ``--user``, and ``--prefix`` mutually exclusive.

		@@ -10,6 +10,12 @@
		from tests.lib.local_repos import local_checkout


		if os.name == 'posix':

	if os.name == 'posix':
	user_filename = ".pydistutils.cfg"
	else:
	user_filename = "pydistutils.cfg"
	user_cfg = os.path.join(os.path.expanduser('~'), user_filename)

Mutually exclusive --target, --user and --prefix #4557

Mutually exclusive --target, --user and --prefix #4557

Conversation

techtonik commented Jun 17, 2017 • edited by pradyunsg Loading

pradyunsg commented Jun 21, 2017

techtonik commented Jun 22, 2017

dpkp commented Jun 24, 2017 • edited by pradyunsg Loading

pradyunsg commented Jun 27, 2017 • edited Loading

Bo98 commented Aug 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pypa-bot commented Aug 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

techtonik commented Aug 26, 2017

pradyunsg commented Aug 26, 2017 • edited Loading

techtonik commented Aug 26, 2017

BrownTruck commented Aug 31, 2017

techtonik commented Dec 31, 2017

dpkp commented Feb 6, 2018 • edited by pradyunsg Loading

pradyunsg commented Aug 13, 2018

techtonik commented Sep 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg Sep 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradyunsg commented Sep 18, 2018

lorencarvalho commented Sep 18, 2018 • edited Loading

pradyunsg commented Sep 30, 2018

BrownTruck commented Oct 30, 2018

jaraco commented Nov 12, 2018

pfmoore commented Nov 12, 2018

jaraco commented Nov 12, 2018

Bo98 commented Nov 12, 2018

jaraco commented Nov 12, 2018

lorencarvalho commented Nov 12, 2018

Bo98 commented Nov 12, 2018 • edited Loading

pradyunsg commented May 23, 2019

techtonik commented Jun 17, 2017 •

edited by pradyunsg

Loading

dpkp commented Jun 24, 2017 •

edited by pradyunsg

Loading

pradyunsg commented Jun 27, 2017 •

edited

Loading

pradyunsg commented Aug 26, 2017 •

edited

Loading

dpkp commented Feb 6, 2018 •

edited by pradyunsg

Loading

pradyunsg Sep 18, 2018 •

edited

Loading

lorencarvalho commented Sep 18, 2018 •

edited

Loading

Bo98 commented Nov 12, 2018 •

edited

Loading