Upgrading to distribute 0.7 (which depends on the new setuptools 0.7) ends in tears

[inducer]

This just happened:

$ pip install --upgrade distribute
Downloading/unpacking distribute from https://pypi.python.org/packages/source/d/distribute/distribute-0.7.zip#md5=93a7dea3c59cf25aa12d0026840f10ba
  Running setup.py egg_info for package distribute

Downloading/unpacking setuptools>=0.7 (from distribute)
  Running setup.py egg_info for package setuptools

Installing collected packages: distribute, setuptools
  Found existing installation: distribute 0.6.45
    Uninstalling distribute:
      Successfully uninstalled distribute
  Running setup.py install for distribute
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named setuptools
    Complete output from command /var/lib/jenkins/jobs/pytential/workspace/cl_dev/amd_a10/node/master/.env/bin/python -c "import setuptools;__file__='/var/lib/jenkins/jobs/pytential/workspace/cl_dev/amd_a10/node/master/.env/build/distribute/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-KbHbEQ-record/install-record.txt --single-version-externally-managed --install-headers /var/lib/jenkins/jobs/pytential/workspace/cl_dev/amd_a10/node/master/.env/include/site/python2.7:
    Traceback (most recent call last):

  File "<string>", line 1, in <module>

ImportError: No module named setuptools

To make this more precise: I get that "upgrading" distribute means uninstalling it and installing setuptools. I just wish that process wouldn't fail.

[qwcode]

the distribute project had released a new "wrapper" version of distribute that does nothing but depend on the new setuptools (the merge of setuptools and distribute).

they've since taken it down for now.
given pip's dependency on setuptools and it's install logic, pip won't be able to perform this upgrade.
distribute upgrades have only been working in python 2 anyway up until now (not python 3; #650)

decision pending on what pip's position will be on supporting upgrading to the new distribute....

[qwcode]

with distribute-0.6.X upgrades, even though pip uninstalled distribute as part of the upgrade, it ran the install subprocess with the cwd equal to the build dir, so it could import setuptools from the unpacked download.

but now, distribute-0.7 is just a shell (that depends on setuptools), that contains no importable setuptools that pip can use (and by default, setuptools gets queued to be installed after distribute)

here's a thought on adding some special logic to make this work:
(will work this out in more detail, as the work actually begins)

when an upgrade to distribute==0.7 is detected:

1. force the "setuptools" dependency to be processed before distribute itself.
2. force pip to perceive "distribute" as a conflict to setuptools (thereby forcing an uninstall of distribute first)

so the order would end up being:

• uninstall distribute-0.6.X
• install setuptools-0.7 (making use of the setuptools build dir to import setuptools)
• install distribute-0.7

cc @jaraco, @pfmoore

[jezdez]

If possible we should ship this with 1.3.x, too.

[pfmoore]

Sounds like a reasonable plan, although I can't get up any enthusiasm for it. I suspect it would still fail for Python 3, because of the unrelated 2to3 requirement/issue. That problem will go away once setuptools removes the requirement for 2to3, but I assume that won't be before 0.8. Actually, you say "when an upgrade to distribute==0.7 is detected" - should that not be ">= 0.7"? I wouldn't want to force someone upgrading to a single-source 0.8 to have to go via the 2to3 0.7...

We should keep the relevant code isolated so that it can be removed again once use of distribute has diminished to the point where it's not an issue - we don't want to have to maintain this indefinitely.

[jaraco]

That's right - a unified Python 2/3 code base won't be before 0.8, although I hope to make that release sooner than later. Don't let the number increment suggest it's anything but top priority following the merge.

And yes, I believe any distribute>=0.7 is correct. I do have plans to release 0.7.1 which will reference setuptools on pypi (removing dependency_links to get setuptools from bitbucket).

[pfmoore]

One other thought. pip 1.4 will support installation from wheels. If setuptools 0.7 is available in wheel format on PyPI, we could simply automatically force wheel installation for setuptools, and bypass the whole problem. That may be less work, and would likely involve less "hacky" code, for what is essentially an interim solution.

@qwcode, is that a possible alternative? @jaraco - it would mean hosting wheels for setuptools, is that viable?

Obviously, if we release a 1.3.x that needs to fix this issue, then this idea is a non-starter.

[qwcode]

ok, my plan is not good enough since the code is not 2&3 compatible yet. sorry, I mistakenly thought it would be. going to the trouble to fix this just for python2 is no good IMO.

4 other thoughts:

1. paul's idea of upgrading from wheels (it would require pip install --use-wheel, or installing directly from the wheel url).
2. @regebro mentioned on distutils-sig including a working 2&3 compatible source tree (if it's close to ready) that doesn't get installed.
3. "unrelease" setuptools 0.7 from pypi, and push for the 2&3 version before any more big announcements; which would mean we could use my original plan if needed
4. in addition to my plan, figure out how to also hack around the current 2to3 egg_info failure we get now in python3.

[qwcode]

for #4, we just need 2to3 to run before we try to run egg_info in python3.
shouldn't that be easy to fix? What am I missing?

[jaraco]

I'll have some time on Thursday. I'll look into merging the code patches to provide native Python 3 support and provide a report back on how close that would be.

Another option: the distribute shell package could include both the Python 2 and 2to3-converted Python 3 code plus wrapper packages/modules ('pkg_resources', 'setuptools') that selectively import the proper code. It would be hacky, but I'd be okay with that in the shell package.

As for wheel support, I'm still largely naive about wheels, so I would need to learn more about them. My suspicion, however, is that migrating all existing clients to wheels is a more challenging problem fraught with even more compatibility struggles. My goal with setuptools 0.7 is to try to remain as compatible as possible and focus primarily on migrating users from distribute back to a single project for setuptools.

[regebro]

Having two source trees is probably a good solution in this case. It makes the package big, but the install would be quicker. First running 2to3 on the distribute source and then not install it, and then install setuptools and run 2to3 on that, would be annoying and slow.

[qwcode]

to be clear, the idea of including a 2&3 importable setuptools in the distribute shell, will still leave pip failing in python3 when we get to installing the new setuptools, which is still not 2&3.

It's 2 problems:

1. needing an importable setuptools, when installing the distribute shell
2. a solution in python3 for pip and it's 2to3 egg_info failure, when installing the new setuptools

for #1, either

• jason's hack import idea he mentioned
• the hack I mentioned originally

for #2, either

• waiting on jason to get the 2&3 changes merged for a 0.8 release of setuptools
• like I mentioned, some solution that gets 2to3 run before egg_info is called. this only needs to happen for setuptools (assuming one of the 2 solutions for problem No instructions for how to install pip #1), so just one execution of 2to3, not two.

[qwcode]

@jaraco I started working on a pip-only solution for this.

one thing I can't fathom is that you can't run "python setup.py egg_info" more than once for distribute-0.7 without it getting confused somehow, and being unable to locate egg_info

qwcode@qwcode:~/downloads/distribute-0.7$ rm -rf distribute.egg-info/
qwcode@qwcode:~/downloads/distribute-0.7$ python setup.py egg_info
running egg_info
creating distribute.egg-info
writing requirements to distribute.egg-info/requires.txt
writing distribute.egg-info/PKG-INFO
writing top-level names to distribute.egg-info/top_level.txt
writing dependency_links to distribute.egg-info/dependency_links.txt
writing manifest file 'distribute.egg-info/SOURCES.txt'
reading manifest file 'distribute.egg-info/SOURCES.txt'
writing manifest file 'distribute.egg-info/SOURCES.txt'
qwcode@qwcode:~/downloads/distribute-0.7$ python setup.py egg_info
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'zip_safe'
  warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'egg_info'

[regebro]

Where is the distribute source now? Tareks repo does't contain the 0.7 attempts as far as I can tell, and none of the pypa repos seem to have a version of Distribute.

The reason I ask is that I wondered if it was possible to make a Distribute 0.7 that simply did not use setuptools at all, and just distutils. I don't know, but it might be worth a try.

[pfmoore]

A thought. For the problem of needing an importable setuptools, there is a 3rd option (I think):

• Vendor in our own copy of pkg_resources
• Special-case the install command for setuptools/distribute to not go through the dance to force using setuptools, but just do python setup.py.

[qwcode]

@regebro I got my distribute source archive from here: https://bitbucket.org/pypa/setuptools/downloads

@pfmoore "just do python setup.py". that's the gist of the solution I'm working on now. it's working, but I need help understanding, why I can't run "egg_info" more than once for the distribute-0.7 source.

[qwcode]

btw, the "can't run egg_info more than twice problem" seems to be due to the empty distribute in the cwd blocking the installed distribute, once the egg_info is built, hence it can't find/use the egg_info plugin

[qwcode]

another plan, based on IRC conversations with @jezdez and @dstufft

override distribute upgrades (to >=0.7) as an uninstall-distribute/install-setuptools operation, i.e. don't ever install distribute>=0.7

this combined with the 2to3/python3 fix I have working, would be a pip-only solution.

working on this now...

[qwcode]

posted a draft of a pip-only solution for this in #992

[jaraco]

@regebro: all the code for distribute and setuptools is now maintained in named branches of /pypa/setuptools. The Distribute 0.7 shell can be found in the 'distribute-legacy' branch. I realize in retrospect that name isn't the best name, but it's what we have.

I've considered a distutils-only package, except that a distutils-only package can't "require" setuptools. Or can it? I was thinking the setup.py could be updated to fallback to distutils (when setuptools isn't available), but because the package metadata would have been built with setuptools/distribute, it would contain the requirement and PIP would have installed it.

[jaraco]

@qwcode: I believe the reason that egg_info only runs once in that environment is because when your run egg_info once, that creates distribute.egg-info, superseding your installation of distribute on which you're relying to run egg_info. Because Distribute 0.7 doesn't implement any distutils commands, once distribute-0.7 egg-info is created, you no longer have setuptools capabilities. Does that explain the behavior?

Thanks for the patch on pip to address the issue. Since this fix only affects new pip installs, it won't address the issue for users using a pre-installed version of pip, but that's possibly an acceptable compromise. It at least provides a path to upgrade.

I'm tempted to accept this approach and then focus on the unified Python 2/3 codebase work, rather than spend time bundling versions into Distribute 0.7.

[jaraco]

@regebro: I remember now why falling back to distutils in Distribute 0.7 is not viable. It's because the error is triggered before the Distribute 0.7 setup.py is invoked. Pip invokes 'python -c import setuptools' itself, and that's where the error occurs. That detail isn't clear from the error as reported above, but it is more clear in the initial reports on distutils-sig.

[qwcode]

superseding your installation of distribute on which you're relying to run egg_info

that's what I surmised as well after some head scratching. my patch resorts to deleting it before the run of egg_info

won't address the issue for users using a pre-installed version of pip, but that's possibly an acceptable compromise

I think it's acceptable. I think pip/virtualenv needs to start stating what version of setuptools it's been tested with anyway, and not promote users eagerly upgrading.

[qwcode]

solution for pip-1.4 merged in #992

[jaraco]

I've made some progress on a distribute 0.7 that will work with older versions of pip, but it still has problems:

vagrant@precise64:~$ virtualenv upg
New python executable in upg/bin/python2.7
Also creating executable in upg/bin/python
Installing distribute...........................................................................................................................................................................................................................done.
Installing pip................done.
vagrant@precise64:~$ upg/bin/pip --version
pip 1.3.1 from /home/vagrant/upg/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg (python 2.7)
vagrant@precise64:~$ upg/bin/pip install https://bitbucket.org/pypa/setuptools/downloads/distribute-0.7.3.zip
Downloading/unpacking https://bitbucket.org/pypa/setuptools/downloads/distribute-0.7.3.zip
  Downloading distribute-0.7.3.zip (145kB): 145kB downloaded
  Running setup.py egg_info for package from https://bitbucket.org/pypa/setuptools/downloads/distribute-0.7.3.zip

Downloading/unpacking setuptools>=0.7 (from distribute==0.7.3)
  Downloading setuptools-0.7.4.tar.gz (755kB): 755kB downloaded
  Running setup.py egg_info for package setuptools

Installing collected packages: setuptools, distribute
  Running setup.py install for setuptools

    Installing easy_install script to /home/vagrant/upg/bin
    Installing easy_install-2.7 script to /home/vagrant/upg/bin
  Found existing installation: distribute 0.6.34
    Uninstalling distribute:
      Successfully uninstalled distribute
  Running setup.py install for distribute

Successfully installed setuptools distribute
Cleaning up...
vagrant@precise64:~$ upg/bin/python
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources.require('setuptools')
[setuptools 0.7.4 (/home/vagrant/upg/lib/python2.7/site-packages)]
vagrant@precise64:~$ upg/bin/easy_install --help
-bash: upg/bin/easy_install: No such file or directory

Although the process runs cleanly and the latest setuptools is installed, the easy_install scripts aren't present (presumably because they were removed by the uninstallation of distribute).

While this technique is nicer than that of distribute 0.7/0.7.1, I believe the recommended procedure will be to upgrade pip first.

[qwcode]

what's happening here is setuptools is installed on top of distribute.
one my hacks was to force setuptools to think of distribute as a conflict, and uninstall it first.

yes, upgrading pip first, will have to be good enough.

[tseaver]

I can't reproduce that problem when updating to distribute 0.7.3 via:

$ /path/to/virtualenv --distribute /tmp/distrib
$ /tmp/distrib/bin/pip install \
  --find-links=https://bitbucket.org/pypa/setuptools/downloads \
  --upgrade distribute

That works for me on both Python 2 and 3.

I don't know whether users for whom that broke could have used
easy_install instead to work around the pip problem. The following works
for me in a fresh distribute-based virtualenv (Python 2 or 3):

$ /path/to/virtualenv --distribute /tmp/distrib2
$ /tmp/distrib2/bin/easy_install \
  --find-links=https://bitbucket.org/pypa/setuptools/downloads \
  -U distribute

AFAICT, we should just push distribute 0.7.3 to the cheeseshop and be done.

[qwcode]

@tseaver , to be clear, 0.7.3, improves the situation by including the setuptools source. the original report from the user was valid. as for the discrepancy between jason and you using 0.7.3, it has to do with the order that pip attempts to install things. pip is normally agnostic to order, but in this one case it matters, and jason's direct install command doesn't result in the "right" order.