Pip isn't accounting for Setuptool's requirement extras

[vbabiy]

Pip's requirements parsing doesn't appear to recognize declared extras.[0] These are supposed to be used in Setuptools to provide a way to list dependencies for a particular feature.

Repeatable through the following:

$ pip install "zope.testbrowser [zope_functional_testing]"

In the above 'zope.app.testing' should have been pulled in as a dependency, because of the extra declaration in the requirements line.

Another example would be "zope.i18n [zcml]".

[0] http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies

---

• Bitbucket: https://bitbucket.org/ianb/pip/issue/170
• Originally Reported By: Michael Mulich
• Originally Created At: 2010-09-27 22:20:34

[vbabiy]

Hmm, no comment?

Is supporting this planned? Is its missing considered a bug? Is there some
other way?

---

Original Comment By: Thomas Waldmann

[vbabiy]

This is also an issue for me. I started digging into the pip code a little
bit, but I'm not sure what the correct fix is.

---

Original Comment By: Dan Sully

[vbabiy]

maybe this will do it:
https://bitbucket.org/MattMaker/pip/changeset/9144e997b918

---

Original Comment By: MattMaker

[MatthewMaker]

Made some adjustments to my proposed patch, see see https://bitbucket.org/MattMaker/pip/changeset/4a967b7de7ad and https://bitbucket.org/MattMaker/pip/changeset/9c681d36b768 . Now it works with uninstalling too! (oops :) )

[dsully]

Matthew - can you fork the new pip repo on GitHub and send a pull request for your changes?

Thanks

[MatthewMaker]

forked, retested, and pull requested!

[carljm]

Fixed. Thanks @MatthewMaker for the patch, and @ssaboum for adding tests.

[poswald]

Just a quick question about this... I don't think it is documented at all and I think it should be, probably in the requirements.txt section. My question is.. does this work in all places where you could use install before? I usually tested things out locally by running pip install . while in the working directory of my project.

I added this to my setup.py:

extras_require = {
    'django':  ['Django<1.4',],
}

... but now I don't see how I could install it. I tried pip install ". [django]" with no luck. Is this format supported? Is putting something like my project [django] in a requirements.txt supported? Should I open this as a new issue?

[carljm]

@poswald Thanks for noting the lack of documentation - if you'd be willing to open an issue for that (or better, a pull request!) that'd be great.

As for the format, using myproject[extra] either from commandline or requirements file should certainly work. And from my reading of InstallRequirement.from_file and InstallRequirement.__init__ (in pip/reqs.py), it appears that extras can also be referenced from the #egg= fragment of a URL requirement (e.g. http://somesite/somepackage.tgz#egg=somepackage[extra]), though I haven't tested this. And this should also work with a local file:// URL to a directory, which might cover your use case. In any case, some experimentation here and documentation of what you find would definitely be helpful. If there are obvious gaps in the functionality, filing bugs for them would make sense (though personally I think supporting file:// URLs with an #egg= fragment is adequate, we don't need to support .[extra] - that just reads wierd.)

[poswald]

There's the documentation for it in the requirements.txt file. I only use the .[extra] format because it works and I wasn't aware that the file:/// was working.

I'll play around and open any issues if I see it isn't working as expected. Thanks!

[carljm]

Oh - I was under the impression from your previous comment that .[extra] didn't work. If it already works, great :-)

[poswald]

I just didn't know how to properly specify it for a local install to test it because I didn't know how to do file://. I'll report back.

[poswald]

Sorry guys, I may have spoken too soon on this. I thought it was working before but I must have done something wrong as doesn't seem to be. Here I have a project with an extras_require that I'm testing out. I checkout the project and create a virtualenv to try to install it into using the file:// technique:

$ which pip
/usr/local/bin/pip

$ pip --version
pip 1.0.2.post1 from /Users/poswald/tmp/src/pip (python 2.6)

$ git clone git://github.com/poswald/example.git example
Cloning into example...
remote: Counting objects: 348, done.
remote: Compressing objects: 100% (188/188), done.
remote: Total 348 (delta 157), reused 329 (delta 138)
Receiving objects: 100% (348/348), 59.39 KiB, done.
Resolving deltas: 100% (157/157), done.

$ cd example/

$ git checkout packaging
Branch packaging set up to track remote branch packaging from origin.
Switched to a new branch 'packaging'

$ pip freeze
distribute==0.6.14
virtualenv==1.6.1
wsgiref==0.1.2

$ virtualenv --distribute --no-site-packages .
New python executable in ./bin/python
Installing distribute....................................................................................................................................................................................done.
Installing pip...............done.

$ source bin/activate

(example)$ pip install "file:///Users/poswald/tmp/example [django]"
Unpacking /Users/poswald/tmp/example [django]
Exception:
Traceback (most recent call last):
  File "/Users/poswald/tmp/example/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/basecommand.py", line 126, in main
    self.run(options, args)
  File "/Users/poswald/tmp/example/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/commands/install.py", line 223, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/Users/poswald/tmp/example/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 961, in prepare_files
    self.unpack_url(url, location, self.is_download)
  File "/Users/poswald/tmp/example/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 1075, in unpack_url
    return unpack_file_url(link, location)
  File "/Users/poswald/tmp/example/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/download.py", line 305, in unpack_file_url
    unpack_file(source, location, content_type, link)
  File "/Users/poswald/tmp/example/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg/pip/util.py", line 473, in unpack_file
    or tarfile.is_tarfile(filename)
  File "/usr/local/Cellar/python/2.6.4/lib/python2.6/tarfile.py", line 2523, in is_tarfile
    t = open(name)
  File "/usr/local/Cellar/python/2.6.4/lib/python2.6/tarfile.py", line 1647, in open
    return func(name, "r", fileobj, **kwargs)
  File "/usr/local/Cellar/python/2.6.4/lib/python2.6/tarfile.py", line 1709, in gzopen
    fileobj = bltn_open(name, mode + "b")
IOError: [Errno 2] No such file or directory: '/Users/poswald/tmp/example [django]'

Storing complete log in /Users/poswald/.pip/pip.log

Assuming I did that correctly, it seems that the issue is that the line to unpack the file:// url is called without knowing about the requirements

Is there an equivalent to urllib.parse but for the pip requirements?

Should this be a separate issue or perhaps should this one be reopened?

[carljm]

Whatever needs to be done here, if anything, should be a separate issue. Extras work just fine for the common case (installing "Foo[Bar]" off PyPI) which solves this ticket; I didn't really ever expect it to work for other cases. If it's possible to make it work, that's great, but it's a separate issue.

Based on reading the code, I didn't expect it to work using the syntax you tried, I expect that it should work if you pip install "file:///Users/poswald/tmp/example#egg=example[django]"