Issue #2140: Build wheels during pip install #2618

rbtcollins · 2015-03-30T04:17:01Z

This branch is building up to tackle issue #2140 - build wheels during install and then install those wheels.

rbtcollins · 2015-03-30T04:17:33Z

The topo sort thuings are from pull #2616

sigmavirus24 · 2015-03-30T22:07:30Z

This sounds exciting

alex · 2015-03-31T03:05:00Z

Can all the ensure_dir changes by refactored out into their own PR?

rbtcollins · 2015-03-31T03:40:53Z

They're a single commit. Can pull it out if desired.

rbtcollins · 2015-04-01T01:48:38Z

Its mainly tested via all the existing tests, but I added an explicit test looking at the output to be sure building was happening and handling of non-wheelable things.

rbtcollins · 2015-04-03T01:53:22Z

I'm not sure why we're seeing those failures. Its not as simple as hashseed - I've tried using the same hashseed locally. It might be an isolation failure, or something. There's obviously an underlying issue, but until I can provoke it I can't fix it :(

I'm going to try adding diagnostic changes on top of the stack to do that.

berkerpeksag · 2015-04-03T09:44:45Z

pip/utils/__init__.py

+    try:
+        os.makedirs(path)
+    except OSError as e:
+        if e.errno == errno.EEXIST:


or a shorter version:

if e.errno != errno.EEXIST: raise

rbtcollins · 2015-04-10T03:44:33Z

This now has a limit on it to not build or cache wheels for unversioned things. E.g. unless the url looks like an archive, we won't cache for it at all, which means that e.g. git+https://github.com/pypa/pip will not get cached wheels built. But a url like https://github.com/pypa/project-2 will. This isn't perfect, but it should be enough to avoid the vast majority of potential breakage.

rbtcollins · 2015-04-13T06:32:57Z

  self.do_handshake()
File "/opt/python/3.2.5/lib/python3.2/ssl.py", line 451, in do_handshake
  self._sslobj.do_handshake()

ssl.SSLError: [Errno 8] _ssl.c:392: EOF occurred in violation of protocol

network failures.

This is needed for determining the wheel-cachability of a requirement.

This won't put wheels into that directory, but will read them if they are there. --no-cache-dir will disable reading such wheels.

Building wheels before installing elminates a cause of broken environments - where install fails after we've already installed one or more packages. If a package fails to wheel, we run setup.py install as normally.

xavfernandez · 2015-04-13T08:50:40Z

pip/commands/install.py

+                            requirement_set,
+                            finder,
+                            build_options=[],
+                            global_options=[],


Wondering if this is not an issue (for build_options and global_options)... maybe the use of these would disable the wheel caching as I dont think we want to build wheels for all possible options ?

dstufft · 2015-04-13T14:00:45Z

I'm going to go ahead and merge this. Some things may need to be adjusted still but I don't think that we're going to fully hammer them out with this sitting in a branch, and having it sit in a branch just makes it harder on @rbtcollins to have to keep updating it. I do think that the overall structure is sound and that the general feature is something we want.

qwcode · 2015-04-13T14:09:20Z

pip/wheel.py

+        if not wheel.supported():
+            # Built for a different python/arch/etc
+            continue
+        candidates.append((wheel.support_index_min(), wheel_name))


I was specifically looking for this sorting by the support index. yay, it's here.

Issue #2140: Build wheels during pip install

qwcode · 2015-04-13T14:14:18Z

to be clear, I don't think we could release without the blacklist, so I assume that's coming.

dstufft · 2015-04-13T14:16:06Z

Added #2675 to track that.

xavfernandez · 2015-04-13T14:21:32Z

The *_options issue seems important also:

pip install foo --install-option some_option

will end up building the wheel and installing from it without the passed option...

dstufft · 2015-04-13T14:22:36Z

So, that's actually already the case if you're installing from Wheels, they just get flat out ignored. We should probably figure something out to deal with them though.

dstufft · 2015-04-13T14:27:47Z

Also added #2676 and #2677

nils-werner · 2015-06-03T11:12:25Z

Is there a way to make pip create wheels and store them in the cache dir without actually installing them? I would like to fill my wheel cache from time to time too...

rbtcollins force-pushed the issue-2140 branch from d73c84f to aec03e0 Compare March 30, 2015 21:58

rbtcollins force-pushed the issue-2140 branch 4 times, most recently from 2e35a5c to 4c89e07 Compare March 31, 2015 02:39

rbtcollins force-pushed the issue-2140 branch 4 times, most recently from 758b819 to 82aba41 Compare April 1, 2015 01:47

rbtcollins force-pushed the issue-2140 branch 11 times, most recently from c2c03a1 to affd2bf Compare April 2, 2015 07:47

rbtcollins force-pushed the issue-2140 branch 2 times, most recently from c08a7f0 to e93b9ed Compare April 3, 2015 02:50

berkerpeksag reviewed Apr 3, 2015
View reviewed changes

rbtcollins force-pushed the issue-2140 branch from e93b9ed to b3229d8 Compare April 6, 2015 23:12

rbtcollins force-pushed the issue-2140 branch from c8bcea2 to 8a8903e Compare April 10, 2015 03:15

rbtcollins force-pushed the issue-2140 branch 3 times, most recently from 869a222 to 4a25059 Compare April 13, 2015 02:24

rbtcollins added 3 commits April 13, 2015 18:33

Break out version extraction for reuse.

786ff8c

This is needed for determining the wheel-cachability of a requirement.

Issue pypa#2563: Read cached wheels from ~/.cache/pip

4926409

This won't put wheels into that directory, but will read them if they are there. --no-cache-dir will disable reading such wheels.

Issue pypa#2140: Build wheels automatically

08acb66

Building wheels before installing elminates a cause of broken environments - where install fails after we've already installed one or more packages. If a package fails to wheel, we run setup.py install as normally.

rbtcollins force-pushed the issue-2140 branch from 4a25059 to 08acb66 Compare April 13, 2015 06:33

xavfernandez reviewed Apr 13, 2015
View reviewed changes

qwcode reviewed Apr 13, 2015
View reviewed changes

dstufft added a commit that referenced this pull request Apr 13, 2015

Merge pull request #2618 from rbtcollins/issue-2140

d043e4b

Issue #2140: Build wheels during pip install

dstufft merged commit d043e4b into pypa:develop Apr 13, 2015

This was referenced Apr 13, 2015

Determine if sdist name is "unique" enough for wheel cache key #2676

Closed

Determine what should be done about --(install|global)-option with Wheels #2677

Open

rbtcollins mentioned this pull request Apr 16, 2015

use ~/.cache/pip/wheels by default #2563

Closed

rbtcollins deleted the issue-2140 branch April 22, 2015 20:44

kreativitea mentioned this pull request Jun 3, 2015

Wheel binary format paylogic/pip-accel#3

Open

pombredanne mentioned this pull request Jun 4, 2015

All packages that contain non-package data are now likely installed in a broken way since 7.0.0 #2874

Closed

philtay mentioned this pull request Nov 16, 2015

pip freeze and wheel package #3250

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 4, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #2140: Build wheels during pip install #2618

Issue #2140: Build wheels during pip install #2618

rbtcollins commented Mar 30, 2015

rbtcollins commented Mar 30, 2015

sigmavirus24 commented Mar 30, 2015

alex commented Mar 31, 2015

rbtcollins commented Mar 31, 2015

rbtcollins commented Apr 1, 2015

rbtcollins commented Apr 3, 2015

berkerpeksag Apr 3, 2015

rbtcollins commented Apr 10, 2015

rbtcollins commented Apr 13, 2015

xavfernandez Apr 13, 2015

dstufft commented Apr 13, 2015

qwcode Apr 13, 2015

qwcode commented Apr 13, 2015

dstufft commented Apr 13, 2015

xavfernandez commented Apr 13, 2015

dstufft commented Apr 13, 2015

dstufft commented Apr 13, 2015

nils-werner commented Jun 3, 2015

Issue #2140: Build wheels during pip install #2618

Issue #2140: Build wheels during pip install #2618

Conversation

rbtcollins commented Mar 30, 2015

rbtcollins commented Mar 30, 2015

sigmavirus24 commented Mar 30, 2015

alex commented Mar 31, 2015

rbtcollins commented Mar 31, 2015

rbtcollins commented Apr 1, 2015

rbtcollins commented Apr 3, 2015

berkerpeksag Apr 3, 2015

Choose a reason for hiding this comment

rbtcollins commented Apr 10, 2015

rbtcollins commented Apr 13, 2015

xavfernandez Apr 13, 2015

Choose a reason for hiding this comment

dstufft commented Apr 13, 2015

qwcode Apr 13, 2015

Choose a reason for hiding this comment

qwcode commented Apr 13, 2015

dstufft commented Apr 13, 2015

xavfernandez commented Apr 13, 2015

dstufft commented Apr 13, 2015

dstufft commented Apr 13, 2015

nils-werner commented Jun 3, 2015