-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Idea: Capture expected Python version & artifact names in Pipfile.lock to improve error messages #295
Comments
This is also somewhat related to https://github.com/kennethreitz/pipenv/issues/236 (where |
Hey @ncoghlan, thanks for checking in on this! #236 seems to be an issue with Kenneth and I spent a fair amount of time discussing hashes a few weeks ago and arrived at the recent I am definitely open to ideas that will make the current experience better but here's the pain points I'm seeing in the suggestions provided.
This does allow us to provide a more explicit message to the user in the event of hash conflict, which is nice. It would however restrict the user to only using a given lockfile on one platform and version, which is what we've been trying to avoid. We could use these fields only for the error message, but I'm betting people who are going to struggle with the "Hash Conflict" message won't have taken the time to add the
I could get on board with this one but it has the same problem as above where we're now bound to the wheel from platform X or version X. This defeats the point of a "universal" lockfile. I also can't seem to find anything in Please let me know if you have further thoughts on this though. As I said above, this is design needs some help so the more ideas we have floating around on it, the better. |
Of the two suggestions, with the current "single universal lock file" structure, the one I'd recommend is to track the filename that the hash belongs to. When pip picks up a pre-built non-universal wheel, any environment mismatch is going to cause a hash conflict anyway, and having the filenames available will make for significantly more meaningful error messages. However, I think a more important outcome of that approach would be the fact that tracking the filenames creates the opportunity to switch to a multi-level lockfile scheme, where you take the hashes out of the platform independent version lockfile entirely and move them to a separate environment specific artifact manifest. That is:
When
Depending on whether it was Python 2 or Python 3 that was used to generate Pipfile.lock. While if it had binary dependencies (as with the cffi example above) you might get:
Actually taking that second step of splitting out the artifact manifest from the version lock files would involve talking to @dstufft et al about moving pip's compatibility tag handling and wheel filename parsing out of pip itself and into the Such a model then only fails in the presence of environment markers that result in |
@nateprewitt Thinking about this a bit more, I can see a third way, which I think is a bit crazy, but I can't really tell if there's something really wrong with it, besides weird gut feeling. Assumption: the main use-case of the hashes in the lock file is making sure that the content of the package that I'm pulling right now matches the package author WAS using during development. This can alone provide nice way to asynchronously verify integrity of the given resource. Now let's try changing the highlighted WAS above into WAS OR WOULD HAD BEEN, and instead of recording a digest of single (package, version, environment) tuple record digests for all (package, version, *) matching tuples. So the format would look like:
Ideally we'd combine the digests with filenames as @ncoghlan suggested, so that we are able to figure out all well-known names we're accounting for.
Security wise I think that checking multiple digests is OK, as there's a strong true/false relation in that we either trust {Digest function} or we don't. It's not a perfect solution, but it fits the universality bill pretty nicely. |
For the record, pipenv moved away from |
Question is — how do we get those digests? is there a pypi api for them? |
If so, that'd make |
Going to tackle the first proposal:
I has an idea. |
$ pipenv lock --hashes
Locking [dev-packages] dependencies..
Locking [packages] dependencies...
$ cat Pipfile.lock
{
"_meta": {
"hash": {
"sha256": "26335cbd7567290586d31b410a5b482081449f5e79695916ce411e52d0f4a926"
},
"host-508-specifiers": {
"implementation_name": "cpython",
"implementation_version": "3.6.2",
"os_name": "posix",
"platform_machine": "x86_64",
"platform_python_implementation": "CPython",
"platform_release": "16.7.0",
"platform_system": "Darwin",
"platform_version": "Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64",
"python_full_version": "3.6.2",
"python_version": "3.6",
"sys_platform": "darwin"
},
"pipfile-spec": 1,
"requires": {
"python_version": "3.6"
},
"sources": [
{
"url": "https://pypi.python.org/simple",
"verify_ssl": true
}
]
},
"default": {
"-e .": "*",
"background": {
"hash": "sha256:8f64df9b1f1d4f91603f16d25f79691fa87eff62244d631b8b69cf121396b725",
"version": "==0.1.1"
},
"certifi": {
"hash": "sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704",
"version": "==2017.7.27.1"
},
"chardet": {
"hash": "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
"version": "==3.0.4"
},
"click": {
"hash": "sha256:29f99fc6125fbc931b758dc053b3114e55c77a6e4c6c3a2674a2dc986016381d",
"version": "==6.7"
},
"dateparser": {
"hash": "sha256:e2629d2f8361722c6047138ca085256c9f2cf5cc657fd66122aa0564afa4dc33",
"version": "==0.6.0"
},
"first": {
"hash": "sha256:41d5b64e70507d0c3ca742d68010a76060eea8a3d863e9b5130ab11a4a91aa0e",
"version": "==2.0.1"
},
"futures": {
"hash": "sha256:51ecb45f0add83c806c68e4b06106f90db260585b25ef2abfcda0bd95c0132fd",
"version": "==3.1.1"
},
"humanize": {
"hash": "sha256:a43f57115831ac7c70de098e6ac46ac13be00d69abbf60bdcac251344785bb19",
"version": "==0.5.1"
},
"idna": {
"hash": "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4",
"version": "==2.6"
},
"maya": {
"hash": "sha256:04c5d5380313d9b928746e0f8c1da914d093cd22e0324e48297347b90eb4043f",
"version": "==0.3.2"
},
"pendulum": {
"hash": "sha256:8b52d8fc1e7a8b3025e434da351c98c34b478e8818d9fedbffed11ffb7a156cf",
"version": "==1.2.5"
},
"pep8": {
"hash": "sha256:4fc2e478addcf17016657dff30b2d8d611e8341fac19ccf2768802f6635d7b8a",
"version": "==1.7.0"
},
"pew": {
"hash": "sha256:259ac7a4603fe41b1fa950f30b2e4ccf4a23f7c89be2c34e0a4cec176c3ec581",
"version": "==0.1.26"
},
"pip-tools": {
"hash": "sha256:8d2b378e514ab6132b7f3215834c13eb5434645d0afacce37bfa1d0451faa015",
"version": "==1.9.0"
},
"py": {
"hash": "sha256:2ccb79b01769d99115aa600d7eed99f524bf752bba8f041dc1c184853514655a",
"version": "==1.4.34"
},
"pytest": {
"hash": "sha256:82c1e44a964ec5922c7c3891787df31c8c4f18b6c97a722df56b6cf20bb38c8a",
"version": "==3.2.1"
},
"python-dateutil": {
"hash": "sha256:95511bae634d69bc7329ba55e646499a842bc4ec342ad54a8cdb65645a0aad3c",
"version": "==2.6.1"
},
"pythonz-bd": {
"hash": "sha256:6dacd9aed6018014c75acfa9c994d755715c73bc786bdc6c6186d3cf184bacf0",
"version": "==1.11.4"
},
"pytz": {
"hash": "sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67",
"version": "==2017.2"
},
"pytzdata": {
"hash": "sha256:8819c5c30384f86a89506b25dc0d4871345f80801cb368c3559bbc00c49ab41a",
"version": "==2017.2.1"
},
"regex": {
"hash": "sha256:27ab18243b1a0aa1467027be93b118c9fcd60dd2e4020da579fad3008bc4638f",
"version": "==2017.07.28"
},
"requests": {
"hash": "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
"version": "==2.18.4"
},
"resumable-urlretrieve": {
"hash": "sha256:947ee07dd68cb5e2989377bdc8f2f4f1f327b6ea531b03da50c6b819b29bc7fb",
"version": "==0.1.5"
},
"ruamel.yaml": {
"hash": "sha256:2d0c3d80dc5e0b6bb8ca6f9b961b1a87b5e67c7ce774287b2773de64f9078be8",
"version": "==0.15.33"
},
"semver": {
"hash": "sha256:a6212f5c552452e306502ac8476bbca48af62db29c4528fdd91d319d0a44b07b",
"version": "==2.7.8"
},
"setuptools": {
"hash": "sha256:4d54c0bfee283e78609169213f9c075827d5837086f58b588b417b093c23464b",
"version": "==36.4.0"
},
"six": {
"hash": "sha256:0ff78c403d9bccf5a425a6d31a12aa6b47f1c21ca4dc2573a7e2f32a97335eb1",
"version": "==1.10.0"
},
"tzlocal": {
"hash": "sha256:05a2908f7fb1ba8843f03b2360d6ad314dbf2bce4644feb702ccd38527e13059",
"version": "==1.4"
},
"urllib3": {
"hash": "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
"version": "==1.22"
},
"virtualenv": {
"hash": "sha256:39d88b533b422825d644087a21e78c45cf5af0ef7a99a1fc9fbb7b481e5c85b0",
"version": "==15.1.0"
},
"virtualenv-clone": {
"hash": "sha256:6b3be5cab59e455f08c9eda573d23006b7d6fb41fae974ddaa2b275c93cc4405",
"version": "==0.2.6"
}
},
"develop": {
"alabaster": {
"hash": "sha256:2eef172f44e8d301d25aff8068fddd65f767a3f04b5f15b0f4922f113aa1c732",
"version": "==0.7.10"
},
"babel": {
"hash": "sha256:e0d07af61ff43729f61dc838f2af283c315e671454e7cf9d744ac9c56cccfca6",
"version": "==2.5.0"
},
"certifi": {
"hash": "sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704",
"version": "==2017.7.27.1"
},
"chardet": {
"hash": "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
"version": "==3.0.4"
},
"delegator.py": {
"hash": "sha256:495e11ada66648650171a6c9a188df4eb050b235abff8771f41ee8a064eb9ded",
"version": "==0.0.13"
},
"docutils": {
"hash": "sha256:02aec4bd92ab067f6ff27a38a38a41173bf01bed8f89157768c1573f53e474a6",
"version": "==0.14"
},
"flake8": {
"hash": "sha256:f1a9d8886a9cbefb52485f4f4c770832c7fb569c084a9a314fb1eaa37c0c2c86",
"version": "==3.4.1"
},
"idna": {
"hash": "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4",
"version": "==2.6"
},
"imagesize": {
"hash": "sha256:6ebdc9e0ad188f9d1b2cdd9bc59cbe42bf931875e829e7a595e6b3abdc05cdfb",
"version": "==0.7.1"
},
"jinja2": {
"hash": "sha256:2231bace0dfd8d2bf1e5d7e41239c06c9e0ded46e70cc1094a0aa64b0afeb054",
"version": "==2.9.6"
},
"markupsafe": {
"hash": "sha256:a6be69091dac236ea9c6bc7d012beab42010fa914c459791d627dad4910eb665",
"version": "==1.0"
},
"mccabe": {
"hash": "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
"version": "==0.6.1"
},
"mock": {
"hash": "sha256:5ce3c71c5545b472da17b72268978914d0252980348636840bd34a00b5cc96c1",
"version": "==2.0.0"
},
"pbr": {
"hash": "sha256:60c25b7dfd054ef9bb0ae327af949dd4676aa09ac3a9471cdc871d8a9213f9ac",
"version": "==3.1.1"
},
"pexpect": {
"hash": "sha256:f853b52afaf3b064d29854771e2db509ef80392509bde2dd7a6ecf2dfc3f0018",
"version": "==4.2.1"
},
"ptyprocess": {
"hash": "sha256:e8c43b5eee76b2083a9badde89fd1bbce6c8942d1045146e100b7b5e014f4f1a",
"version": "==0.5.2"
},
"py": {
"hash": "sha256:2ccb79b01769d99115aa600d7eed99f524bf752bba8f041dc1c184853514655a",
"version": "==1.4.34"
},
"pycodestyle": {
"hash": "sha256:6c4245ade1edfad79c3446fadfc96b0de2759662dc29d07d80a6f27ad1ca6ba9",
"version": "==2.3.1"
},
"pyflakes": {
"hash": "sha256:cc5eadfb38041f8366128786b4ca12700ed05bbf1403d808e89d57d67a3875a7",
"version": "==1.5.0"
},
"pygments": {
"hash": "sha256:78f3f434bcc5d6ee09020f92ba487f95ba50f1e3ef83ae96b9d5ffa1bab25c5d",
"version": "==2.2.0"
},
"pytest": {
"hash": "sha256:82c1e44a964ec5922c7c3891787df31c8c4f18b6c97a722df56b6cf20bb38c8a",
"version": "==3.2.1"
},
"pytz": {
"hash": "sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67",
"version": "==2017.2"
},
"requests": {
"hash": "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
"version": "==2.18.4"
},
"setuptools": {
"hash": "sha256:4d54c0bfee283e78609169213f9c075827d5837086f58b588b417b093c23464b",
"version": "==36.4.0"
},
"six": {
"hash": "sha256:0ff78c403d9bccf5a425a6d31a12aa6b47f1c21ca4dc2573a7e2f32a97335eb1",
"version": "==1.10.0"
},
"snowballstemmer": {
"hash": "sha256:9f3bcd3c401c3e862ec0ebe6d2c069ebc012ce142cce209c098ccb5b09136e89",
"version": "==1.2.1"
},
"sphinx": {
"hash": "sha256:3ea0faf3e152a0e40372d8495c8cbd59e93f89266231c367d8098ec0dfede98f",
"version": "==1.6.3"
},
"sphinxcontrib-websupport": {
"hash": "sha256:f4932e95869599b89bf4f80fc3989132d83c9faa5bf633e7b5e0c25dffb75da2",
"version": "==1.0.1"
},
"toml": {
"hash": "sha256:b3953bffe848ad9a6d554114d82f2dcb3e23945e90b4d9addc9956f37f336594",
"version": "==0.9.2"
},
"urllib3": {
"hash": "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
"version": "==1.22"
}
}
} |
Nice! How would you feel about something like |
@kennethreitz AFAICT pypi.python.org currently provides only |
@ncoghlan maybe |
Updated to |
@shaded-enmity excellent, i'll look into using that — that would vastly improve our lock times for |
@kennethreitz Heh, I initially had " |
This is now released! |
Closing issue as complete. Comment if you experience issues. |
Well, only partially. I think we have a ways to go still. |
Going to make |
It's so fast! |
v6.0.0 released, which includes hashes for all editions of each given release. 💫✨🍰✨💫 It is also on by default, and can't be turned off (at least, yet). I don't foresee anyone wanting to turn it off, it works so well. Now, this issue can be closed, successfully. Thanks again! |
Gah, there's a bug. I'll fix it. |
Fixed!
|
✨🍰✨ |
Working on a pipenv based development setup in oamg/leapp#8 we're finding it fairly easy for people to get into a situation where they accidentally create the venv with Python 2, but the lock file was generated with Python 3, so the hashes for pre-built wheel files may not match.
Example from oamg/leapp#8 (comment):
@shaded-enmity and I thought of two possible ways to make the resulting hash mismatch errors less confusing:
For the first option, any hash mismatches would be preceded by a warning that the Python versions didn't match so the lock file may need to be regenerated.
For the second, if a hash mismatch was found, it would be reported as a filename mismatch first, and only reported as a hash mismatch if the filenames matched.
The text was updated successfully, but these errors were encountered: