Inconsistent Sort Direction in Lock File

[Kurt-von-Laven]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 20.04 LTS

• Python version: 3.10.5

• Poetry version: 1.1.14

• Link to the contents of your pyproject.toml file: pyproject.toml

Issue

Today (August 11th, 2022), between 12:48 and 18:25 UTC poetry lock --no-update began producing inconsistent output on the same pyproject.toml, breaking all of our CI pipelines. When run locally on Ubuntu 22.04 LTS, poetry lock --no-update continues to consistently produce this poetry.lock. This is consistent with past runs on Ubuntu 22.04 LTS in CI and hence what I perceive to be the correct output. Here is a CI workflow on Ubuntu 20.04 LTS showing the diff (reproduced below) between the old (correct) and new (incorrect) output. The new output appears consistent across CI runs. Note that the new order of lines within [package.extras] is reverse alphabetical order, and the new order of dependencies within each line is reversed with respect to the original order. One possible root cause may be that the lock command's sorting behavior is not fully self-contained (i.e., it's system dependent), and GitHub Actions rolled out a seemingly innocuous change to their ubuntu-20.04 image that exposed this preexisting bug. Note also that two spaces were added before metadata.python-versions.

diff --git a/poetry.lock b/poetry.lock
index f9756ea..26afd4e 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -146,8 +146,8 @@ optional = false
 python-versions = ">=3.7"
 
 [package.extras]
-docs = ["furo (>=2021.7.5b38)", "proselint (>=0.10.2)", "sphinx-autodoc-typehints (>=1.12)", "sphinx (>=4)"]
-test = ["appdirs (==1.4.4)", "pytest-cov (>=2.7)", "pytest-mock (>=3.6)", "pytest (>=6)"]
+test = ["pytest (>=6)", "pytest-mock (>=3.6)", "pytest-cov (>=2.7)", "appdirs (==1.4.4)"]
+docs = ["sphinx (>=4)", "sphinx-autodoc-typehints (>=1.12)", "proselint (>=0.10.2)", "furo (>=2021.7.5b38)"]
 
 [[package]]
 name = "pre-commit"
@@ -185,7 +185,7 @@ optional = false
 python-versions = ">=3.6.8"
 
 [package.extras]
-diagrams = ["railroad-diagrams", "jinja2"]
+diagrams = ["jinja2", "railroad-diagrams"]
 
 [[package]]
 name = "pyyaml"
@@ -264,8 +264,8 @@ platformdirs = ">=2,<3"
 six = ">=1.9.0,<2"
 
 [package.extras]
-docs = ["proselint (>=0.10.2)", "sphinx (>=3)", "sphinx-argparse (>=0.2.5)", "sphinx-rtd-theme (>=0.4.3)", "towncrier (>=21.3)"]
-testing = ["coverage (>=4)", "coverage-enable-subprocess (>=1)", "flaky (>=3)", "pytest (>=4)", "pytest-env (>=0.6.2)", "pytest-freezegun (>=0.4.1)", "pytest-mock (>=2)", "pytest-randomly (>=1)", "pytest-timeout (>=1)", "packaging (>=20.0)"]
+testing = ["packaging (>=20.0)", "pytest-timeout (>=1)", "pytest-randomly (>=1)", "pytest-mock (>=2)", "pytest-freezegun (>=0.4.1)", "pytest-env (>=0.6.2)", "pytest (>=4)", "flaky (>=3)", "coverage-enable-subprocess (>=1)", "coverage (>=4)"]
+docs = ["towncrier (>=21.3)", "sphinx-rtd-theme (>=0.4.3)", "sphinx-argparse (>=0.2.5)", "sphinx (>=3)", "proselint (>=0.10.2)"]
 
 [[package]]
 name = "wcwidth"
@@ -277,7 +277,7 @@ python-versions = "*"
 
 [metadata]
 lock-version = "1.1"
-python-versions = "^3.10.5"
+  python-versions = "^3.10.5"
 content-hash = "f32d8b6b140e1c797b913bcd598aee7a8ad3344114ee787c3050243c9655bc80"

[jkillian]

We've also started hitting this all the time recently and it's caused us a lot of CI issues as well. We're looking at a adding a custom post-processing step to make sure that the file is always in the same order, but would be nice if this was addressed in poetry of course.

[dylanscott]

In case it's useful to anyone else here's a quick and dirty script to canonicalize the order of entries in [package.extras], as at least for us those are the only sections that seem to be getting messed up (haven't run into the metadata.python-versions one). The script assumes it's in a file that's a sibling of poetry.lock but that should be easy to tweak

[sigprof]

This also affects PRs generated by Dependabot (e.g., this diff has lots of unrelated changes; in this case there are no extra spaces before python-versions though).

[dimbleby]

if anyone cares enough to do anything about this, I reckon the relevant code is

poetry/src/poetry/packages/locker.py

Lines 588 to 596 in 35281d9

	if package.extras:
	extras = {}
	for name, deps in package.extras.items():
	# TODO: This should use dep.to_pep_508() once this is fixed
	# https://github.com/python-poetry/poetry-core/pull/102
	extras[name] = [
	dep.base_pep_508_name if not dep.constraint.is_any() else dep.name
	for dep in deps
	]

.

Putting a couple of sorted()s in there should get these to come out in the same order each time.

[sigprof]

One possible complication is that at least Dependabot uses the poetry==1.1.14 release, so it's the equivalent code in the 1.1 branch:

poetry/poetry/packages/locker.py

Lines 572 to 580 in d2abf92

	if package.extras:
	extras = {}
	for name, deps in package.extras.items():
	# TODO: This should use dep.to_pep_508() once this is fixed
	# https://github.com/python-poetry/poetry-core/pull/102
	extras[name] = [
	dep.base_pep_508_name if not dep.constraint.is_any() else dep.name
	for dep in deps
	]

[dimbleby]

Maintainers have indicated that there will be no new 1.1 releases so you just have to play the long game: fix in master and trust that 1.2 will arrive, eventually.

[jeffwidman]

The fix was backported and released as https://github.com/python-poetry/poetry/releases/tag/1.1.15

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.