Only write lock file when installation is success #7498

wagnerluis1982 · 2023-02-10T06:22:05Z

Pull Request Check List

Resolves: #395

Added tests for changed code.
Updated documentation for changed code.

How it appear to users

The user experience is almost the same, but "Writing lock file" only appears in the end of the process, and only if the installation process succeeds.

On success ("Writing lock file" displays after installation)

$ poetry add jinja2      
Using version ^3.1.2 for jinja2

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 2 installs, 0 updates, 0 removals

  • Installing markupsafe (2.1.2)
  • Installing jinja2 (3.1.2)

Writing lock file

On failure ("Writing lock file" is not displayed)

$ poetry add pyaudio
Using version ^0.2.13 for pyaudio

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing pyaudio (0.2.13): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel

  ...bdist logs omitted for brevity...

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyaudio (0.2.13) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "pyaudio (==0.2.13)"'.

dimbleby · 2023-02-12T11:44:05Z

#395 was scoped only at not leaving the lock file inconsistent with the pyproject.toml. This goes further in also rolling back changes to the virtual environment.

I'm not sure whether this is a good idea. Though I can see how this might be desirable, it adds complexity and edge cases. (What if the rollback itself fails? What if the user had previously gone pip install some-tool in the environment and the dependencies being removed had overlap with some-tool's dependencies? in that second case, the environment after rollback won't be restored to its original state)

The failed install left the user with no hint that the lockfile might be inconsistent with pyproject.toml. No question that #395 identifies a bug here that wants fixing.

However, the failed install made plenty of logs saying "Installing package-foo" etc and so it should be obvious to a user that their environment has changed. Given the difficulty of reliably restoring exactly the previous environment, it's simpler and safer to let the user decide what to do about that.

Eg maybe they'd actually prefer to keep the 90% of the install that was successful while they fix up some missing libwhatever-dev dependency to allow the rest to succeed.

I'd suggest limiting the scope of this MR to leaving pyproject.toml and poetry.lock in a good state

wagnerluis1982 · 2023-02-12T11:55:48Z

I'd suggest limiting the scope of this MR to leaving pyproject.toml and poetry.lock in a good state

I don't doubt this PR requires some polishing, but this is exactly what I'm doing here, no? In case of installation failure, I'm not removing any package, not really. I'm cleaning up the objects in the scope of poetry add and running installer in lock only mode to keep the lock consistent.

wagnerluis1982 · 2023-02-12T11:59:49Z

Oh, now I realized the source of confusion. Here I'm not in no way making changes to environment.

I mean, AFAIK installer in lock only mode don't change the environment, so I'm relying on it.

dimbleby · 2023-02-12T12:05:03Z

Oh, perhaps I misread. Still I am surprised that this fix requires any re-locking: I'd expect to arrange things so that the file isn't written (or is explicitly rewritten with a saved copy of the old file) in case of failure - more like the handling of pyproject.toml.

Also maybe cf #7401 - it seems to be uncertain whether anyone likes that enough to merge it as a whole, but the locker.refresh() method there looks like it might be an improvement that could be useful in this one too.

dimbleby · 2023-02-12T12:14:42Z

What about poetry update? Should that update the lockfile, if installing updated packages fails?

Maybe you already looked into this, perhaps a better place to catch all such examples would be the Installer's _do_install() method. That could defer writing the lockfile until it knew that it had successfully executed operations.

wagnerluis1982 · 2023-02-12T13:11:44Z

What about poetry update? Should that update the lockfile, if installing updated packages fails?

Maybe you already looked into this, perhaps a better place to catch all such examples would be the Installer's _do_install() method. That could defer writing the lockfile until it knew that it had successfully executed operations.

Good points. Excellent points, actually. This will require a further investigation, and probably a more complex change. At least I already have the test 😸

dimbleby · 2023-02-13T19:15:51Z

looking at your latest draft, I'd definitely suggest that simpler than backing up and restoring the lockfile would be to refactor _do_install() so that the write happens later, and only on success.

do_install() is certainly kinda messy today, if you can find a sensible way to extract something out of it so that it becomes a bit mroe readable that would probably be welcome too!

wagnerluis1982 · 2023-02-13T20:33:45Z

looking at your latest draft, I'd definitely suggest that simpler than backing up and restoring the lockfile would be to refactor _do_install() so that the write happens later, and only on success.

Thank you very much for looking at the draft and for the suggestion.

Lazy as I am, I was avoiding to make any rewrite, but I was definitely starting to think along these lines 😸

do_install() is certainly kinda messy today, if you can find a sensible way to extract something out of it so that it becomes a bit mroe readable that would probably be welcome too!

I have a few ideas I'll try this week.

But do_install mess isn't the only problem here. To work with the Locker is a lot confusing.

wagnerluis1982 · 2023-02-14T22:36:15Z

@dimbleby when you have a chance, check my latest update.

I am applying a delay in the write to lock file by creating an internal Locker wrapper that holds the lock data in memory is signalized that lock data is good to be dumped to file.

wagnerluis1982 · 2023-02-17T00:50:09Z

Okay, I feel this is ready for a review.

There are only 2 things I am uncertain:

The message "Writing lock file" is now a lie, since the lock is not anymore written at that place, but if moved to the end, it will change the existing output, where "Writing lock file" appears before install the dependencies. Since I am not sure this is desirable, I kept the output as is.
Related to the previous, I think we can keep the existing "Writing lock file" and add on the rollback a message "Rolling back lock file". Despite the fact the lock file was never written on installation file, I feel that is a good message to appear.

@dimbleby I am still thinking on a possible refactor on _do_install, but, if I do, I will do in context of a separate PR (that code is currently driving me nut).

tests/console/commands/test_add.py

tests/helpers.py

tests/installation/test_installer.py

wagnerluis1982 · 2023-02-17T00:57:37Z

tests/installation/test_installer.py

        self._written_data = None
        self._locked = False
+        self._lock_data = None


Reason: some tests were failing due to the lack of this property (that exists on parent Locker)

tests/installation/test_installer.py

tests/installation/test_installer_old.py

dimbleby · 2023-02-17T08:36:25Z

I don't much like it! This adds a pile of code that I don't think ought to be needed.

If the _TransactionalLocker is a useful API, let's just change the Locker rather than wrapping it up
But is it needed? can't you just rearrange _do_install() so that the call to write the lockfile happens later (and only on success)?

wagnerluis1982 · 2023-03-12T21:20:46Z

I didn't include the wheel files. They are in another branch that I am experimenting.

dimbleby · 2023-03-13T08:50:48Z

I have a dependency on setuptools??? It makes no sense.

like the earlier dependency on wheel, this will be a default build requirement for a package that needs building and that does not specify its build requirements.

certainly it's undesirable for these tests to rely on having an internet connection

wagnerluis1982 · 2023-03-14T21:51:55Z

like the earlier dependency on wheel, this will be a default build requirement for a package that needs building and that does not specify its build requirements.

I will take your word.

Still, don't you think that's weird the solver to show to the user

  • Installing setuptools (39.2.0)

and in the end to display

  SolveFailure

  Because -root- depends on setuptools (>=40.8.0) which doesn't match any versions, version solving failed.

dimbleby · 2023-03-14T23:42:54Z

pretty sure it's just an unlucky coincidence that this testcase happens to pick up a dependency on setuptools 39.2.0, while some other dependency has (by default) a build-time dependency on setuptools 40.8.0.

Agree that it's not completely obvious that this is what is happening, if you haven't got enough to fix in this MR already feel free also to make the messages better!

radoering · 2023-03-15T04:56:38Z

For example, we should replace -root- with the actual name of the package we are trying to build if possible. Originally, I thought this comes from

poetry/src/poetry/installation/chef.py

Line 69 in 40061f9

package = ProjectPackage("__root__", "0.0.0")

but it's spelled differently so it may be unrelated...

dimbleby · 2023-03-15T08:46:00Z

it definitely comes from there, -root- is __root__ after normalisation

dimbleby · 2023-04-03T16:25:35Z

@wagnerluis1982 if you're still interested in progressing this then it's going to be worth rebasing on master, the testcase that was giving you trouble is fixed as at #7759

wagnerluis1982 · 2023-04-03T19:00:52Z

@dimbleby thanks. I'm still interested. Will do the rebase ASAP.

src/poetry/installation/installer.py

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [poetry](https://python-poetry.org/) ([source](https://github.com/python-poetry/poetry), [changelog](https://python-poetry.org/history/)) | minor | `1.4.2` -> `1.5.0` | --- ### Release Notes <details> <summary>python-poetry/poetry</summary> ### [`v1.5.0`](https://github.com/python-poetry/poetry/blob/HEAD/CHANGELOG.md#150---2023-05-19) [Compare Source](python-poetry/poetry@1.4.2...1.5.0) ##### Added - **Introduce the new source priorities `explicit` and `supplemental`** ([#7658](python-poetry/poetry#7658), [#6879](python-poetry/poetry#6879)). - **Introduce the option to configure the priority of the implicit PyPI source** ([#7801](python-poetry/poetry#7801)). - Add handling for corrupt cache files ([#7453](python-poetry/poetry#7453)). - Improve caching of URL and git dependencies ([#7693](python-poetry/poetry#7693), [#7473](python-poetry/poetry#7473)). - Add option to skip installing directory dependencies ([#6845](python-poetry/poetry#6845), [#7923](python-poetry/poetry#7923)). - Add `--executable` option to `poetry env info` ([#7547](python-poetry/poetry#7547)). - Add `--top-level` option to `poetry show` ([#7415](python-poetry/poetry#7415)). - Add `--lock` option to `poetry remove` ([#7917](python-poetry/poetry#7917)). - Add experimental `POETRY_REQUESTS_TIMEOUT` option ([#7081](python-poetry/poetry#7081)). - Improve performance of wheel inspection by avoiding unnecessary file copy operations ([#7916](python-poetry/poetry#7916)). ##### Changed - **Remove the old deprecated installer and the corresponding setting `experimental.new-installer`** ([#7356](python-poetry/poetry#7356)). - **Introduce `priority` key for sources and deprecate flags `default` and `secondary`** ([#7658](python-poetry/poetry#7658)). - Deprecate `poetry run <entry point>` if the entry point was not previously installed via `poetry install` ([#7606](python-poetry/poetry#7606)). - Only write the lock file if the installation succeeds ([#7498](python-poetry/poetry#7498)). - Do not write the unused package category into the lock file ([#7637](python-poetry/poetry#7637)). ##### Fixed - Fix an issue where Poetry's internal pyproject.toml continually grows larger with empty lines ([#7705](python-poetry/poetry#7705)). - Fix an issue where Poetry crashes due to corrupt cache files ([#7453](python-poetry/poetry#7453)). - Fix an issue where the `Retry-After` in HTTP responses was not respected and retries were handled inconsistently ([#7072](python-poetry/poetry#7072)). - Fix an issue where Poetry silently ignored invalid groups ([#7529](python-poetry/poetry#7529)). - Fix an issue where Poetry does not find a compatible Python version if not given explicitly ([#7771](python-poetry/poetry#7771)). - Fix an issue where the `direct_url.json` of an editable install from a git dependency was invalid ([#7473](python-poetry/poetry#7473)). - Fix an issue where error messages from build backends were not decoded correctly ([#7781](python-poetry/poetry#7781)). - Fix an infinite loop when adding certain dependencies ([#7405](python-poetry/poetry#7405)). - Fix an issue where pre-commit hooks skip pyproject.toml files in subdirectories ([#7239](python-poetry/poetry#7239)). - Fix an issue where pre-commit hooks do not use the expected Python version ([#6989](python-poetry/poetry#6989)). - Fix an issue where an unclear error message is printed if the project name is the same as one of its dependencies ([#7757](python-poetry/poetry#7757)). - Fix an issue where `poetry install` returns a zero exit status even though the build script failed ([#7812](python-poetry/poetry#7812)). - Fix an issue where an existing `.venv` was not used if `in-project` was not set ([#7792](python-poetry/poetry#7792)). - Fix an issue where multiple extras passed to `poetry add` were not parsed correctly ([#7836](python-poetry/poetry#7836)). - Fix an issue where `poetry shell` did not send a newline to `fish` ([#7884](python-poetry/poetry#7884)). - Fix an issue where `poetry update --lock` printed operations that were not executed ([#7915](python-poetry/poetry#7915)). - Fix an issue where `poetry add --lock` did perform a full update of all dependencies ([#7920](python-poetry/poetry#7920)). - Fix an issue where `poetry shell` did not work with `nushell` ([#7919](python-poetry/poetry#7919)). - Fix an issue where subprocess calls failed on Python 3.7 ([#7932](python-poetry/poetry#7932)). - Fix an issue where keyring was called even though the password was stored in an environment variable ([#7928](python-poetry/poetry#7928)). ##### Docs - Add information about what to use instead of `--dev` ([#7647](python-poetry/poetry#7647)). - Promote semantic versioning less aggressively ([#7517](python-poetry/poetry#7517)). - Explain Poetry's own versioning scheme in the FAQ ([#7517](python-poetry/poetry#7517)). - Update documentation for configuration with environment variables ([#6711](python-poetry/poetry#6711)). - Add details how to disable the virtualenv prompt ([#7874](python-poetry/poetry#7874)). - Improve documentation on whether to commit `poetry.lock` ([#7506](python-poetry/poetry#7506)). - Improve documentation of `virtualenv.create` ([#7608](python-poetry/poetry#7608)). ##### poetry-core ([`1.6.0`](https://github.com/python-poetry/poetry-core/releases/tag/1.6.0)) - Improve error message for invalid markers ([#569](python-poetry/poetry-core#569)). - Increase robustness when deleting temporary directories on Windows ([#460](python-poetry/poetry-core#460)). - Replace `tomlkit` with `tomli`, which changes the interface of some *internal* classes ([#483](python-poetry/poetry-core#483)). - Deprecate `Package.category` ([#561](python-poetry/poetry-core#561)). - Fix a performance regression in marker handling ([#568](python-poetry/poetry-core#568)). - Fix an issue where wildcard version constraints were not handled correctly ([#402](python-poetry/poetry-core#402)). - Fix an issue where `poetry build` created duplicate Python classifiers if they were specified manually ([#578](python-poetry/poetry-core#578)). - Fix an issue where local versions where not handled correctly ([#579](python-poetry/poetry-core#579)). </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.walbeck.it/walbeck-it/docker-python-poetry/pulls/717 Co-authored-by: renovate-bot <bot@walbeck.it> Co-committed-by: renovate-bot <bot@walbeck.it>

github-actions · 2024-03-03T22:47:41Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

wagnerluis1982 changed the title ~~poetry add rollback poetry.lock when installation failure~~ poetry add to rollback poetry.lock when installation failure Feb 11, 2023

wagnerluis1982 changed the title ~~poetry add to rollback poetry.lock when installation failure~~ poetry add to rollback poetry.lock when installation fails Feb 11, 2023

wagnerluis1982 force-pushed the poetry-add-rollback-lock-file-when-installation-failure branch 3 times, most recently from 4d52c91 to dd7be3b Compare February 12, 2023 09:41

wagnerluis1982 marked this pull request as ready for review February 12, 2023 09:50

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

wagnerluis1982 marked this pull request as draft February 12, 2023 13:12

wagnerluis1982 force-pushed the poetry-add-rollback-lock-file-when-installation-failure branch from 5d25b55 to 7e8e8fb Compare February 14, 2023 05:08

wagnerluis1982 changed the title ~~poetry add to rollback poetry.lock when installation fails~~ Only write lock file when installation is success Feb 14, 2023

This comment was marked as resolved.

Sign in to view

wagnerluis1982 force-pushed the poetry-add-rollback-lock-file-when-installation-failure branch 2 times, most recently from 3ef4e00 to 7562620 Compare February 17, 2023 00:31

wagnerluis1982 marked this pull request as ready for review February 17, 2023 00:38

wagnerluis1982 commented Feb 17, 2023

View reviewed changes

This comment was marked as duplicate.

Sign in to view

wagnerluis1982 force-pushed the poetry-add-rollback-lock-file-when-installation-failure branch from e197b6f to eb18b79 Compare February 17, 2023 01:46

radoering mentioned this pull request Mar 17, 2023

installer: improve error messages for building dependencies #7667

Merged

2 tasks

dimbleby mentioned this pull request Apr 2, 2023

installer tests should verify result of install #7759

Merged

wagnerluis1982 force-pushed the poetry-add-rollback-lock-file-when-installation-failure branch 2 times, most recently from 109cafc to ce91b86 Compare April 3, 2023 19:29

radoering reviewed Apr 5, 2023

View reviewed changes

src/poetry/installation/installer.py Show resolved Hide resolved

wagnerluis1982 added 8 commits April 6, 2023 10:41

Refactor tests to keep correct types

af10aee

Add test to test_add.py

1203ade

Add test to test_installer.py

f610e8c

Only write lock file when installation is success

47e0e0d

Refactor Locker.set_lock_data

d5ac62d

Fix tests by shifting "Writing lock file" to the end

b9871c6

add assertion to installer.run()

8695bcf

improve test_add_keyboard_interrupt_restore_content

0e65084

wagnerluis1982 force-pushed the poetry-add-rollback-lock-file-when-installation-failure branch from ce91b86 to 0e65084 Compare April 6, 2023 08:41

radoering approved these changes Apr 6, 2023

View reviewed changes

radoering merged commit 623bfff into python-poetry:master Apr 6, 2023

wagnerluis1982 deleted the poetry-add-rollback-lock-file-when-installation-failure branch April 6, 2023 15:32

wagnerluis1982 mentioned this pull request Apr 7, 2023

Update "Writing lock file" placement python-poetry/website#114

Merged

radoering mentioned this pull request May 18, 2023

release: bump version to 1.5.0 #7947

Merged

github-actions bot locked as resolved and limited conversation to collaborators Mar 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Only write lock file when installation is success #7498

Only write lock file when installation is success #7498

wagnerluis1982 commented Feb 10, 2023 •

edited

Loading

This comment was marked as outdated.

This comment was marked as resolved.

dimbleby commented Feb 12, 2023 •

edited

Loading

wagnerluis1982 commented Feb 12, 2023

wagnerluis1982 commented Feb 12, 2023

dimbleby commented Feb 12, 2023

dimbleby commented Feb 12, 2023

wagnerluis1982 commented Feb 12, 2023

dimbleby commented Feb 13, 2023

wagnerluis1982 commented Feb 13, 2023

wagnerluis1982 commented Feb 14, 2023

This comment was marked as resolved.

This comment was marked as resolved.

wagnerluis1982 commented Feb 17, 2023

wagnerluis1982 Feb 17, 2023

This comment was marked as duplicate.

dimbleby commented Feb 17, 2023

wagnerluis1982 commented Mar 12, 2023

dimbleby commented Mar 13, 2023

wagnerluis1982 commented Mar 14, 2023

dimbleby commented Mar 14, 2023

radoering commented Mar 15, 2023

dimbleby commented Mar 15, 2023

dimbleby commented Apr 3, 2023

wagnerluis1982 commented Apr 3, 2023

github-actions bot commented Mar 3, 2024

Only write lock file when installation is success #7498

Only write lock file when installation is success #7498

Conversation

wagnerluis1982 commented Feb 10, 2023 • edited Loading

Pull Request Check List

How it appear to users

On success ("Writing lock file" displays after installation)

On failure ("Writing lock file" is not displayed)

This comment was marked as outdated.

This comment was marked as resolved.

dimbleby commented Feb 12, 2023 • edited Loading

wagnerluis1982 commented Feb 12, 2023

wagnerluis1982 commented Feb 12, 2023

dimbleby commented Feb 12, 2023

dimbleby commented Feb 12, 2023

wagnerluis1982 commented Feb 12, 2023

dimbleby commented Feb 13, 2023

wagnerluis1982 commented Feb 13, 2023

wagnerluis1982 commented Feb 14, 2023

This comment was marked as resolved.

This comment was marked as resolved.

wagnerluis1982 commented Feb 17, 2023

wagnerluis1982 Feb 17, 2023

Choose a reason for hiding this comment

This comment was marked as duplicate.

dimbleby commented Feb 17, 2023

wagnerluis1982 commented Mar 12, 2023

dimbleby commented Mar 13, 2023

wagnerluis1982 commented Mar 14, 2023

dimbleby commented Mar 14, 2023

radoering commented Mar 15, 2023

dimbleby commented Mar 15, 2023

dimbleby commented Apr 3, 2023

wagnerluis1982 commented Apr 3, 2023

github-actions bot commented Mar 3, 2024

wagnerluis1982 commented Feb 10, 2023 •

edited

Loading

dimbleby commented Feb 12, 2023 •

edited

Loading