Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade tooling on the project. #1329

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Upgrade tooling on the project. #1329

wants to merge 10 commits into from

Conversation

almet
Copy link
Member

@almet almet commented Sep 29, 2024

  • Replace black by ruff, as it's quicker ;
  • Drop python 3.7, since it's end of life since last year ;
  • Use uv wherever possible as a replacement for pip, as it's way faster to run, add an uv.lock file which will be synced before the releases and published here ;
  • Remove tox, it's too complex for this project and can easily be replaced by uv ;
  • Apply ruff formatting ;
  • Update the makefile accordingly ;
  • Update the CI accordingly

I'm not sure why, but pytest-libfaketime is currently not working when running inside uv, at least on an apple silicon machine. I uninstalled it otherwise the test couldn't run, failing with the following error:

uv run --extra dev pytest .
re-exec with libfaketime dependencies
dyld[16935]: terminating because inserted dylib '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' could not be loaded: tried: '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file)
dyld[16935]: tried: '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file)

- Replace black by ruff, as it's quicker ;
- Use `uv` wherever possible as a replacement for pip, as it's way faster to run, add an `uv.lock` file which will be synced before the releases and published here ;
- Remove tox, it's too complex for this project and can easily be replaced by `uv` ;
- Apply `ruff` formatting ;
- Update the makefile accordingly ;
- Update the CI accordingly
@almet
Copy link
Member Author

almet commented Sep 29, 2024

Tests are failing with python 3.10 + sqlite, but I cannot reproduce locally with the following command:

TESTING_SQLALCHEMY_DATABASE_URI="sqlite:///budget.db"; uv run --python 3.10 --extra dev --extra database pytest .

Edit: It's fixed after a re-run.

@almet
Copy link
Member Author

almet commented Sep 29, 2024

Curious about your feelings on this @zorun ?

@almet almet requested review from Glandos and zorun and removed request for Glandos September 29, 2024 23:01
Copy link
Member

@Glandos Glandos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still alive, and sometimes, I even have time to review!

pyproject.toml Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
almet and others added 2 commits September 30, 2024 14:24
Co-authored-by: Glandos <bugs-github@antipoul.fr>
Co-authored-by: Glandos <bugs-github@antipoul.fr>
@zorun
Copy link
Collaborator

zorun commented Oct 1, 2024

Keep in mind that I tend to be conservative for this kind of reviews ;) I once inherited a project using poetry and I really didn't like it.

Here are general questions:

  • What would be the main advantage and disadvantages of this change? (I don't buy the argument "it's newer fancy tooling"). Do we have performance issues with the current tooling?
  • What guarantee do we have that the new tooling will still work reliably X years from now?
  • Does it affect releases? Would the project still be installable via pip? If a user installs the project with pip, do they get the same dependencies that we have in the lockfile? Can we run tests for that situation?
  • Does it affect users directly or indirectly in other ways?
  • Does it create a higher bar for entry for new contributors?

@almet
Copy link
Member Author

almet commented Oct 1, 2024

No problem, thanks for the questions.

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

With that out of the way, let's discuss the points you bring here 👍

What would be the main advantage and disadvantages of this change?

I believe the main pro is speedup. I personally also like to be able to install different versions of python in a simple way with uv python install in just a few seconds, as it makes it easier to run tests without having to install these manually with another tool.

Do we have performance issues with the current tooling?

I don't believe we have performance issues with the current tooling, no.

What guarantee do we have that the new tooling will still work reliably X years from now?

My feeling is that uv is here to stay, but that's a reasonable question to ask. The postscriptum on the announce covers it a bit:

What does that mean for the future of these tools? Here is my take on this: for the community having someone pour money into it can create some challenges. For the PSF and the core Python project this is something that should be considered. However having seen the code and what uv is doing, even in the worst possible future this is a very forkable and maintainable thing. I believe that even in case Astral shuts down or were to do something incredibly dodgy licensing wise, the community would be better off than before uv existed.

Does it affect releases?

No, not really. It can affect how we release, because the tools we use would be installed by uv.

Would the project still be installable via pip?

Yes.

If a user installs the project with pip, do they get the same dependencies that we have in the lockfile?

When using pip, the dependencies are not compiled by the uv lockfile, but by what's defined in the pyproject.toml, so I believe we're covered here.

Can we run tests for that situation?

I don't think there is a need for it :-)

Does it affect users directly or indirectly in other ways?

It's mainly a DX change.

Does it create a higher bar for entry for new contributors?

I believe it's actually the other way around, as it makes things simpler and faster.

@zorun
Copy link
Collaborator

zorun commented Oct 1, 2024

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

Thanks, I wanted to hear your motivation. If it motivates you, that's good, but let's not break things for free.

Actually, thinking about it, I think my motivation comes from repairing broken things, so maybe we make the perfect combo :D

What would be the main advantage and disadvantages of this change?

I believe the main pro is speedup. I personally also like to be able to install different versions of python in a simple way with uv python install in just a few seconds, as it makes it easier to run tests without having to install these manually with another tool.

This is a good argument.

If a user installs the project with pip, do they get the same dependencies that we have in the lockfile?

When using pip, the dependencies are not compiled by the uv lockfile, but by what's defined in the pyproject.toml, so I believe we're covered here.

Can we run tests for that situation?

I don't think there is a need for it :-)

Let me be more clear, because this is a general grip I also had with poetry.

Lockfiles are very good when working between developers: we are sure we have the exact same version of dependencies. However, users will have different dependencies because they don't use the lockfiles, they use pip.

If we run tests with the dependencies from the lockfile, we don't test what the actual users will install, and this is quite bad.

We could run tests using the dependencies from pyproject.toml instead of the lockfile, but then, what would be the point of the lockfile? In that case, let's just use pyproject.toml also for development.

What do you think? Did I miss something obvious?

@almet
Copy link
Member Author

almet commented Oct 1, 2024

Thanks, I wanted to hear your motivation. If it motivates you, that's good, but let's not break things for free.

Actually, thinking about it, I think my motivation comes from repairing broken things, so maybe we make the perfect combo :D

🙌 🤭

Let me be more clear, because this is a general grip I also had with poetry.

Lockfiles are very good when working between developers: we are sure we have the exact same version of dependencies. However, users will have different dependencies because they don't use the lockfiles, they use pip.

If we run tests with the dependencies from the lockfile, we don't test what the actual users will install, and this is quite bad.

We could run tests using the dependencies from pyproject.toml instead of the lockfile, but then, what would be the point of the lockfile? In that case, let's just use pyproject.toml also for development.

Oh, that's nice, and I understand better now the reason why we have the "minimal" set of dependencies in our tests. I actually think this is a good idea.

I see the .lock file as a "known working set" of dependencies. I don't believe we need to ship it to other users, but that could be something documented, and also part of the release, for those who want to have more control over the environment they are running the project into.

For instance, for the Docker image, we could ensure that it ships with exactly the versions that were present at the time of the build, and I believe that would be a good thing. Not a requirement, but something useful for some people.

But, to be honest, these .lockfiles are just starting to make sense to me.

@almet
Copy link
Member Author

almet commented Oct 1, 2024

Also: thanks for the tone of the discussion, I told you already but it's been a pleasure to disagree with you, and find out how we can better understand each other and come to the same conclusion.

(This is probably because we're not trying to push our decisions onto each other, but really care about what we think, and why)

@Glandos
Copy link
Member

Glandos commented Oct 2, 2024

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

It's also my point-of-view. Since I knew nothing about uv, I checked it. It's pretty new. It's pretty cool. The only reassuring stuff is that it's made by the same team doing ruff which is here to stay. Otherwise, uv would have been the "15 competing standard" regarding Python dependency management.

Oh, that's nice, and I understand better now the reason why we have the "minimal" set of dependencies in our tests. I actually think this is a good idea.

I remember taking part in this, and this was a really good move, because it also tests installation in every cases. I also found myself in situation where I needed to install something (hello Ansible) at not the very last version, and was happy it worked :)

@almet
Copy link
Member Author

almet commented Oct 28, 2024

Hey, @zorun, after some rest, what's your feeling on this? Should we merge or close?

Copy link
Collaborator

@zorun zorun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not opposed on principle, but there are some unknowns for me. I have left a few comments.

Here is the main missing stuff, I think:

  • did you check that all instructions in docs/contributing.md still work after this change?
  • what's the position on the uvlock? As I said, I'm a bit dubious about it because users will install IHM via pip, so I believe we shouldn't use the uvlock at all. Does uv use the lock by default? Once we reach a consensus, please document the result somewhere in the doc.
  • should the Dockerfile use uv or continue using pip?


test:
# Dependency on linting to avoid running our expensive matrix test for nothing
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you remove this comment and the other ones below?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure! (tbh I don't remember). Let's add them back.

- name: Change dependencies to minimal supported versions
run: sed -i -e 's/>=/==/g; s/~=.*==\(.*\)/==\1/g; s/~=/==/g;' pyproject.toml
run: sed -i -e '/requires-python/!s/>=/==/g; /requires-python/!s/~=.*==\(.*\)/==\1/g; /requires-python/!s/~=/==/g;' pyproject.toml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this do? Maybe add a comment.

if: matrix.database == 'mariadb'
env:
TESTING_SQLALCHEMY_DATABASE_URI: 'mysql+pymysql://root:ihatemoney@localhost:3306/ihatemoney_ci'
TESTING_SQLALCHEMY_DATABASE_URI: ${{ matrix.database == 'sqlite' && 'sqlite:///budget.db' || matrix.database == 'postgresql' && 'postgresql+psycopg2://postgres:ihatemoney@localhost:5432/ihatemoney_ci' || 'mysql+pymysql://root:ihatemoney@localhost:3306/ihatemoney_ci' }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't find this very readable, what's wrong with the current approach?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can add a comment there. It's just a different approach, that I find better because it has less repetition. I can surely add a comment here and format it better so it's easier to parse :-)

@almet
Copy link
Member Author

almet commented Oct 31, 2024

did you check that all instructions in docs/contributing.md still work after this change?

No, not yet. I will if we decide to go this way.

what's the position on the uvlock? As I said, I'm a bit dubious about it because users will install IHM via pip, so I believe we shouldn't use the uvlock at all. Does uv use the lock by default? Once we reach a consensus, please document the result somewhere in the doc.

I believe uv.lock files will allow us to ship a known working set of dependencies, that can be used for instance in the Docker file.

But, it won't be used for installations via pip. I think we should keep it, but it won't have any effect on the distribution via PyPI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants