feat: relax dependencies in pyproject.toml and add a requirements.txt #2669

mistercrunch · 2024-11-26T01:55:19Z

Related to this: #2666

made a call on using uv as it seems it's what the cool kids are using nowadays
looked at .github/workflows/ and modified so that it would install the pinned files

NOTE:

I noticed local-requirements.txt file and added uv to it, though I'd recommend moving to optional-dependencies in pyproject.toml, maybe as unpinned there, and generate a requirements-dev.txt or similar from there. Though not super important
I might have missed other workflows/actions internal or external to the repo, but generally a good practice to install the pinned deps in all CI/automation

mistercrunch · 2024-11-26T02:00:16Z

@microsoft-github-policy-service agree [company="Preset Inc."]

mxschmitt · 2024-11-27T09:28:02Z

requirements.txt

@@ -0,0 +1,8 @@
+# This file was autogenerated by uv via the following command:
+#    uv pip compile pyproject.toml -o requirements.txt
+greenlet==3.1.1


What I'm worried about here is that when e.g. greenlet 3.1.2 comes out, which includes e.g. a breaking change or a bug for some reason, our CI would still run with 3.1.1 and not install the bad version.

Why is the requirements.txt file needed at all in this case? To provide reproducible builds?

our CI would still run with 3.1.1
Right, this is currently the case. But now you can enable @dependabot, who will auto-submit PRs periodically as new versions of dependencies come out. Those PRs will run your CI, so you can confirm that bumping is safe.

To provide reproducible builds?
Yes, so you get reproducible builds, and you let people who use your library choose their version within the specified range in pyproject.toml . Say in our case for Apache Superset, we have multiple other dependencies that rely on greenlet, each one specifying a know version range.

About the topic of maintaining the range, clearly you'll be running-CI only against what's in the pinned file. If/when people report something like "playwright doesn't work on top of greenlet>=5.0.0", as a maintainer you typically want to go and edit the known working range in pyproject.toml. Meaning we'd change the rule to greenlet>=3.1.1,<5.0.0 then, and dependabot would respect that rule, only submitting PRs to bump within the specified range.

Yeah lets do something like: greenlet>=3.1.1,<4.0.0. For pyee something similar, so its only 12.x for now and we can increase if there is a new version at some point. Just to make sure 13.x does not break us. Typing extensions is backed by the Python team, I think usually you don't depend on a max version there..

mxschmitt · 2024-11-27T09:28:26Z

pyproject.toml

@@ -13,8 +13,8 @@ license = {text = "Apache-2.0"}
 dynamic = ["version"]
 requires-python = ">=3.9"
 dependencies = [
-    "greenlet==3.1.1",
-    "pyee==12.1.1",
+    "pyee>=12.1.1",


Make sure to relax the version in the meta.yaml as well for our conda customers.

mmmmh yeah it's not great for this to not be DRY. I'm not familiar enough with conda to know the best practices on that side of the fence. Here's a related GPT thread discussing the different options/pros/cons -> https://chatgpt.com/share/67476ce2-0288-8010-bc05-e20ff72d15c0

I don't like the fact conda doesn't seem to be compatible with dependabot. Is it a requirement to support conda? Seems it's fading https://theregister.com/2024/08/08/anaconda_puts_the_squeeze_on/

Yes, I'd prefer to not drop it for now. Updating it manually on conda-side is good enough for us.

A1exKH

@mistercrunch could sync you branch with main, please?
This branch is out-of-date with the base branch.

mistercrunch · 2024-12-14T22:08:38Z

@A1exKH I just rebased the PR

mxschmitt · 2024-12-18T16:07:22Z

local-requirements.txt

@@ -20,3 +20,4 @@ service_identity==24.2.0
 twisted==24.11.0
 types-pyOpenSSL==24.1.0.20240722
 types-requests==2.32.0.20241016
+uv==0.5.4


I'd prefer to not introduce a new package for this. What do we loose if we don't use it? afaik its only used to update the command in the requirements.txt?

I needed something to compile the requirements, it's either this uv or pip-tools or maybe poetry (?). Unless conda has options to do this but I think conda is slowing down, and uv is the future of package management in python, replacing/improving-on/consolidating pip, virtualenv, pyenv, conda, ...

Why do we need to 'compile' the requirements? Maybe I miss something. I was thinking about updating them manually (or dependabot does it) - since these are only two - we should be able to deal with it? They get automatically installed using pip install -r requirements.txt anyways?

Here https://chatgpt.com/share/6764e629-4344-8010-8465-c1efe5e1813e

I see - so both dependencies: pyee and greenlet have no dependencies on itself - this is where I got confused. uv helps us to pin the transitive dependencies (where there aren't any yet). In the future there might be some. Is this correct?

that's right, generally good to have pin deps for CI otherwise a new lib comes out some day and your CI starts failing.

mxschmitt

Sorry for being slow! I think once the feedback is applied we can merge it. Please also adjust meta.yml so it has the same ranges as pyproject.toml. Thanks!

A1exKH

@mistercrunch please, sync your branch with main branch to solve this problem:

This branch is out-of-date with the base branch

mistercrunch · 2024-12-20T03:34:38Z

Would suggest disabling that GH feature forcing to rebase, isn't scalable, forces rebasing all PRs on every merge... But happy to rebase

A1exKH

LGTM.

stollero · 2025-01-13T07:08:52Z

Hi there @mxschmitt,

can this PR be applied ? We would like to update pyee but playwright currently blocks it 😬 Thanks a lot

pciporin25 · 2025-01-16T17:37:59Z

Would also appreciate this being merged as it's blocking a pyee upgrade; thank you in advance!

mxschmitt · 2025-01-27T15:19:30Z

Fixed in #2698

mistercrunch · 2025-01-27T16:36:46Z

Hey @mxschmitt, this is not a big deal to me, but by re-implementing a similar solution and closing this PR I don't get proper attribution for my contribution on GitHub. What matters to me is having a solution in the main branch, but would recommend supporting contributors to completion as a better open source practice.

mistercrunch mentioned this pull request Nov 26, 2024

[Feature]: relax greenlet dependency version requirements in pyproject.toml #2666

Closed

mxschmitt reviewed Nov 27, 2024

View reviewed changes

A1exKH suggested changes Dec 13, 2024

View reviewed changes

mistercrunch force-pushed the relax_deps branch from 7c562c1 to 69df9d6 Compare December 14, 2024 22:07

mxschmitt reviewed Dec 18, 2024

View reviewed changes

A1exKH suggested changes Dec 19, 2024

View reviewed changes

mistercrunch added 3 commits December 19, 2024 19:36

feat: relax dependencies in pyproject.toml and add a requirements.txt

c9d687d

pip install in GHA workflows

db860b3

add uv to local-requirements.txt

a893a3c

mistercrunch force-pushed the relax_deps branch from 69df9d6 to a893a3c Compare December 20, 2024 03:37

A1exKH approved these changes Dec 29, 2024

View reviewed changes

mxschmitt closed this Jan 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: relax dependencies in pyproject.toml and add a requirements.txt #2669

feat: relax dependencies in pyproject.toml and add a requirements.txt #2669

mistercrunch commented Nov 26, 2024 •

edited

Loading

mistercrunch commented Nov 26, 2024

mxschmitt Nov 27, 2024

mistercrunch Nov 27, 2024 •

edited

Loading

mxschmitt Dec 18, 2024

mxschmitt Nov 27, 2024

mistercrunch Nov 27, 2024

mistercrunch Nov 27, 2024

mxschmitt Dec 18, 2024

A1exKH left a comment

mistercrunch commented Dec 14, 2024

mxschmitt Dec 18, 2024

mistercrunch Dec 19, 2024

mxschmitt Dec 19, 2024

mistercrunch Dec 20, 2024

mxschmitt Dec 20, 2024

mistercrunch Dec 21, 2024

mxschmitt left a comment

A1exKH left a comment

mistercrunch commented Dec 20, 2024

A1exKH left a comment

stollero commented Jan 13, 2025

pciporin25 commented Jan 16, 2025

mxschmitt commented Jan 27, 2025

mistercrunch commented Jan 27, 2025

feat: relax dependencies in pyproject.toml and add a requirements.txt #2669

feat: relax dependencies in pyproject.toml and add a requirements.txt #2669

Conversation

mistercrunch commented Nov 26, 2024 • edited Loading

mistercrunch commented Nov 26, 2024

Choose a reason for hiding this comment

mistercrunch Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

A1exKH left a comment

Choose a reason for hiding this comment

mistercrunch commented Dec 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxschmitt left a comment

Choose a reason for hiding this comment

A1exKH left a comment

Choose a reason for hiding this comment

mistercrunch commented Dec 20, 2024

A1exKH left a comment

Choose a reason for hiding this comment

stollero commented Jan 13, 2025

pciporin25 commented Jan 16, 2025

mxschmitt commented Jan 27, 2025

mistercrunch commented Jan 27, 2025

mistercrunch commented Nov 26, 2024 •

edited

Loading

mistercrunch Nov 27, 2024 •

edited

Loading