-
-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
package maintanence #180
Comments
I also offered to help with maintenance and @lebigot gave me commit access but I haven't been able to make any maintenance updates yet. I mainly work with uncertainties through working on https://github.com/Qiskit-Extensions/qiskit-experiments (which also uses lmfit) but you have a bigger stake in uncertainties than I do @newville. I had just been planning some simple clean up and leaving publishing new releases to @lebigot until he told me otherwise. I am happy with any hierarchy of maintainership @lebigot prefers. I don't want to slow anything down. |
@wshanks if you or anyone else is willing, I would happily add you to the lmfit organization to participate in any projects there. |
@lebigot @wshanks I admit that it has been a while since I looked at the code, existing Issues, and PR for this package. I think that there are a few different things that could be improved:
Now that I see these and the explicit call for a maintainer from @lebigot, I think these all need to be done. A crude analysis shows 216K downloads/month for this project (https://www.pepy.tech/projects/uncertainties) and 177K downloads/month for lmfit-py (https://www.pepy.tech/projects/lmfit), which has I propose that this project move to the Does anyone have any preferences for how this is done, or interest in participating? |
@newville I am interested in participating. The things you listed were the simple clean up I had in mind (updating deprecated code including dropping support for old versions of Python, replacing the deprecated nose tests with pytest, replacing the out of fashion Travis CI with GitHub, moving from setup.py to pyproject.toml, etc.). I am also interested in the array interface work but others might be more motivated to do that then I am. I am mainly interested from a performance perspective which I think is a more involved project (operating on arrays of values in a way that could be compatible with jax or tensorflow). I don't have a strong opinion about how the project is organized. I will let you and @lebigot decide that. |
@wshanks Thanks - I agree with all of that! I suspect that a smoother array interface would benefit many folks who aren't making use of this library but probably should. Like, if calculations on pandas Datasets could propagate uncertainties properly, that could be very useful. |
I started writing an arr = np.array([ufloat(1, 0.01), ufloat(2, 0.1)])
uarr = UArray(arr)
np.sin(uarr)
<UArray [0.8414709848078965+/-0.005403023058681398
0.9092974268256817+/-0.04161468365471424] > However when I operate on a [np.sin(x) for x in arr]
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
AttributeError: 'Variable' object has no attribute 'sin'
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last)
Cell In[30], line 1
----> 1 [np.sin(x) for x in arr]
Cell In[30], line 1, in <listcomp>(.0)
----> 1 [np.sin(x) for x in arr]
TypeError: loop of ufunc does not support argument 0 of type Variable which has no callable sin method This makes it difficult to write tests; having to use umath or unumpy is clunky and makes the module difficult to use. I'd suggest adding functions to |
@andrewgsavage Hm, yeah I think I see what you mean - and maybe this needs a deeper re-design for both a float-like object and then ndarray of floats built on that (that I think might also make it easier for pint, pandas, xarray, ...). That's a more radical change than I was expecting, but I think it might be for the best. In the context of this PR about "maintenance", it seems that we're seeing a fair number of issues and PRs that are not getting much attention, and many of those are either "very old Python2-like" issues, or related to better wrappings for other numerical objects (pint, pandas, xarray, ...) I would prefer to keep the history, precedence, credit, and name of "uncertainties". I do not quite see that progressing very quickly. At some point, it might be better to start afresh, taking what we can from this project, but leaving behind outdated legacy. I want to say that we are not at that point yet, but I am becoming afraid that we might actually be there. Do you have an opinion on that? |
I think it may be doable without a deep re-design by impelementing numpy functions for I agree, the old Python2 like issues and codebase hinder further development of the module. It would be great to keep the name and history of uncertainties. The lack of response to your earlier question to @lebigot, the review in your PR asking for python2 compability, and the other long standing PRs that without response makes feel like it will be difficult and time consuming to make progress. Your |
@lebigot added me as a collaborator so I could merge some of the PRs here, but that happened shortly before this issue was opened. So I have held off doing any maintenance here until there is more consensus of how the project will be maintained (or forked and maintained) going forward. I can't change repo settings or publish to PyPI, so we still need more agreement with @lebigot (or to start a new project if there is not agreement). I think the main question is whether to keep the project here or move it under the |
@wshanks @andrewgsavage Thanks. I think that it would be much better to move to a GitHub "organization", where there can be multiple people with ownership and management roles. The As for retaining the name "uncertainties", I think the main issue is whether or how new maintainers could push updates to PyPI.com. I think there is already a reasonable mechanism for changing who pushes to I guess that we could start thinking about a new potential name for a "friendly fork" in case that is not possible. Again, I think that would not be anyone's first choice, but it seems like @lebigot may not have time or inclination to make any merges or to be able to make that move happen. I suggest that the three of us (@wshanks, @andrewgsavage, @newville but we could certainly add others who are instrumentation) decide on a time frame for deciding whether we need a new project with a new name. I suggest "March 1st", but am I not in a rush or committed to that timeline. Any other thoughts? |
I would also rather move to I'm not fussed as to what owership/management/stake I have - happy to just contribute or to review PRs and such. March 1st sounds reasonable |
@andrewgsavage Yes, we can start making changes at I see a few issues:
It would be preferable to have a smooth transition. I think that means that @lebigot would have to move the package to the I hope he is willing and able to do this, but it does not seem clear if this will happen. If not, we could just fork and work from there. We could either request a PyPI transfer or pick another name for the package. |
I have used uncertainties for some time and looked under the hood a little bit. I would be interested in helping out with some cleanup of I guess there are two issues being discussed in this issue. One is how maintenance of That said, the two above issues are a little bit intermingled if the idea of changing the name and releasing a heavily refactored or new package is on the table.
Requesting PyPi transfer seems possible and preferable. In the event he can't manage even those transfers it would be a shame for the "official" package to be the fork, but I guess so be it? Even if the code changes are massive it seems like the I'm going to throw in a little bit of a personal plug here: One of the main uses I've had for |
@jagerber48 thanks! I think it will be very good to have more interested parties involved. Yes, I agree that there are two separate topics:
I think the is sort of an immediate-ish need to support Python 3.12 and probably drop support for Python 2 (and no need to support Python < 3.7 or so), and then there are several ideas for feature improvements. There are several open PRs, some of them related and overlapping. I think an incomplete list (in no particular order) would be
I would be happy to see that list expanded (if I missed a favorite PR, I apologize, and let's add it!). I think we could start discussing priorities and order of operations for those. Just to be clear: I am willing to be engaged and help, but do not feel in any way that I should be "in charge". If anyone is willing to take the lead here, I would be happy to be a contributor and helper. On If you think it might be appropriate for |
I put my thoughts on this in #185
Yes this would be nicer. The information in the docstrings is really useful, just a little wordy. I'd prioritise issues like so, with no particular order in each bucket: Highest bucket
Next highest: 2a. Remove python2 specific code/comments Then improvements: 3a. better Numpy support (for pandas, xarray etc) That's my ideal ordering to make easier for us to review PRs and to reduce files moving around, but any PRs are appericated, it shouldn't stop anyone working on what they want. As are suggestions to add to the list or reordering if I've missed anything. |
might be worth noting that hgrecco is in the process of rewriting pint's formatting section and asking for feedback here |
@andrewgsavage Thanks! I agree with all of that. And, yes it would be nice to agree with the pint folks or formatting. I think that a separate project like |
Ideally, @lebigot will give his opinion on this. He added the call for maintainers to the README.md two months ago and has responded to an issue just over a month ago in #181 (comment), so I wouldn't say it would be in the spirit of PEP541 to classify the project as abandoned and transfer the name without hearing from him. |
Two questions: (1) I just want to clarify if we can begin working on the forked version of this repo in the |
I think so.
Sure. We're hoping that the current repo gets migrated, but then we'll be able to merge those. Many of the outstanding PRs probably need some attention and maybe some need discussion.
I think this repo currently has neither. It would probably need discussion, perhaps using something like pre-commit But I would recommend focusing attention on getting code working for supported versions of Python
All I know about ruff is that it boasts about being a fast linter. Do we care about linter speed? If so, why? To me, this suggests that ruff might prioritize other things that I would also personally not care about. But, I don't know anything else about it. How would ruff "modernize" the code? For linter/formatter, I would probably prefer to rely on PEP-8 and common sense. My bias is that discussions about formatting Python code that are beyond "we follow PEP-8, including but have the good sense to allow bending it on occasion" seem to come from people looking to pick a fight where none is needed. ;). |
I'll familiarize myself more with those. I was eyeing the list andrewsavage posted above with item 1 being getting CI and tests going green.
Yes, Obviously there are other ways to do this, and we don't need consensus for e.g. me to use the linter/formatter locally and then push the results to the PR (the PR is the step that needs approval, doesn't matter how the code got there.)
That's reasonable. I just wanted to bring up the linter/formatter question early because if there is linting/formatter that we would prefer to use I think it would be easier to implement it earlier before a large bought of code changes rather than later. To be clear I would personally prefer What versions of python should be supported? I'll just throw into the ring the idea of support Python 3.8+ since older versions are at python end-of-life. That said, I wouldn't see a need for dropping support for older python versions once they reach end-of-life. That is, 3.8 would be support indefinitely until something comes up that gives pressure to drop it. (Like Python 3 gives pressure to drop Python 2 after many years) I very much do not like non-f-string formatting so I would be pretty frustrated to support anything < Python 3.6. I know there are a lot of typing changes through 3.8 through 3.10 or 3.11 at least. Otherwise I'm not personally aware of any important python features that would encourage dropping support for any specific versions. Again, not trying to advocate hard for any specific support (other than >=3.6...), just trying to put some ideas out there. |
I find the linter/formatter helpful for me as it means I don't need to think about formatting, the CI will do it for me when I run pre-commit. There are some code patterns in uncertainties that I haven't seen in other libraries which I think would be picked up by a linter, which would help in modernizing the code. eg Having PEP-8 and other rules (we don't need to use everything in I too would like uncertainties to have a linter. I'll leave it up to whoever sets it up to decide which :p
I thought it would be easier to get it running when there's no/few open PRs, as you end up with a load of merge conflicts in them.
I think this is a reason to set one up as a CI - if you do this you'll end up with formatting changes unrelated to the PR in some files, which makes reviewing slightly more difficult.
The numpy If someone in the future wants to set up typing then that would be a reason to go to a newer version, but I'm happy to leave that for the time being. |
I figure there's no harm in submitting changes here for review now. Then they can be ready to merge in March. |
@jagerber48 @andrewgsavage Thanks -- I agree with all of that. Python 3.8+ or even Python 3.9+ would be acceptable to me. Specifying a version of Numpy (maybe even later than 1.17) would be OK with me. Standardizing on f-strings for most things would be fine with me, though I'll also say that sometimes In general, my basic view is that while linters and formatting conventions are useful, valid Python is already very readable, and forbidding valid Python should be done sparingly. That said, setting up automated use of a linter, with pre-commit scripts would be fine with me. I would vote for "runs on Python 3.8 to 3.12" to be the first priority. |
This is a very good point. Better to clean up the backlog of important PRs then do a major reformatting. I wouldn't want the linter/formatter to slow down progress on more important functionality.
fair point. In general the inclusion of automated linting/formatting will add overhead in some places (setting it up at first, possibly new contributors familiarizing themselves with how it works) even though it removes it in other places (contributors don't have to worry about little details like e.g. hanging indents). We can see how it looks once we get through the more important issues.
Yes, especially when working with custom formatting like that provided by
Seems like a good place to code against at first, and then the cost of supporting lower versions (3.7, 3.6, maybe 3.5) can be assessed. |
I'm interested in setting up typing, at the very least for the public API but again, happy to leave this very low on the priority list for now, including below supporting 3.8-3.12. |
@lebigot March 1st is coming up at the end of next week. We had (somewhat arbitrarily but I think not unreasonably) set this as a target deadline for a transition from this repo to one within the lmfit organization. I think everyone interested here would prefer that the name and the full history of the Is this feasible for you to do by March 1st? If not, can you provide some guidance on your wishes for this project? For example, is retaining the name "uncertainties" important? Let us know if there is anything we can do to help. Thanks, and hope you are doing well. |
The way I contacted @lebigot was via email. I used the address given in Just to comment on a couple other things above: The access I have right now is maintainer access, the ability to merge PRs and probably a few other things like editing someone's PR before merge or closing someone else's issue. What I don't have is the ability to change repository settings (add maintainers or change CI settings), publish to pypi, or update the docs. I think @newville was hoping for more sharing of those roles as well so that the project truly does not depend on one person for anything. That makes sense to me. If others hadn't stepped forward here, I would have tried the model of pushing package updates and prodding @lebigot to publish new releases. If there is a fork (hopefully it won't be necessary), I like the name |
I am very happy to see that this package has reasons, and the means to continue to live! Thank you to all of you. Passing the torchI am fully ready to have the full stewardship of I am also ready to help with its evolution; realistically, this will be when I feel that I can quickly give a short answer to a question where my experience with the package can have some real added value. The consensus is that I understand from this thread that keeping the name Making
|
@lebigot Thanks for your thorough response! I definitely have a breath of relief that we get some guidance from you and we'll be able to keep Re Some responses on
Yes, I love the f-string formatting in
Yes, I thought about this for
I don't think support Python < 3.6 would affect the formatting code that users have access to. The main issues I see are
I think it should also be explored what typing features might be precluded by trying to include python versions < 3.7 or 3.8 or so. |
#192 Here is a discussion dedicated to discussing adoption of |
There are instructions on transferring a repository at
From https://pypi.org/help/#collaborator-roles
I'd add @newville as owner, and add @andrewgsavage @wshanks, and @jagerber48 as maintainers for the time being
I think there is limited value to supporting python <3.8 as 3.7 is at end-of-life. I agree with @jagerber48's arguements for >=3.6. There was mention of typing in this issue, which requires more recent python versions (3.7?). Mostly I'd like to prioritise reducing development time, as there's plenty of improvments we'd like to do and ensuring compatability with earlier versions hinders this. Therefore I propose supporting >=3.8 (as is tested in #191 ), and if anyone raises an issue about needing earlier versions we can revisit this. edited to add @wshanks |
The conda-forge feedstock pulls from the pypi sdist rather than from GitHub so it won't break no matter what happens with the GitHub repo. Also, if the repo gets transferred into the lmfit organization, github.com/lebigot/uncertainties will redirect to github.com/lmfit/uncertainties as long as a new github.com/lebigot/uncertainties repo is not created. So anything pulling from the current repo url should keep working. Transferring the repo to the lmfit organization like @andrewgsavage outlined would be the best course of action in my opinion for preserving history and maintaing url redirects like I mentioned above, while still moving the repo into an organization which is more conducive to shared ownership than keeping the repo under lebigot/uncertainties. One note for @lebigot -- you might want to fork the lmfit/uncertainties repo after the transfer in order to follow the standard practice of opening PRs against it from a personal fork, but you should be careful to change the name of your fork to something like lebigot/uncertainties-fork because if you use the default name of lebigot/uncertainties that will break the redirect from lebigot/uncertainties to lmfit/uncertainties. I agree that I see value in only supporting Python versions that are supported by the CPython team itself. numpy for example stopped supporting 3.8 in 1.25.0 released in June 2023, so well before the end of life of Python 3.8 in October 2024. If you are using a version of Python released 6 years ago or more you probably don't expect the latest features and you can just keep using the last version released that did support the version of Python you are using. I could see an argument for backporting bug fixes to patch releases of the version of uncertainties that supported a particular version of Python if feasible, but the current code base has stayed useful despite no significant changes in the past two years, so this might be the kind of project that doesn't need much backporting of fixes. I would not break older versions unnecessarily but it is nice to be able to make use of new features in Python and numpy (f-strings, assignment expressions, match statements, typing, etc). Regarding PyPI, similar to moving the repo under the lmfit GitHub organization, it may make sense to create an lmfit PyPI organization and put uncertainties under that. PyPI organizations are still in beta at the moment though but PyPI started accepting applications to try them almost a year ago. I think the PyPI access shouldn't matter too much because likely the best course is to set up triggering publishing releases from GitHub CI and then GitHub privileges for adding a tag would gate PyPI releases (though some people need PyPI access in case settings need to be changed). |
@lebigot Thanks! and thanks for the helpful replies from @wshanks @andrewgsavage and @jagerber48 As @andrewgsavage says, to transfer the repo, you would go into the "settings" for your repo, then down to the "Danger Zone" and then select Transfer. You should be able to select transferring it to lmfit/uncertainties -- if anything seems uncomfortable, wrong, or strange, let me know and we can try to work out what's going wrong. We removed lmfit/uncertainties (but do have lmfit/uncertainties_fork) so that this could be done seamlessly. More about transferring a repository is at To be clear, PRs, stars and wiki are included in a transfer. The docs say that issues are included, but then also says that issues from people outside of the new organization get lost. I recommend that we copy all the Issues and Pull requests (including closed ones) for the historical record before transferring. I will try to do that this weekend and put them somewhere (maybe the wiki?). I think all the webhooks get moved too, but I think we'll want to revisit the CI tools anyway. Yes to multiple people for PyPI. I like the suggestion of an lmfit organization, and will look into that. I think all other package management systems will work themselves out. I have not looked at who is maintaining the conda-forge feedstock, but I think that if we can push to PyPI, anything else can use that or Github release. For Python versions: In my opinion, f-strings and (maybe not vital here) dictionaries being ordered make supporting "Python < 3.7" painful. I agree with @andrewgsavage that we officially support (and test) only the supported Python version (currently, 3.8 through 3.12). But we can have a policy that changes made that would intentionally break support for a recent end-of-life version (currently, 3.7) really ought to have a discussion. I don't see many recent features that are likely to impact this library. I think it is also reasonable to try to add testing for new versions when they reach the beta stage. But, we can also be forgiving if we slip a bit behind, and rely on whatever CI we use (a fair topic for discussion). I suspect that a) new versions will generally work and b) adoption of the "latest Python version" is rather slow in scientific/data science work where lots of support libraries are probably needed and have bigger challenges than this library. Thanks! |
From the transfer documentation, it sounds like you might not be able to transfer to an account that has a fork of the repo being transferred, so
When I have worked with transferred repositories before, all the issues were transferred but it doesn't hut to be careful.
For what it's worth, 3.6 has both of these (dictionary ordering was not officially documented in 3.6 but was in place), but it sounds like I am less concerned about supporting old versions than some others here (I don't mind supporting older versions, but I just don't feel it is worth too much extra effort). Personally I am still upgrading projects from 3.8 with the upcoming end of life, but for those still on 3.8 I don't have an expectation that I should be getting the latest features of the libraries I am using. |
Thanks -- that's a good point. There was nothing in FWIW, I did apply for an organization on PyPI named "lmfit". For Python versions: I am in favor of supporting and testing with only supported Python versions, currently Python 3.8 to 3.12. Let's plan to stop testing 3.8 later this year (I think Oct 2024). What I would say though is that we may want to discuss and document when we intentionally break something for an old version. For example, we may someday decide that we want to use |
I missed a point: What is the advantage to using the |
I don't think anyone is advocating for using the fork now. We have been talking about moving this repo into the lmfit organization. It should keep the continuity of the issues/prs/urls but more clearly indicate that project is maintained by the lmfit team now rather than by a single person. I think functionally it is possible to grant full ownership to other people to a personal repository but moving it under an organization makes it a little easier to manage shared access rights and makes the shared maintenance more clear. (I think GitHub makes it hard to tell who owns and has access to a repository intentionally for security reasons). |
@jagerber48 Yes, see https://docs.github.com/en/organizations
All the people in this conversation (@andrewgsavage @wshanks @jagerber48 @lebigot @newville) are already members of the lmfit organization. I am one of 4 Lmfit owners, and we add anyone here to be an owner too. |
Got it, makes sense, thanks both for the clarification! |
@lebigot for consistency, and for all the same reasons as transferring this repo to an organization is a good idea, and as a test of the process, I transferred newville/asteval to lmfit/asteval today. It was very smoothly, took a few minutes, and all issues and pull requests migrated (there was no wiki or discussions to test). So, I think transferring should be safe. I had been using GH Pages as the docs, and that (newille.github.io/asteval) failed to transfer, but this is FWIW, I then forked lmfit/asteval back to newville/asteval. |
According to transfer documentation linked above, you could rename your fork something other than |
@wshanks thanks - actually, GitHub did a very good job of automatically redirecting and updating links. It did less well with the mapping of "gh-pages" from the personal repo to the new organization. But that was easy enough to copy (done first) and then turn into a redirect once the docs were installed for https://lmfit.github.io/asteval. No complaints from me. Anyway, that's not relevant for this repo so I am reasonably sure that it should go smoothly. I did a semi-automated download of the HTML for all the issues and pull requests for |
Thank you for helping with the transfer. I did transfer the repository to Here are some things that you may want to update:
I don't think I have access to the roles settings for the Since keeping a trace of the origin of the project is important to me, I added a very short History section at the bottom of the README (with a hidden ReST note indicating that I would like this origin to remain in the README). For PyPI, can you send me a PyPI user name so that I can make you an owner? Thanks! I was convinced by arguments for supporting Python 3.8+. Let me know if/when you need anything else for the transfer. Thanks again for picking up this project!! |
@lebigot Thanks -- that looks great. It looks like the issues and pull requests transferred too. We'll work on tidying up the authors, readme, etc. FWIW, my PyPI user name is "newville". Hope that another one gets added too. Thanks! |
Thanks. I added you as an owner on PyPI for uncertainties. @wshanks @andrewgsavage @jagerber48: I'd be happy to make you owners too. Just give me a PyPI username (you can send it to eric.lebigot@normalesup.org). |
My PyPI username is While I am posting, I will make one more pitch about forks and transferred repositories because I think what I was saying before might not have been understood. When the repository is transferred, the issues and pull requests are preserved, and links within the issues and pull requests are rewritten with the new location, so for example what linked to |
my pypi username is andrewgsavage |
@wshanks Ah, I get your point better now - thanks for the clarification! |
Thanks @wshanks and @andrewgsavage: I invited you to become owners of the PyPI uncertainties project. @wshanks: I had understood what you mentioned, so you were clear enough, thanks. 😀 If I have to fork |
My PyPi user name is jagerber48 |
@jagerber48: you should have received an invite! |
@wshanks: I sent a new invite, as the previous one expired. |
In the future, @newville, @wshanks, @andrewgsavage, feel free to ping me with @lebigot! I won't be following everything, so please do this when you think I can help. Thanks! |
Thanks, @lebigot. I didn't realize they expired quickly. I will accept soon. |
@lebigot You have called for a maintainer and offered to help with the transition to a new maintainer.
I have multiple projects that depend on this package and would like to make sure this project is maintained. By age alone, I might be closer to retirement than you ;), but I might be willing to fork this to the
lmfit
Github organization (which can have multiple projects and members), and maintain this package from there. I would add you as a member of that if you are willing.If that is acceptable to you, we could then figure out to push to PyPI, and conda-based "feedstocks" and packages.
The text was updated successfully, but these errors were encountered: