PySlurm Future and Maintaining the Project

[tazend]

Hi @giovtorres @gingergeeks

(as per suggestion, I moved this from this issue to here - didn't know team discussions were a thing, but fits a bit better for this definitely :)

just wanted to get out some thoughts about the current state of the project.

In general, the current codebase definitely needs a bit of a rework/improvements and especially the documentation and PyPi releases are very much outdated (as mentioned here)

For quite some time I've been working on lots of changes to improve/rework the current API, that I feel like are now finally ready (#224 and almost #269), with I think a solid foundation to also rework other parts of the API more easily and quickly.

However, I am a bit hesitant to merge these changes, mostly because of the uncertainty about the projects future. And especially for the fact that even if the changes were merged, users would not be able to use them properly right
now without documentation. And I believe I do not have the permission to update the documentation on https://pyslurm.github.io/
This has been talked about here as mentioned, but unfortunately my multiple attempts to reach out about the permission to push documentation/PyPi releases have not been answered, which has gotten a bit frustrating lately ;(

With that and realizing that you guys have been inactive for quite some time now within the project, I'd like to ask the following question:

What do you think about the idea of officially passing on the PySlurm project to me as the new Maintainer?

As mentioned, I've already spent quite some time on the project and have gathered a lot of experience with the Slurm API itself, and would therefore like to continue to improve/support the project (as it also has become a passion project for me).
I feel like the project is in a state where it definitely needs active development/maintenance, especially to improve the codebase (making it also easier for the community to contribute) and to update the old documentation/pypi releases.

Please let me know what you think about my proposal.

[gingergeeks]

Took me a while to find which discussion but that's better 👍

• I agree the code base needs rework - PySlurm originally spawned from a 30 minute hack to get a job done that the CLI could not provide and to be honest I'm not proud of the fact that I did not go back and redo it 👎
• I have noticed the amount of work you have put in and it is much appreciated.
• Yes, both @giovtorres and myself have been inactive but I'm very hoping to get back into the fray later this year.
• I have added you as a maintainer for now to both PySlurm and pyslurm.github.io, if that does not work for you that we can move you to admin if needed.
• At some point the Slurm REST API will become more mature and have better coverage. At this point Giovanni and I did discuss having PySlurm as a wrapper to REST API. At the moment I've looking to have a lsjob utility on the HPC desktop that will have 3 underlying mechanisms - CLI (ssh), PySlurm (ssh), REST (http) but the generated data should be the same for all three and output via JSON.
• We should arrange a video call at some point - Giovanni and I did this a few times but we have not arranged one in years given he moved from HPC to move dev-ops role.

Regards

[gingergeeks]

I forgot to add ....

• Remove BlueGene support - I haven't checked if BG support is still valid in the latest releases
• Could we provide a precompiled PySlurm entity ? I wonder if this is is very much dependent on how Slurm was build and with which options enabled ?
• Resolve the packages identified in https://github.com/PySlurm/pyslurm/security/dependabot
• Also looks like we need to use organisation discussion not team discussion :( that's progress for you !

[tazend]

Hi - appreciate your response @gingergeeks, and I'm happy to hear that you plan on jumping back into the project soon!
Yeah I think a video call at some point would be nice :)

Thank you for adding me as a Maintainer for both projects - I will check to see if I am able to update the documentation.
In regards to the changes I mentioned I have prepared:

The changes I've made of course deviate from the current API in place, but I think its a necessary change and with the correct documentation it will be easier in the future for the end-user to use.
I've also written it in a way that both APIs are able to live alongside each other without interference, so a User would be able to opt-in to the new API if wanted, without abruptly replacing current parts of the API (we could mark it as deprecated in the documentation first, and at some point in the future remove the old API if everything is replaced by a reworked counterpart)

If you are OK with it, I plan to merge the changes soon (can add you as a reviewer of course first if you'd like to). From that point on I would continue touching also other Parts of the API, like Partitions, Reservations, the whole Database accounting stuff etc...If you'd like to, I would definitely keep you in the loop about these changes by adding you as a reviewer when I plan to make such merge-requests :)

---

In regards to bluegene and precompilation, I would suggest discussing this in seperate issues. I have also thought about the possibility of precompilation some time ago, but there are ups and downs to it.

[gingergeeks]

• I'm not currently in a position to review the code and that's unlikely to change in the short-term and I need to sort out my container instances.
• If you have tested why you are proposing then go ahead and merge
• Yes, you should announce possible breakage on the community announcement/discussion. Having the APIs live side by side is good.
• Let me know if you run into issues in either repo
• I assume you wish to move from Sphinx to mkdocs
• We should fix the highlighted security fixes from GitHub mentioned (setuptools & pytest newer version requirements)
• Giovanni's build system seems to be failing - do we know why ?
• We did have some people using Python2.x and I did try to sunset that support :(
• I will now move this to the organisation discussion as I have a warning about it.

[tazend]

Hi,

alright, I will then go ahead and make everything ready for a merge, update the docs and announce it properly :)

I assume you wish to move from Sphinx to mkdocs

yes things are already in place for mkdocs, and I think where it especially shines for our use is that multi-version support for the documentation is pretty easy to achieve and deploy to the github.io page. Being able to switch between multiple-versions of pyslurm documentation should be very nice for the users, considering that many sites are not always updating to the latest slurm version straightaway.

We should fix the highlighted security fixes from GitHub mentioned (setuptools & pytest newer version requirements)

Definitely, I'll check it out :)

Giovanni's build system seems to be failing - do we know why ?

It fails mostly because the latest container available on dockerhub that is used in the CI pipeline is with slurm 21.08. This is the repository - I already have made some changes and bumped the version on my test-setup, and I hope to make a pull-request with those changes soon. However, for pushing a new version to dockerhub I believe we need @giovtorres
Right now I believe only the tests are running in the container, although I think that they might fail anyway even if the version is up to date, as they haven't been touched in a long time. With the new API changes I mentioned I also have created a new set of tests, so I'll also try to improve it there a bit :)