Automate Arch deployment

[wren]

Feature Request

Use Case/Motivation

There is currently a listing for jrnl in the official arch repo (https://www.archlinux.org/packages/community/any/jrnl/). This is great! We didn't have to do anything, and some kind soul is doing this work for us.

What I would like is to make contact with the maintainer of this listing to see if there's anything we can do to help. Ideally, we should be able to put something into our CI pipeline that will streamline the release process on Arch, and get updates to users faster.

This is similar #686, but for Arch instead of Homebrew. We've gotten a lot of interest from Arch in the past, and I'm happy that we're getting to the point where we can support the native package manager more easily.

Example Usage

Basically what it already looks like now, but automated in the CI so new releases get to users faster.

[wren]

After some research, it looks like a contributor to jrnl (@felixonmars) is the maintainer of the Arch listing.

First off, I want to give a huge thanks to @felixonmars for doing this work! Like a really big, huge thank you. We've had some troubles in the past maintaining an Arch listing, and it was a very pleasant surprise to find that you'd gone ahead and done all the work for us already. So, again, thank you!

Secondly, is there any way we can help? I'm happy to do the work of automating this into our CI, but I would just need a bit of Arch-specific guidance from you.

[felixonmars]

Thanks for reaching out :)

Arch always follows the latest release quickly, so IMHO it doesn't make much sense to provide a separate build just to provide new versions a little bit faster. However, since Arch applies this to all or jrnl's dependencies as well, it would be greatly appreciated if jrnl always adapts to its dependencies' changes quickly and uses the latest versions.

[wren]

Thanks for reaching out :)

Of course, @felixonmars!

I have a few questions, about the process, if you have some time. I'm sorry if any these don't make sense, but I'm really not very familiar with Arch's update system, so I would really appreciate any help you're willing to provide.

• Updating dependencies quickly: I think you're very right about this. We just finished a dependency audit (Conduct a dependency audit #1052). We went through each of jrnl's dependencies and updated the required versions as much as we could given each project's update policy. We were also considering adding something like dependabot (or similar) to help notify us when dependencies update. Do you think that would be enough to keep up with Arch's updates?

• Updating Arch listing quickly: As you said, Arch does seem to usually update quickly. For jrnl, though, v2.5 took about over three weeks before showing up in Arch. I understand if a separate build doesn't make sense, but is there any way to speed up this process? Is there something like a notification we can send that a new version is out? To be absolutely clear, I'm not complaining here. Your work is much better than what we had before, so I'm just looking for where we can improve. My goal would be to have same-day (or as close as possible) updates in Arch whenever we release a new version of jrnl. Do you think that's possible?

• Testing: From what I see in the Arch listing, only pytest is being used to run the check. Most of jrnl's tests, though, use behave, and pytest only has a few miscellaneous unit tests. Is behave supported? If not, do we need to move those tests to pytest for compatibility? I'm worried that there might be a release that's broken on Arch, and it won't be caught in time because of not running all the tests.

• Changing dependencies: Because of the dependency audit I mentioned above, the next version of jrnl (v2.6) has some changes in dependencies. This raises the question about what's required for Arch to keep the list of dependencies synced with jrnl. If we add a new dependency, does the Arch listing need to be manually updated? Does dephell do that work? Is there anything we can do to help with this?

Sorry to bombard you with so many questions, but we really want to make sure that Arch (and other distros, soon) get first class support for jrnl, and you're the first knowledgeable Arch person that seems willing to help. Thanks, again!

[felixonmars]

I have a few questions, about the process, if you have some time. I'm sorry if any these don't make sense, but I'm really not very familiar with Arch's update system, so I would really appreciate any help you're willing to provide.

• Updating dependencies quickly: I think you're very right about this. We just finished a dependency audit (Conduct a dependency audit #1052). We went through each of jrnl's dependencies and updated the required versions as much as we could given each project's update policy. We were also considering adding something like dependabot (or similar) to help notify us when dependencies update. Do you think that would be enough to keep up with Arch's updates?

This sounds exactly great! Yes I think this would be enough.

• Updating Arch listing quickly: As you said, Arch does seem to usually update quickly. For jrnl, though, v2.5 took about over three weeks before showing up in Arch. I understand if a separate build doesn't make sense, but is there any way to speed up this process? Is there something like a notification we can send that a new version is out? To be absolutely clear, I'm not complaining here. Your work is much better than what we had before, so I'm just looking for where we can improve. My goal would be to have same-day (or as close as possible) updates in Arch whenever we release a new version of jrnl. Do you think that's possible?

I use nvchecker to track updates myself, and jrnl is currently monitored from the pypi version:
https://github.com/felixonmars/dotfiles/blob/479176977c90b117192ebb00836b05367c879421/nvchecker.toml#L3932

Since I maintain way too many packages currently, I have a "less troublesome or upstream more willingly to help" list of them in memory so I update them faster :P (yes, responsive upstreams will surely get bumped!)

Just in case this doesn't work, the normal process, out-of-date flags, could be utilized too (on the package page):

https://www.archlinux.org/packages/community/any/jrnl/

(This can be easily automated, it seems.)

• Testing: From what I see in the Arch listing, only pytest is being used to run the check. Most of jrnl's tests, though, use behave, and pytest only has a few miscellaneous unit tests. Is behave supported? If not, do we need to move those tests to pytest for compatibility? I'm worried that there might be a release that's broken on Arch, and it won't be caught in time because of not running all the tests.

You are right, I actually overlooked this. Arch currently doesn't have behave packaged, so I'll first take a look at it this way. I'll report back later if behave somehow doesn't work.

• Changing dependencies: Because of the dependency audit I mentioned above, the next version of jrnl (v2.6) has some changes in dependencies. This raises the question about what's required for Arch to keep the list of dependencies synced with jrnl. If we add a new dependency, does the Arch listing need to be manually updated? Does dephell do that work? Is there anything we can do to help with this?

I use a very simple script to compare the metadata of each release, so it won't cause too much trouble as long as the metadata format doesn't change too often (like from setup.cfg to pyproject.toml, etc). There is no tooling to automate this in Arch currently, so this is indeed manual work. Please do let me know if I missed something.

[felixonmars]

UPDATE: I have packaged behave for Arch and enabled it for jrnl. Currently all tests are passing.

archlinux/svntogit-community@f2e5bf6

[wren]

I have a few questions, about the process, if you have some time. I'm sorry if any these don't make sense, but I'm really not very familiar with Arch's update system, so I would really appreciate any help you're willing to provide.

• Updating dependencies quickly: I think you're very right about this. We just finished a dependency audit (Conduct a dependency audit #1052). We went through each of jrnl's dependencies and updated the required versions as much as we could given each project's update policy. We were also considering adding something like dependabot (or similar) to help notify us when dependencies update. Do you think that would be enough to keep up with Arch's updates?

This sounds exactly great! Yes I think this would be enough.

Okay, sounds good. I made an issue to keep track of this (#1120).

• Updating Arch listing quickly: As you said, Arch does seem to usually update quickly. For jrnl, though, v2.5 took about over three weeks before showing up in Arch. I understand if a separate build doesn't make sense, but is there any way to speed up this process? Is there something like a notification we can send that a new version is out? To be absolutely clear, I'm not complaining here. Your work is much better than what we had before, so I'm just looking for where we can improve. My goal would be to have same-day (or as close as possible) updates in Arch whenever we release a new version of jrnl. Do you think that's possible?

I use nvchecker to track updates myself, and jrnl is currently monitored from the pypi version:
https://github.com/felixonmars/dotfiles/blob/479176977c90b117192ebb00836b05367c879421/nvchecker.toml#L3932

Since I maintain way too many packages currently, I have a "less troublesome or upstream more willingly to help" list of them in memory so I update them faster :P (yes, responsive upstreams will surely get bumped!)

Haha, that's fair. Hopefully you'll keep jrnl in that list of helpful and responsive upstreams. 😄

Just in case this doesn't work, the normal process, out-of-date flags, could be utilized too (on the package page):

https://www.archlinux.org/packages/community/any/jrnl/

(This can be easily automated, it seems.)

This is interesting. I don't know the etiquette around this, but would it be okay to flag jrnl as out-of-date as part of our release process? Or would that just be annoying instead of actually useful?

• Testing: From what I see in the Arch listing, only pytest is being used to run the check. Most of jrnl's tests, though, use behave, and pytest only has a few miscellaneous unit tests. Is behave supported? If not, do we need to move those tests to pytest for compatibility? I'm worried that there might be a release that's broken on Arch, and it won't be caught in time because of not running all the tests.

You are right, I actually overlooked this. Arch currently doesn't have behave packaged, so I'll first take a look at it this way. I'll report back later if behave somehow doesn't work.

No worries, and thank you. Also, I don't know how much this affects what you're doing, but we've been considering moving from behave to pytest-bdd. Would that make your process easier? We could maybe prioritize that work, if doing so would reduce the duplicated effort. Would that be helpful?

• Changing dependencies: Because of the dependency audit I mentioned above, the next version of jrnl (v2.6) has some changes in dependencies. This raises the question about what's required for Arch to keep the list of dependencies synced with jrnl. If we add a new dependency, does the Arch listing need to be manually updated? Does dephell do that work? Is there anything we can do to help with this?

I use a very simple script to compare the metadata of each release, so it won't cause too much trouble as long as the metadata format doesn't change too often (like from setup.cfg to pyproject.toml, etc). There is no tooling to automate this in Arch currently, so this is indeed manual work. Please do let me know if I missed something.

Is there any way we can help with this manual part of the process?

Maybe we could generate a PR and send it over to you? That way you could still review before merging, but you wouldn't have to do the manual work. It looks like there are some python packages geared towards generating PKGBUILD files (like pip2pkgbuild or others), so maybe we could use one of those. What do you think about that?

[wren]

Oh, nice!

UPDATE: I have packaged behave for Arch and enabled it for jrnl. Currently all tests are passing.

archlinux/svntogit-community@f2e5bf6

Nice! That's good news.

[felixonmars]

Just in case this doesn't work, the normal process, out-of-date flags, could be utilized too (on the package page):
https://www.archlinux.org/packages/community/any/jrnl/
(This can be easily automated, it seems.)

This is interesting. I don't know the etiquette around this, but would it be okay to flag jrnl as out-of-date as part of our release process? Or would that just be annoying instead of actually useful?

It would be less useful as I am the only maintainer of the package, currently. But perhaps it would make sense if there are someone else willing to help me.

You are right, I actually overlooked this. Arch currently doesn't have behave packaged, so I'll first take a look at it this way. I'll report back later if behave somehow doesn't work.

No worries, and thank you. Also, I don't know how much this affects what you're doing, but we've been considering moving from behave to pytest-bdd. Would that make your process easier? We could maybe prioritize that work, if doing so would reduce the duplicated effort. Would that be helpful?

I have added behave already :P And turns out we don't have pytest-bdd either. It wouldn't be a problem to us as long as that project is well maintained upstream.

I use a very simple script to compare the metadata of each release, so it won't cause too much trouble as long as the metadata format doesn't change too often (like from setup.cfg to pyproject.toml, etc). There is no tooling to automate this in Arch currently, so this is indeed manual work. Please do let me know if I missed something.

Is there any way we can help with this manual part of the process?

Maybe we could generate a PR and send it over to you? That way you could still review before merging, but you wouldn't have to do the manual work. It looks like there are some python packages geared towards generating PKGBUILD files (like pip2pkgbuild or others), so maybe we could use one of those. What do you think about that?

Our svntogit repositories are converted from SVN, so unfortunately we cannot accept PRs there. An idea would be to provide a section about packaging in the changelog, with entries like newly introduced dependency, etc.

[wren]

Thinking this through, and I still think we could make this work (github actions supports svn along with git), if you're interested.

As for additional maintainers, that's pretty much what I'm volunteering for. I couldn't be the sole maintainer (I don't know enough about arch), but would be happy to help get it set up for automation, and troubleshoot any problems that happen down the road.

What do you think?

[wren]

@felixonmars Hi! Just letting you know that we've implemented dependabot (and have merged in a few PRs from the bot already). We also added a new "packaging" section in the changelog like you asked for.

We're still happy to co-maintain and automate releases to arch if you'd like. But if you prefer to keep it like this, that's fine, too. Just let us know.

[felixonmars]

@felixonmars Hi! Just letting you know that we've implemented dependabot (and have merged in a few PRs from the bot already). We also added a new "packaging" section in the changelog like you asked for.

Thanks! It would be very helpful.

We're still happy to co-maintain and automate releases to arch if you'd like. But if you prefer to keep it like this, that's fine, too. Just let us know.

Sorry for the late response. Unfortunately we don't have a clear or easy method to cooperate on official repository packages (co-maintainers are limited to Arch members), so for now I think it would be more preferred to keep it as-is. Thanks a lot for trying to help though!

[wren]

@felixonmars No worries. I'm going to close this issue out since we've done everything we can. But please let us know if there's anything else we can do to help. And thanks again for taking this on!