Publishing the core extensions

[robinkrahl]

How should the core extensions be published? In my opinion, they should be published the same way as nitrocli, both for usability and maintainability reasons. So I see three options:

• Ditch crates.io for all our releases. We should have good reasons for this step, and I don’t see any.
• Publish nitrocli and all core extensions as separate crates on crates.io. This is how I understand our current approach.
• Have the core extensions as additional binary targets for the nitrocli crate (maybe behind feature flags) and release them together with nitrocli. This might be easier to maintain than separate crates, but then we wouldn’t really need the extension mechanism any more.

[d-e-s-o]

So far my thinking was to not publish them to crates.io. We'd still be publishing nitrocli, though. nitrocli has started off being published there and it is useful by itself.

Extensions change the picture: for one, they (generally) are not useful without nitrocli. So now there is a dependency. They are also not entirely trivial to make work with the main program: you need to install them at a proper location. In my opinion that is the job of some kind of distribution package, to take care of all of that.

I understand (hope?) that all Linux distributions that we support build binary packages from source. And by that I mean the actual repository contents, not the archive uploaded to crates.io. So nothing changes for them and they don't need anything published to crates.io.
I don't know about Mac & Windows, but I am also reluctant to call them supported to begin with. In any case, the option to build from source is always available and just having something on crates.io in my opinion only helps marginally.

Of course, publishing extensions there also does not hurt much. And I am not fundamentally against publishing them. I just think that given that there is more involved to getting them to work than what cargo can do out of the box, taking care of having proper packages seems more important. So from my perspective, publishing of extensions is not necessary but we can go down that route if there is a convincing use case.

[d-e-s-o]

Do you have concrete use cases for publishing to crates.io or is that more a "we should do it because it seems to be the right thing to do" kind of stance, @robinkrahl? As you can probably tell, I am much less opposed to publishing core extensions than I am to publishing nitrocli-ext, so I am trying to get a better picture here.

[robinkrahl]

I understand (hope?) that all Linux distributions that we support build binary packages from source. And by that I mean the actual repository contents, not the archive uploaded to crates.io.

The Debian packages are built from the releases published to crates.io.

They are also not entirely trivial to make work with the main program: you need to install them at a proper location.

Well, the same is true for nitrocli, no? And if cargo install nitrocli installs nitrocli in the PATH, so does cargo install nitrocli-otp-cache. What extra work is necessary?

[robinkrahl]

Do you have concrete use cases for publishing to crates.io or is that more a "we should do it because it seems to be the right thing to do" kind of stance, @robinkrahl? As you can probably tell, I am much less opposed to publishing core extensions than I am to publishing nitrocli-ext, so I am trying to get a better picture here.

For me, publishing to crates.io would be the default as it is how we started off. And I see several benefits with it, for example full build reproducibility and an archive of all published versions, not dependent on this repository which might disappear, be force-pushed to or change its internal structure; also simple installation of the extensions in a manner consistent with nitrocli.

[robinkrahl]

And to repeat my point from the first post: If we on the hand think that the core extensions should be stable and well-tested, as established in #144, and on the other hand, we don’t intend to publish them on crates.io – why do we need separate crates in the first place? Why not have additional binary targets in nitrocli? Or just have subcommands behind a feature flag?

[d-e-s-o]

Fair enough. If some distribution consume contents for crates.io by default then we should consider publishing them.

So what should the versioning story be? I suspect you are then also insisting on versioning them independently of nitrocli? If so, how would we ensure compatibility with the main program? How would users know?

Do you have concrete use cases for publishing to crates.io or is that more a "we should do it because it seems to be the right thing to do" kind of stance, @robinkrahl? As you can probably tell, I am much less opposed to publishing core extensions than I am to publishing nitrocli-ext, so I am trying to get a better picture here.

For me, publishing to crates.io would be the default as it is how we started off. And I see several benefits with it, for example full build reproducibility and an archive of all published versions, not dependent on this repository which might disappear, be force-pushed to or change its internal structure; also simple installation of the extensions in a manner consistent with nitrocli.

I can agree with it being another mirror that could be fallen back to. But anything trust related is off the table in my opinion. Nothing is signed, there is little to no guarantee of tamper detection (yes, there is some checksumming here and there, but if you don't know what it should be [e.g., by having a lock file lying around], you can't tell whether anything changed).

And to repeat my point from the first post: If we on the hand think that the core extensions should be stable and well-tested, as established in #144, and on the other hand, we don’t intend to publish them on crates.io – why do we need separate crates in the first place? Why not have additional binary targets in nitrocli? Or just have subcommands behind a feature flag?

Yes, we could choose to make them mere binaries. The downside to me is that things become very monolithic. That does not change anything about the fact that they are built on top of the extension mechanism, the entire premise of which (the way I understood it), was to provide a polyglot way for other parties to add functionality to nitrocli that was unlikely to be included in the program itself. Whether or not we contain some of those extensions (be it as examples or actually "useful" functionality) with the main program does not invalidate that.

[robinkrahl]

So what should the versioning story be? I suspect you are then also insisting on versioning them independently of nitrocli? If so, how would we ensure compatibility with the main program? How would users know?

I haven’t made up my mind yet. And I don’t think that this question only applies to publishing on crates.io. Even if we don’t use crates.io, we need a versioning model.

One option would be to just release if necessary and to document the compatible versions in the readme.

Another option would be to have one extension release per breaking nitrocli release. The version numbers could be synchronized, i. e. we release an extension version 0.5.0 when we release nitrocli 0.5.0, but we don’t have to release an extension version 0.5.1 with nitrocli 0.5.1. This would be similar to the versioning model I use for nitrokey-sys-rs. Arguably, using additional binaries that are released together with nitrocli would solve the versioning issue more elegantly.

But anything trust related is off the table in my opinion.

I agree that crates.io cannot be a source of trust, but it can act as a part of a trusted system, just like we use it to resolve our dependencies. But I don’t think we have to discuss this here.

[robinkrahl]

The more I think about it, the more I like the idea of having the core extensions as additional binary targets of the nitrocli crate. Yes, it would be slightly more monolithic, but it would still be clearly separate from the core crate due to being a different target and maybe behind a feature flag (and still using the extension mechanism).

But we wouldn’t have to deal with separate crates, their distribution strategy, their versioning strategy etc. We would have a much smaller risk of compatibility issues. We could even have nitrocli-ext as an internal module (though I would still be in favor of releasing it eventually) and we could share some data structures between the extensions and the core without having to make them part of a public API.

The distributions could decide how to package the extensions. For example, in Debian, we would probably have the rust-nitrocli source package that provides the nitrocli and nitrocli-otp-cache binary packages that can be installed separately by the user. In Arch, I would probably ship everything with nitrocli.

[d-e-s-o]

The more I think about it, the more I like the idea of having the core extensions as additional binary targets of the nitrocli crate.

Yeah, I tend to agree: it's the lesser evil.

We could even have nitrocli-ext as an internal module (though I would still be in favor of releasing it eventually) and we could share some data structures between the extensions and the core without having to make them part of a public API.

Sounds good. I am not fundamentally against releasing it down the line if we at least have some more use cases/suspected consumers, I just don't want to rush it. It is true that nothing is set in stone API wise and we can always push a new version, but to me every published API is a commitment in one form or another, and the more of those we have the more constrained we become/the more things we need to consider. So short to midterm I would even go so far to say that the relevant code could just be duplicated on the user's end. One added advantage I see of not publishing it early on is that we set a bit of an incentive for users to at least consider bringing up the possibility of making what they are writing a core extension, which gives us a better overview of what is going on.

[d-e-s-o]

I think we can close this issue now. Let's just open a new one once we get to the point of thinking about separate crates and all the complications they entail again.