Proposal: Merge github.com/ostreedev/ostree and github.com/ostreedev/ostree-rs{,-ext} into this repo

[cgwalters]

This is a followup to https://mail.gnome.org/archives/ostree-list/2021-April/msg00000.html

So far I think the model has been pretty successful (at least for us in rpm-ostree land).

But...I'm already hitting metal complexity + ergonomic issues in dealing with 3 git repos. For example,

Proposed: Create a 2021-lts git branch

Take the current state of this repo, and we will create a "long term support" branch. Anyone who still cannot take a hard dependency on Rust in 2022 can use this branch.

Or, if you are mostly satisfied with the existing feature set and want stability, you can follow this branch. If we execute on this proposal, I'd plan to use this for RHEL8.

Proposed: Fold ostree-rs and ostree-rs-ext into main

In the short term, the main branch still will support the existing C library and binaries as is.

However, in the medium term (e.g. somewhere early in 2022?) we actively take a hard dependency on Rust (and cargo etc) to build and run.

Benefits:

• Being able to version + release a single thing
• Single test suite infrastructure (right now bits are duplicated across multiple repos)
• We can write new key functionality in Rust

What about non-Rust consumers?

A key question here is whether libostree-1.so depends on Rust. If we do, it means that projects like rpm-ostree will end up linking in Rust code twice, which is going to lead to bloat and problems.

In this model, we would probably need to do soname bumps corresponding to the Rust crate semver bumps.

One possibility here is that the shared library calls back into Rust via weak symbols. So the Rust code gets compiled in the consumer binary (e.g. rpm-ostree).

[AdrianVovk]

Would the gobject-introspection api stick around or continue to get extended with new features? I'm not interested in porting os-updated to rust...

Edit: ostree itself being written in rust is a-ok 👍 just want to make sure non-rust consumers can stick around

[cgwalters]

Thanks for the feedback!

Would the gobject-introspection api stick around or continue to get extended with new features? I'm not interested in porting os-updated to rust...

Yeah, I was trying to touch on this in the last proposal but did so indirectly.

I think a pivotal question indeed is "do we break the libostree API or not". I had been leaning towards not breaking the API, and this feedback strengthens that. (Also if we did break the C API this would be a parallel-installable library, but let's keep that out of scope for now)

[jlebon]

As a first step, I think folding the -rs* repos into this one makes total sense for all the reasons mentioned. It also makes clearer the focus on Rust going forward.

I'm wary of an LTS branch, but I see the appeal. I think we need clarification on the status of Rust support in the various toolchains libostree consumers use so we better understand what kind of "long-term support" we'd be looking at.

[cgwalters]

As a first step, I think folding the -rs* repos into this one makes total sense for all the reasons mentioned. It also makes clearer the focus on Rust going forward.

A mix of the tradeoffs there. That'd help some, but for example it wouldn't help the issue of duplicate test suites which is mainly problematic between ostree-rs-ext and ostree for me right now.

[cgwalters]

This would also greatly help solve #2480

[cgwalters]

OK copying from #2575 (comment) - a rhel8 branch exists now which is forked from 2022.2.

[cgwalters]

@smcv can Debian ship an ostree that depends on Rust?

[smcv]

Debian can ship libraries that depend on Rust, and we already do for librsvg and mozjs. All release architectures (the ones the release team keeps track of) have a working rustc and cargo. Retrocomputing enthusiasts will complain about non-release architectures like m68k being left behind, but they already do that.

I, personally, should not be the sole Debian maintainer of a library that depends on Rust, because I don't know Rust (I know I need to learn it, but there are only so many hours in a day), and Debian package maintainers should in principle be domain experts who would be able to fork the upstream package if it became necessary. At the moment I am the single point of failure for libostree in Debian (in theory I have two co-maintainers, but I think they have only done one upload between them, in 2016), so I will need to find a Rust-literate co-maintainer and/or hand off maintenance to someone who knows what they're doing.

The Debian Rust team is currently trying to handle Rust in the same way as C/C++, packaging each crate as a separate Debian source package (which means a crate cannot be in Debian until all its dependencies have individually been packaged and got past archive administrator review, and a crate with new dependencies cannot be in Debian until all the new dependencies have had that treatment). I think that's unsustainable, because our packaging system is designed for medium-sized C/C++ libraries (somewhere between the sizes of zlib and GTK) and has poor scalability for packages that are much smaller or larger than that range, but it's not my decision, because I am not the one doing the work. If I understand correctly, Fedora has the same structural problems around Rust.

For a few packages that mix C/C++ and Rust, like librsvg and mozjs, we are currently using the vendored copies of Rust dependencies as-is, rather than build-depending on the whole recursive dependency tree. This is arguably a policy violation, but seems like the only realistic way to continue to have those packages in the distribution. Checking an updated version of one of these heavily-vendored packages for copyright issues or malicious changes can take several hours, so I'm not delighted about the prospect of doing so, but if that's what it takes then I'll have to make the time.

To the best of my knowledge, the cargo toolchain and/or Debian's tooling around it does not have the ability to operate between those extremes, with some but not all dependencies taken from separately-packaged system libraries, and the remaining dependencies taken from vendored copies.

[cgwalters]

because I don't know Rust (I know I need to learn it, but there are only so many hours in a day),

I am happy to help btw! Seriously, if you decide to spend some time on this, don't hesitate to reach out with any random questions! The broader community is also friendly and helpful.

For a few packages that mix C/C++ and Rust, like librsvg and mozjs, we are currently using the vendored copies of Rust dependencies as-is, rather than build-depending on the whole recursive dependency tree. This is arguably a policy violation, but seems like the only realistic way to continue to have those packages in the distribution. Checking an updated version of one of these heavily-vendored packages for copyright issues or malicious changes can take several hours, so I'm not delighted about the prospect of doing so, but if that's what it takes then I'll have to make the time.

Right. Though this is something I also think we should take a strong stance on upstream. I just did ostreedev/ostree-rs-ext#291 related to this.

[dbnicholson]

I, personally, should not be the sole Debian maintainer of a library that depends on Rust, because I don't know Rust (I know I need to learn it, but there are only so many hours in a day), and Debian package maintainers should in principle be domain experts who would be able to fork the upstream package if it became necessary. At the moment I am the single point of failure for libostree in Debian (in theory I have two co-maintainers, but I think they have only done one upload between them, in 2016), so I will need to find a Rust-literate co-maintainer and/or hand off maintenance to someone who knows what they're doing.

I'm pretty novice with rust myself, but I think I could help you there. I'm the person that does the ostree maintenance for Endless and we use Debian's packaging. I sent an email to your Debian address a couple months ago with this in mind.

[cgwalters]

This came up again, and my latest thoughts are that basically we take the very small step of building an internal /usr/lib/ostree/libostree-rs-internals.so that is only linked into the process for things related to OstreeSysroot to start. This would mean that it wouldn't affect flatpak, and could be turned off or segregated into a separate e.g. ostree-boot package alongside the other things like the dracut module and ostree-prepare-root etc.

[cgwalters]

cc #3054