bpftrace should have a strong dependency on libbpf

[BurntBrunch]

While working on #2321, I had to manually - and poorly - reimplement functionality that's already in libbpf in a more complete form. (namely, subprogram and func_info relocation)

I'm opening this issue to discuss making libbpf a required dependency, especially given its imminent 1.0 release. This dependency would allow us to

1. Pass in-memory ELF files to libbpf, removing all of the current orc jit hacks
2. Not worry about relocations and other BTF minutiae
3. Make the object files emitted by --emit-elf loadable by other tools (!)
4. Open a clean path to CO-RE usage (!)
5. Potentially simplify USDT handling by relying on libbpf's support
6. Simplify feature probing by relying on libbpf's probes and fallbacks

Obeying the libbpf layout requires some work (e.g. BTF-based map definitions and the very particular section naming conventions) but honestly that's probably more shallow a rabbit hole than what I endured to get #2321 to the finish line.

I'm opening this issue primarily because I don't know the project's policy on build dependencies and would love to discuss any objections to this change.

[viktormalik]

Hi @BurntBrunch, thanks for opening this!

libbpf is currently required for building bpftrace (since #2265) so you can freely use functions that it provides (although we currently support also older libbpf versions). On the other hand, many of the ideas above are great and we should consider implementing them eventually. I'd suggest to continue discussion in this thread and then open separate issues for each of the features.

[fbs]

Yeah no objections to any of this. Moving to libbpf will a lost of things simpler and standardised :)

[BurntBrunch]

Alright, I'm glad this is well-received! :)

How do we go about formalizing the min version? Is that strictly in the cmake config or do we do it anywhere else?
I'd love to make the version requirements pretty high (e.g. 0.8.1 or the upcoming 1.0.0) to avoid distros getting by with much older libbpf than is advisable.

Also, does this mean we can clean up all the HAVE_LIBBPF_* checks and fallbacks? I really don't think we care about a libbpf version that doesn't have bpf_prog_load, for example (if one ever existed).

[fbs]

While from a development perspective only bleeding edge would be great we do try to support a wide range of kernels and library versions. Stable distros as still quite popular, so I'd say as low as we can reasonably go.

It's tricky, bpf moves way to fast for the distros (and us) to keep up

[danobi]

I am open to this as well. I evaluated the same idea ~1 year ago but stopped short of implementing for a few reasons:

1. bpftrace does some tricky things and might do more tricky things in the future. One avenue of concern is that libbpf may lack support for various advanced functionality. But going through libbpf might delay when bpftrace users are able to actually use the new feature (for lack of a new libbpf version). Or if even libbpf is amenable to such specialized APIs.
2. The above concern can be addressed by vendoring libbpf (for which Andrii and Alexei are in strong support of). However, distro packagers generally do not like submodules or vendored libraries in packages. I'm personally sympathetic to distros about this but don't necessarily agree. If you look through the bpftrace issue tracker there should be some issues where bpftrace is built in a network-disabled environment and we had to add some cmake flags. I would feel better about this if we got input from packagers. Could be as easy as looking through package info and emailing various package maintainers. (cc @davide125 who may have input here).
3. It is going to be quite a lot of plumbing work. However, if you are signing up for it I wouldn't mind :)
4. It may break bpftrace in old/weird environments. We don't have the best test coverage for some weird edge cases esp. b/c we've taken drive by fixes here and there for env issues over the years. But I don't think this point should be a blocker, esp. if this is the price of progress.

[danobi]

To expand, I think vendoring libbpf is really the only sane long term solution. It is more or less inevitable that we will have to carry some out of tree libbpf patches at some point, especially if we punt all loading operations to libbpf. And it is also a very practical way to keep up with the rapid pace of bpf development (esp. in the interest of end user experience).

For example, rust vendors llvm for most likely similar reasons.

[fbs]

I agree, vendoring seems like the best way to go forward, also considering our limited dev power.

I do think the end goal should be to 'unvendor' it again when (if ever) everything stabilizes. Or atleast track stable releases at a pace that works for distros as well. Doing so should also prevent collecting patches that never get upstreamed

[BurntBrunch]

I do think the end goal should be to 'unvendor' it again when (if ever) everything stabilizes

That's kinda why I suggested 1.0 as the min version - it's supposed to be the last big API cleanup before "stability" is a thing. It's been baking for quite a while.

Now, requiring 1.0 would also require that distros update all their stragglers but overall I don't think it's an unreasonable design decision at this point in time.

bpftrace does some tricky things and might do more tricky things in the future. One avenue of concern is that libbpf may lack support for various advanced functionality.

I'd need examples but the thing about libbpf is that you can always bypass it. You'd have to do some of its job yourself (as I found myself doing) but there's no vendor lock-in per se. It just lowers the bar for most new features while keeping the bar for "exotic" features as-is.

To expand, I think vendoring libbpf is really the only sane long term solution. ... It is more or less inevitable that we will have to carry some out of tree libbpf patches at some point

Maybe? Beyond bug fixes that haven't landed yet, I don't see why we'd want to carry out of tree libbpf patches.

Besides, I can't predict the future - all I know is that libbpf can today simplify bpftrace in the particular ways I described above. Beyond small things like exposing more helpers (we can just keep our own header), whether it's vendored or min-versioned at something high enough to be considered stable, I don't particularly mind.

While from a development perspective only bleeding edge would be great we do try to support a wide range of kernels and library versions.

Are the library versions formalized anywhere? I tried looking but I'm not seeing it clearly stated in any docs. If we can say "last X Ubuntu LTS releases + last X centos etc," then we can get very precise about the libbpf API coverage we'd have and the bugs we'd have to avoid.

That said, I also don't think it's particularly unreasonable to say "yes, we support latest bpftrace on Ubuntu 16.04 but you need to use an AppImage/static build/flatpak/snap/whatever." If these distros want latest bpftrace packaged in-repo, they'd have to go through the trouble of also updating its dependencies.

I guess I don't understand how being able to build against LLVM 6 on a stable distro helps people in practice - after all, the distro is not going to push bpftrace updates anymore. Is this purely for local builds against system libraries? Or for copr/etc builds in user repos? Isn't that better addressed by the official AppImage/static builds/etc?

Sorry if these questions are obvious, I am just trying to build a complete mental model here :)

[davide125]

The above concern can be addressed by vendoring libbpf (for which Andrii and Alexei are in strong support of). However, distro packagers generally do not like submodules or vendored libraries in packages. I'm personally sympathetic to distros about this but don't necessarily agree. If you look through the bpftrace issue tracker there should be some issues where bpftrace is built in a network-disabled environment and we had to add some cmake flags. I would feel better about this if we got input from packagers. Could be as easy as looking through package info and emailing various package maintainers. (cc @davide125 who may have input here).

Yes, vendoring / bundling library dependencies is problematic for distributions and it should be avoided if possible. Fedora's policy is that it's only allowed as a matter of last resort (meaning, if it is at all possible to build a package against system libraries is should be build against system libraries). I'm not as familiar with Debian (cc @michel-slm who might know), but I believe they have a similar policy in place. Bundling is frowned upon because, among other things, it leads to a major increase in complexity when dealing with security updates for one of the bundled dependencies. It also makes auditing harder as it might not be obvious when a bundled dependency has been modified (or even where it came from in the first place in some cases).

That said, I also don't think it's particularly unreasonable to say "yes, we support latest bpftrace on Ubuntu 16.04 but you need to use an AppImage/static build/flatpak/snap/whatever." If these distros want latest bpftrace packaged in-repo, they'd have to go through the trouble of also updating its dependencies.

IMO this is a perfectly reasonable stance to take.

[Conan-Kudo]

Having done the backport work for Ubuntu 16.04 and 20.04 I can tell you that even if you do it, the result isn't terribly functional. You wind up missing kernel headers or libc headers and things of that nature. You need minimum kernel versions defined alongside libbpf versions.

Insofar as Debian's policy on bundling, it's generally forbidden rather than allowed as a last resort.

[danobi]

I do think the end goal should be to 'unvendor' it again when (if ever) everything stabilizes

That's kinda why I suggested 1.0 as the min version - it's supposed to be the last big API cleanup before "stability" is a thing. It's been baking for quite a while.

Now, requiring 1.0 would also require that distros update all their stragglers but overall I don't think it's an unreasonable design decision at this point in time.

Agreed.

bpftrace does some tricky things and might do more tricky things in the future. One avenue of concern is that libbpf may lack support for various advanced functionality.

I'd need examples but the thing about libbpf is that you can always bypass it. You'd have to do some of its job yourself (as I found myself doing) but there's no vendor lock-in per se. It just lowers the bar for most new features while keeping the bar for "exotic" features as-is.

One example I had in mind was how bpftrace used to encode map FDs directly into bytecode during codegen. I ended up fixing that to use ids instead of fds and fixing up the ids -> fds at load time. This helped make the codegen tests less hacky (no longer dependent on a specific fd allocation order).

I suppose if we used libbpf we could instead codegen however clang does it (emit relocations?) and have libbpf do the relocation.

I can't think of any other examples right now so I think I agree with you for now.

To expand, I think vendoring libbpf is really the only sane long term solution. ... It is more or less inevitable that we will have to carry some out of tree libbpf patches at some point

Maybe? Beyond bug fixes that haven't landed yet, I don't see why we'd want to carry out of tree libbpf patches.

Yeah, bugs are one reason. I remember I fixed a bug in libbpf a while ago where libbpf was incorrectly closing fd 0 in some scenarios. Stuff like that is painful for users and hard to explain.

Another reason would be to reduce amount of build time probing and ifdefs.

IOW I think vendoring is a way to make maintenance less painful given our limited resources. Less stuff to think about when writing code. An AFAIK statically linking / vendoring libbpf is the recommended way to use libbpf. For example bpftool statically links libbpf and is shipped by all distros I know of.

[viktormalik]

To expand, I think vendoring libbpf is really the only sane long term solution. It is more or less inevitable that we will have to carry some out of tree libbpf patches at some point

I'd strongly advocate against doing this. Once we start patching the vendored libbpf, rebasing to new versions will eventually become a pain and un-vendoring (which I agree would be the goal in such a case) will be quite difficult. Ofc we won't have the latest libbpf bug patches immediately (especially the packaged versions of bpftrace won't) but I think (or hope) that it's a rare situation that we'd need them. As for the features, I believe we'll be able to implement in bpftrace what is not provided by libbpf.

Another thing that we should keep in mind is that libbpf is in a way connected to the kernel. So I'd expect that if we vendor a new version of libbpf for a system with a much older kernel, that could cause some trouble. Maybe I'm wrong and it'll be alright, but we should at least give a thought about this.

That being said, I prefer setting a minimum required libbpf version rather than vendoring it. Unfortunately, libbpf is moving very fast, so at this point, 1.0 seems a too strong requirement to me (e.g. Fedora 35, which is less than a year old, has 0.6.1).

Are the library versions formalized anywhere? I tried looking but I'm not seeing it clearly stated in any docs. If we can say "last X Ubuntu LTS releases + last X centos etc," then we can get very precise about the libbpf API coverage we'd have and the bugs we'd have to avoid.

AFAIK no, but it'd be nice to have as deciding on the minimum supported libbpf (and other libs) version would be very easy.

IOW I think vendoring is a way to make maintenance less painful given our limited resources. Less stuff to think about when writing code. An AFAIK statically linking / vendoring libbpf is the recommended way to use libbpf. For example bpftool statically links libbpf and is shipped by all distros I know of.

The difference here is that bpftool is built as a part of the kernel, so you always end up with exactly the same version of libbpf bundled as is the system one.

[BurntBrunch]

Sigh, github ate my first comment so I apologize if I'm a bit shorter in this iteration.

First off, thanks for chiming in, everyone, it's much appreciated! I'll address some specific points first and discuss a possible formalization after.

Having done the backport work for Ubuntu 16.04 and 20.04 I can tell you that even if you do it, the result isn't terribly functional. You wind up missing kernel headers or libc headers and things of that nature. You need minimum kernel versions defined alongside libbpf versions.

@Conan-Kudo, can you expand a bit on what exactly you found broken w.r.t. headers? My hope is that going forward, with BTF being enabled in the last LTS Ubuntu release, headers will be less of an issue. As for minimal kernel versions, I actually disagree - feature probing should be enough and errors should be raised at runtime. We want the same binary to be portable across the widest possible range of kernel versions and libbpf also has a similar mindset.

I suppose if we used libbpf we could instead codegen however clang does it (emit relocations?) and have libbpf do the relocation.

@danobi, yeah, that's the idea. Emit the same global variables with relocations and all and let libbpf handle instantiating and managing the maps. It may even be simpler from a codegen and runtime perspective.

Another reason would be to reduce amount of build time probing and ifdefs.

I actually disagree, I think the best way to reduce build time probing is a high enough minimum version but that's orthogonal to vendoring the library itself. I'll discuss this in more detail below.

For example bpftool statically links libbpf and is shipped by all distros I know of.

The difference here is that bpftool is built as a part of the kernel, so you always end up with exactly the same version of libbpf bundled as is the system one.

Unfortunately, that seems to be defining difference - bpftool lives in linux/tools, so its packaging can follow different rules. I think vendoring libbpf for bpftrace will not fly with the distros but that doesn't mean that it's the only way forward. (See below)

Unfortunately, libbpf is moving very fast, so at this point, 1.0 seems a too strong requirement to me (e.g. Fedora 35, which is less than a year old, has 0.6.1).

I'll address this below but I don't think it's a problem per se but rather an artifact of the current "weak" dependency on libbpf.

Feature requirements and packaged versions

From a brief skim at the libbpf history, I think these requirements stand out:

1. At least 0.7.0 because of CVE-2021-45941, which affects things we'd need to use to delegate more to libbpf (namely, bpf_object__open_mem)
2. At least 0.8.0 because of KPROBE_MULTI support (libbpf/libbpf@2e6e39e)

On a separate note, no non-rolling distro is packaging a recent enough libbpf at all. I'm trusting pkgs.org on this but the list is not encouraging. Last Ubuntu LTS is at 0.5, CentOS 9 Stream at 0.6 and Amazon Linux 2 at 0.3. At the same time, their bpftrace versions are much newer (0.14, 0.13 and 0.11). This tells me that there's strong interest in bpftrace but no reason to update libbpf on its own (though it looks like Ubuntu Kinetic will have 0.8.0). From a certain angle, bpftrace raising the minimum libbpf version may be the impetus for distros to update their libbpf.

Proposal

I propose the following formalization of the libbpf dependency:

1. We land the version bump and the work delegation changes in the same bpftrace release. Minimum version at 1.0 is probably cleanest though if there are actual material reasons not to do 1.0, we can do 0.8 + set_strict_mode(ALL). My opinion is that the difference between 0.8 and 1.0 is probably insignificant for the package maintainers.
2. For local builds, we vendor libbpf with no patches. This enables us to work with features that haven't made it into an official libbpf release yet and only delaying the bpftrace release until libbpf is released as well.
3. For official static releases, we formalize the compatibility requirements (e.g. min glibc version). Right now, the static releases won't run on Amazon Linux 2 for example and that's fixable via manual work, AppImage, or something like build-anywhere.

With these requirements, we get this process for new BPF features:

1. Change lands in kernel and libbpf
2. Update vendored libbpf for local builds, bpftrace work can start
3. bpftrace work finished, wait for libbpf release
4. Bump libbpf min version and release bpftrace

I think it overall does a good job of 1) giving users better compatibility via static builds, 2) improving the developer experience so we don't have reinvent the wheel or be able to build against ancient system libraries, and 3) giving clear feedback to distros on what we need as minimum library versions. It's also in line with how other projects operate, so I don't expect it to be a surprise to package maintainers.

This is only an MVP proposal, so please poke holes in it! :)

[danobi]

Thanks for the detailed analysis @BurntBrunch . I've only skimmed the details (about to head off the grid for a few days) but I think this is a nice compromise.

I think running bpftrace in lockstep with libbpf is a good solution. We might initially prevent users on older distros from building bpftrace, but hopefully there is enough downstream pressure on distros to update libbpf such that things work out in the long term. We also provide dockerized builds + static CI builds so IMHO it's not really essential to have users natively build from source.

And hopefully in doing this we can lift up the broader bpf ecosystem by putting pressure on distros to frequently update libbpf.

I also like the vendor libbpf w/ no patches approach. That should help simplify development + CI a bit.

[viktormalik]

I really like the proposal @BurntBrunch. Setting up a minimal version and vendoring libbpf with no patches is a nice compromise that will make distro packager's life easier and give developers the latest libbpf features at the same time.

On a side note, from a distro packager point of view, doing a libbpf version update is not as simple as for other userspace tools (e.g. bpftrace). Since libbpf is a part of kernel, you typically need to update the entire BPF subsystem and often the kernel is what differs the most from the upstream version, so the backport requires quite some work. So, this may be the reason why distros have a much older version of libbpf than of bpftrace.

All in all, this seems like a good piece of work ahead of us. I might be able to find some free cycles for it if necessary, but it'd be nice to split the work into smaller tasks so that we can synchronize. @BurntBrunch I'm completely fine if you want to do this on your own, but if you needed help, could you perhaps open several issues for individual steps?

Thanks for moving this forward! I think that this is an important step in bpftrace development.

[danobi]

On a side note, from a distro packager point of view, doing a libbpf version update is not as simple as for other userspace tools (e.g. bpftrace). Since libbpf is a part of kernel, you typically need to update the entire BPF subsystem and often the kernel is what differs the most from the upstream version, so the backport requires quite some work. So, this may be the reason why distros have a much older version of libbpf than of bpftrace.

Clarifying in case I misunderstand:

• IIUC distros are encouraged to package libbpf from the GH mirror: https://github.com/libbpf/libbpf
• libbpf is designed to be forwards/backwards compatible with older and newer kernels. Here are two quick citations:
  • https://patchwork.ozlabs.org/project/netdev/patch/1571995035-21889-1-git-send-email-magnus.karlsson@intel.com/#2292905
  • https://lore.kernel.org/bpf/20201104024559.gxullc7e6boaupuk@ast-mbp.dhcp.thefacebook.com/

IOW libbpf source tree is already decoupled from kernel sources. And libbpf is designed to work across as many kernels as possible.

So I'm not sure there is a reason libbpf is so out of date other than no other package is forcing packagers to upgrade.

[BurntBrunch]

I think running bpftrace in lockstep with libbpf is a good solution. We might initially prevent users on older distros from building bpftrace

Oh, but that's the thing - we wouldn't. Local builds will by default use the vendored libbpf, so most users would be fine. Package maintainers will have to do a different dance but that's fine, it's part of the deal.

from a distro packager point of view, doing a libbpf version update is not as simple as for other userspace tools (e.g. bpftrace). Since libbpf is a part of kernel, you typically need to update the entire BPF subsystem and often the kernel is what differs the most from the upstream version, so the backport requires quite some work

Daniel responded to this but I just want to add that any such issues (newer libbpf not working on older kernels or vice versa) should be raised upstream as they are 100% bugs.

So I'm not sure there is a reason libbpf is so out of date other than no other package is forcing packagers to upgrade.

Right, that's my read too. If anything, libbpf not being used by kernel tools as a dynamic library has hurt the up-to-dateness - otherwise the distro kernel teams would package it with perf, bpftool et al.

Or maybe it's just the fact that so far libbpf has been backwards compatible to the beginning, more or less, and no one is using the newer features in distro-packaged software. 1.0 is the first API break and it can be simulated with set_strict_mode on earlier versions, so I think it may be the first real hurdle for distros (a bunch of things disappear or become errors in 1.0).

@BurntBrunch I'm completely fine if you want to do this on your own, but if you needed help, could you perhaps open several issues for individual steps?

I'll take any and all help I can get! I'll write up a reasonable set of refactoring steps in a new issue and people can claim whichever pieces they want either on that issue or on a new one? I think I have some reasons to do certain things before others but I'll document all that.

As a point of clarification, are we happy with the current static builds requiring glibc 2.27? The glibc 2.27 requirement is being brought in by log2f (no idea what brings it in) and glob, removing those would get us to 2.16, which is a significant relaxation. Amazon Linux 2 seems to ship glibc 2.26, so this would make the binary work out of the box.

Alternatively, a fully static build (builtin libc) or just postponing this decision is also a path forward.

[Conan-Kudo]

IOW libbpf source tree is already decoupled from kernel sources. And libbpf is designed to work across as many kernels as possible.

While this is true, bpftrace is more tightly coupled because it uses both libbpf and bcc. The latter is what fails on older kernels. Both bcc and bpftrace have much tighter coupling with Linux APIs, and my attempts around backporting to older distributions has led to enough funky results that I would probably recommend declaring baseline kernel uapi versions too.

[Conan-Kudo]

I should also mention that there's an LLVM dependency here, and outside of RHEL and SLE, LTS distributions typically don't ship fully functioning up to date LLVM stacks to use. And even when they are available (e.g. in Ubuntu LTS as part of HWE stuff), it usually requires some serious contortions to get it to work properly at build time.

For example, I carried this downstream patch for work when I backported the Fedora bpftrace package to Ubuntu 16.04 over two years ago (yes, you read that right... I gave a talk about it a while ago!): bpftrace-0.9.3-add-custom-clang-module-files.patch.txt

There is a fair amount of complexity around this, and even if you manage to compile it, it behaves badly enough at runtime that it's basically not worth it.

[BurntBrunch]

because it uses both libbpf and bcc. The latter is what fails on older kernels.

Right, a lot of the refactoring here will involve moving things to use libbpf's equivalent functionality instead.

I should also mention that there's an LLVM dependency here

Hey, one major dependency at a time! 😄

But I hear you, that's definitely the next big thing to clear up and figure out. I'm hoping that a lot of people would be okay with using the static binaries but I understand that that's not always allowed and I don't have a silver bullet for that case (I'm assuming that vendored LLVM was not an option at your work place?).

[Conan-Kudo]

But I hear you, that's definitely the next big thing to clear up and figure out. I'm hoping that a lot of people would be okay with using the static binaries but I understand that that's not always allowed and I don't have a silver bullet for that case (I'm assuming that vendored LLVM was not an option at your work place?).

Not really, no. That's usually creates way more trouble.

[Conan-Kudo]

IIUC distros are encouraged to package libbpf from the GH mirror: https://github.com/libbpf/libbpf

Then it should be dropped from the kernel source tree. The current situation is way too confusing.

[BurntBrunch]

@Conan-Kudo feel free to start a discussion at the libbpf repo here on GitHub or on the bpf mailing list.

In this issue, we don't really control how libbpf is version controlled or distributed but we do control how bpftrace finds and uses it. The proposed changes will likely make your life easier, llvm versioning notwithstanding.

[viktormalik]

• IIUC distros are encouraged to package libbpf from the GH mirror: https://github.com/libbpf/libbpf

I didn't know that this is the recommended way, thanks for pointing out.

Then it should be dropped from the kernel source tree. The current situation is way too confusing.

Probably yes, but as has been pointed out, that's a question for another forum.

I'll take any and all help I can get! I'll write up a reasonable set of refactoring steps in a new issue and people can claim whichever pieces they want either on that issue or on a new one? I think I have some reasons to do certain things before others but I'll document all that.

Yes, that sounds like a good plan. We should have one "overview" issue with all the steps (and dependencies between them) clarified and then we can create individual issues for each step where it's worth.

As a point of clarification, are we happy with the current static builds requiring glibc 2.27? The glibc 2.27 requirement is being brought in by log2f (no idea what brings it in) and glob, removing those would get us to 2.16, which is a significant relaxation. Amazon Linux 2 seems to ship glibc 2.26, so this would make the binary work out of the box.

We use glob to do path expansion when attaching to uprobe and usdt targets. I guess that we could implement it by ourselves if that was necessary. I'm not sure what log2f is used for, AFAIK, logarithmic histograms are implemented directly in LLVM IR.

I should also mention that there's an LLVM dependency here

Hey, one major dependency at a time! smile

Again, this is a discussion for another place, but there is this issue #1845 which would drop dependency on LLVM completely. Ofc it's a long way to go, I just wanted to point it out since we're at it.

[viktormalik]

Right, a lot of the refactoring here will involve moving things to use libbpf's equivalent functionality instead.

That would be great. AFAIK, there are currently 3 major parts where we depend on BCC: USDTs, probe attachement, and symbol resolution. I think that at least the first two will be replaced by libbpf as a part of this, so there's hope we could decouple from bcc eventually.

[BurntBrunch]

Closing as resolved.
POR is to vendor libbpf for local builds and clearly state the dependencies, moving libbpf along as it gains features we need.

Let's discuss the actual work/steps in #2334.

[martinetd]

Chiming in late here (thanks to the link from the bcc issue), but I can also add the nixos and alpine packager hats for not liking vendoring.

for nixos it's ok as last resort, but I don't think I've ever seen any alpine package with one yet (explicitly looking right now I see libcamera has some, but I see more patches that remove them than packages using one)

I think it's ok to only support libbpf 1.0+ (or 0.8.1) though -- distros that are stuck at older libbpf versions like fedora 35 will also never upgrade bpftrace. If you're just thinking of making it easy for developers then an optional submodule never hurts, but I'd hugely appreciate if it's not made mandatory.

EDIT: re-reading the thread now and it looks like this is what's intended already, sorry for the noise -- and thanks for the work :)