-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Include wireguard-tools package in FCOS #362
Comments
We discussed this during the coreos community meeting today. We asked a lot of questions to @jdoss. Thanks @jdoss for helping to answer those questions. To summarize the discussion, we are leaning towards including
|
NetworkManager supports initiating tunnels, yes sort of, but the whole situation is pretty hairy to use without wireguard-tools and you'll wind up needing another computer. And you won't have much visibility into the tunnels either or be able to manipulate them in meaningful ways. I consider wireguard-tools to be a "required" package for all uses of WireGuard except the most mega-minimal totally custom userlands. |
I just read through https://meetbot.fedoraproject.org/fedora-meeting-1/2020-02-05/fedora_coreos_meeting.2020-02-05-16.30.log.html and the conclusion doesn't seem quite accurate. Both systemd-networkd and networkmanager are capable of setting up some aspects of WireGuard, but it's going to be a bad frustrating experience without wg(8) and systems will be very difficult to maintain. If FCOS ships ip(8), then you need to ship wg(8). If FCOS is focused on some kind of absolutist minimalism and does not ship ip(8), then I could see some argument for not including wg(8). AFAICT, there's ip(8); therefore, there must also be wg(8). |
We do ship Remember for CoreOS though we are here to emphasize containers not just for application code, but components that people might traditionally have installed on the root filesystem. On a CoreOS system for example when you want to use Having If one wants to use If it's containerized (or run from a container, anyways) then its lifecycle isn't tied to the host. For example, someone could run upstream git master of That all said, anything that affects the host core networking/storage is a "fuzzy line" for containerization. As I noted in the meeting at least for OpenShift 4 we ended up shipping openvswitch on the host. Personally I'm a fan of WireGuard and its goals - obviously excellent code and has been rightfully widely adopted. But anything we propose adding to the host here has to go through this "why isn't it a container?" process. |
Okay, that's the same situation here. wg(8) is small. It's smaller than ip, in fact. It's really really really tiny. And it has no dependencies. Like the rest of the networking stack, WireGuard uses netlink, which means we have a backwards compatible situation too. If you have to run a container to get access to an essential administrative utility for WireGuard, then I'd recommend that people don't use FCOS for WireGuard, since the overhead to use it is rather high. Really, it's tiny:
Dependencies:
|
Another point here: one of the principle features of WireGuard is using it as a means to network between containers. That means doing things in the root and not in containers. WireGuard is made for use in containerization situations. This is an administrative utility that is important in setting up that sort of thing. |
One of the great things about Linux "containers" is there's no such thing - there's a collection of orthogonal technologies one can pick and choose. In particular, a lot of the ecosystem makes use of |
Yes, I know how linux namespaces work. That doesn't change the fact that if you're having to launch containers in order to run a 79k administrative tool on par with ip(8) that makes modifications to the init proc network namespace, nobody is going to be happy with wireguard on coreos. And no, networkmanager's support really doesn't cut it either here, as @dustymabe and I have discussed offline. If you want to containerize ip(8) and remove it from the root image, then I'll trust that you're true to your principles and want to stick every administrative binary into these as part of some radical system architecture. But if not, all the arguments for keeping ip(8) in the root image apply to wg(8). That is, if you want to support wireguard on FCOS. If you don't want that, then that's your prerogative of course. But supporting wireguard means having wg(8) as available as ip(8). |
With some trivial changes, we're actually now at 49k:
If you really are counting bits because you want this to run off of 1.44 floppies, I have more tricks in mind too. Let me know what you need. |
@zx2c4 thanks for the feedback (and all your work on wireguard)! I've very little specific knowledge on
Would it be possible to have a summary of the technical shortcomings here? Or, if it is already properly written elsewhere, just a reference? Specifically, I'm interested in reaching a final UX flow where Ignition writes NM wireguard profiles (keyfiles) and secret material, and tunnel provisioning works without further custom scripting. If there are known blockers for us getting there, it would be good to track them and let the NM folks aware of them. For clarity, this has no direct impact on the discussion whether FCOS should ship |
Regardless of whatever network management framework used, WireGuard is not easily useful or usable with out |
Thanks everyone for the fruitful discussion. While I haven't investigated the original questions on an FCOS node I have discussed with the experts (including Jason and Thomas Haller) and come to the following conclusions:
Yes, According to Thomas it can work in NM in Fedora today without any extra plugins.
According to Thomas it does not strictly need the tools package but he agrees that not having
We were leaning towards including it already (according to #362 (comment)), but we can bring it up at our next meeting to see if there are any remaining points to discuss.
I agree with you. That is our end goal. I think we should strive to have some documentation to that effect, but maybe that can be a separate ticket in this tracker and we can concentrate on the inclusion of the tools package here. WDYT? |
Also here is a link to some yet to be implemented work in NM to improve wireguard support: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/issues/358 |
Moved the doc task to coreos/fedora-coreos-docs#43. However I was not asking how to do it, but if it currently works. Some comments in this thread seems to hint towards a negative answer, and I'd rather avoid ending up in a scenario where people start blogging their own creative workarounds as as soon there is |
We discussed during the meeting today. For what it's worth we have chosen NetworkManager as our recommended gatekeeper to all things networking. That means that we recommend and document things be done using NetworkManager, but we acknowledge connections can be established in other ways, using other tools. In the meeting some use cases we discussed for FCOS users of the tools:
Considering the above use cases and also that maintainers of wireguard and NetworkManager have recommended installing the tools we made this proposal in the meeting:
To highlight: yes, we will include wireguard tools (eventually).
The last bullet point in the proposal was a contingency for if we don't have documentation or fixes for the "happy path" in a reasonable amount of time. I'd like to propose that if we don't have a recommended way for users before the F33 rebase then we include the tools anyway. For any users that need |
Conflating |
Any update on this issue? |
I don't quite understand why this issue has been delayed from February to now: If there is no able to implement the recommended way, at least a software package should be provided. Not to mention calico support wireguard as backend from version 3.15 and flannel already support wireguard years ago! You could use
It's not the recommended preference you planned, but I didn’t see any actual progress on happy path and still it's better than install with After almost six month, I hope this could be done asap instead of "eventually". |
Hey folks, what is the remaining holdup on getting |
I've proposed a pull request to add |
@Conan-Kudo I don't know what remaining holdup on getting They seems want to manage wireguard with NetworkManager and make And almost six month pass, nothing happend. Even if they find a way to achieve “happy path”, they still need I don't know what are they waiting for, why don't they provide Not to mention this issue is just about getting wireguard out of the box. And happy to see you open that pull request. |
No. If you believe that, you didn't even read this thread. |
Clearly you didn't either, because #362 (comment) indicates that it will be included. I'm just making the PR to make that happen. |
To be clear, I am not entirely opposed to just saying "whatever, let's ship wireguard by default" today. But a whole lot of the entire point of Fedora CoreOS is to push for containerization. The automatic updates model works much better when you've done that, etc. And yes, wireguard is in the grey area that we are planning to address with an extension system. I also have some pending ostree work for updates without rebooting. This is absolutely an arguable thing. I am very sympathetic to the "it's so small" argument combined with its obvious utility, also combined with the fact that one often would want wireguard to control the default route. In OpenShift right now I'm also in the middle of a debate with the SDN team who are trying to run This is absolutely arguable. But anyone who isn't seeing the reason for debate at all - then you need to try harder. |
From my perspective, if I want the hosts to have a consistent base set of functionality that container workloads can expect to be able to use, I would not want that in a container because that is, by its nature, inconsistent. Moreover, as you are already aware, it's pretty difficult to use containers to orchestrate host functionality without essentially granting it access to the world, and that leads to other problems with maintaining the stability of the system. And shipping a ~50-100MB tarball for 121KB is ridiculous. Finally, WireGuard functionality is tied to the Linux kernel, which you cannot really change from a container. Thusly, it makes sense for
The real solution there is that SELinux needs namespaces, but I don't know of anyone actually working on implementing that...
From my perspective, there's no more debate to be had. The WireGuard tools going to be included in FCOS. The only thing remaining is documenting how to use and diagnose WireGuard connections with FCOS. |
WireGuard is not in any form of "gray area". It's very simple: if ip(8) is in fcos, then wg(8) should be in fcos. If wg(8) is not in fcos, then I'd suggest removing ip(8) too, and putting them into whatever kind of overlay system you have in mind for whatever date sometime in the future maybe. Neal has sent a PR for the former. I can produce one for the latter if you'd like. |
@cgwalters @Conan-Kudo please check https://github.com/shivarammysore/ovs where I have already done the work of running OVS on FCOS. I would be happy to answer any questions. In reference to
What I have not seen is a list of use cases and limitations of getting it to work with FCOS. Getting it to work with Containers on a one off machine is not that hard. Getting to work on a production system which is what I would use |
based on the outcome of the meeting earlier this year:
We put in a clause for if the "happy path" documentation got delayed. It's fair to say it's been long enough. I will merge coreos/fedora-coreos-config#529 later today unless someone brings new information to light that hasn't been brought up already. Documenting the "happy path" is still something we want to do, we just won't block on that. |
I think it makes sense to add |
Why would wireguard-tools not be part of the container that is doing the wireguard setup? Why does the userspace part need to be on the base OS? |
I agree `wireguard-tools` just has administration modules for `iptables` or
`nftables`. Once things are setup, not sure why you would need them.
Alternatively, one could spot install that package, configure and remove
the same if they so desire.
…On Wed, Jul 22, 2020 at 9:10 AM Dan Williams ***@***.***> wrote:
Why would wireguard-tools not be part of the container that is doing the
wireguard setup? Why does the userspace part need to be on the base OS?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#362 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMZ74ZZAY2TUYPLRQVIKGDR44FNVANCNFSM4KONTSRQ>
.
|
Why would wireguard-tools not be part of the container that is doing the
wireguard setup? Why does the userspace part need to be on the base OS?
One common scheme is to create and manage tunnels outside containers and
then move them inside, like veth. See wireguard.com/netns/ . Or to just
have the host generally connected via WireGuard. If ip(8) is in the host,
wg(8) belongs too.
|
The followup to this for documentation of a "happy path" is captured in coreos/fedora-coreos-docs#43 |
Jason, you've stated this several times. I still disagree, for two reasons: First, basically every single real world use case of FCOS that I can think of involves IP networking, and you need IP networking to pull containers. Our automatic updates on by default pull over IP networking. That's not true of WireGuard. Second, the bigger picture here is that while WireGuard is great, there are users/organizations out there that made nontrivial investment in prior VPN technology like OpenVPN or IPSec. If we bake in any WireGuard functionality, this weakens our story when users who need those things come asking for it to be baked into the host too, because "WireGuard is there". |
The comparison doesn't hold. Both OpenVPN and IKE (usual IPsec key exchange protocol) require userspace daemon software to be running. However, there are usages of IPsec that are enabled by straight up |
The fix for this went into testing stream release |
The fix for this went into stable stream release |
WireGuard will be included in Linux 5.6 torvalds/linux@bd2463a and it would be nice to include the wireguard-tools package so WireGuard can be managed better out of the box on FCOS.
https://src.fedoraproject.org/rpms/wireguard-tools
I am opening this issue to discuss what it will take for this to happen.
The text was updated successfully, but these errors were encountered: