Firewall Management

[mskarbek]

What is the desired firewall management method for FCOS at the moment? Atomic Host is using firewalld, Container Linux plain iptables. AFAIK Fedora 29 will switch over from iptables to nftables and set nftables as a default backend in firewalld. FCOS, with a strong desire to not ship Python, plans to stay with iptables, switch to plain nftables?

[dustymabe]

ok discussed in the meeting today we discussed this item. We discovered we have quite a bit of investigation to do before we can really make a decision on this. Some obsverations/open questions from the meeting:

• if we don't have python then we can't have firewalld (unless that team is itching for a rewrite)
• do we have to choose between using nftables or iptables or can we support both?
• can iptables userspace interact transparently with nftables?
• we prefer for users to set up firewall rules using ignition and then higher level applications (docker/podman/kube) can configure the firewall themselves. Preferably not user interacting with the firewall directly.

we'll need to have someone (or small group of people) look into these items and report back to the group with a recommendation.

[cgwalters]

if we don't have python then we can't have firewalld (unless that team is itching for a rewrite)

I'm not sure I agree with this. I still think the "platform python" approach could work. Or, if we wanted to make it a whole lot harder for external people to use it, we could randomize the location of the binary, e.g. /usr/libexec/platform-python-$(uuidgen).

(Walking over every python binary and rewrite the shebangs)

[dustymabe]

I'm not sure I agree with this. I still think the "platform python" approach could work.

i'm going to refrain from commenting on that subject. While it definitely has implications I think it warrants it's own card as to not derail the firewall discussion. Would you like to open a separate issue?

[dustymabe]

Would you like to open a separate issue?

He opened one here: #32

[cgwalters]

While it's not quite the same thing, note that systemd is working on bpf firewalling which is service-oriented rather than system oriented.

[dustymabe]

blocked by discussion in #32

[oliwer]

The latest versions of iptables use the nf_tables backend. So we can safely ship both. https://wiki.nftables.org/wiki-nftables/index.php/Legacy_xtables_tools

As for firewalld, I don't think many people would be happy to see that thing in FCOS (bloated, stateful, complex, xml...). Cloud deployments usually don't need complex firewall configuration. It is handled elsewhere. BPF firewalling in systemd looks much more appealing.

[dustymabe]

no longer blocked on #32 since that has been decided. removing blocked label

[lucab]

First, I would like to split concern between static rules (that are immutable and attached to a node lifetime) and dynamic rules (that are environment-dependant, ephemeral, and require active management at runtime).

In general, FCOS is only concerned with static rules that are configured at machine-creation time. This allow us to unify the “firewalling setup” flow under the usual “first-boot provisioning via Ignition-files” flow.
While I agree that dynamic rules are ubiquitous and needed for most usecases, those are better owned and manipulated by containerized higher-level tools in an ephemeral way.

As the landscape of Linux firewalling is currently evolving, we'd like to provide both plain iptables and nftables, and let users and software consumers pick the one relevant for their environment (and don’t rely on the alternative symlink). We can ship relevant *tables-restore services and documentation, and expect the final UX to mirror the ContainerLinux one: https://www.jimmycuadra.com/posts/securing-coreos-with-iptables/

At the moment I'd expect to not ship any distro-hardcoded static rules by default, leaving full ownership of both static and dynamic firewalling to the user.

For the long term, we like the approach taken by systemd with per-slice eBPF filtering. For static rules, it already fits with Ignition provisioning of service dropins. For dynamic rules, there is likely some integration surface for delegation between systemd and (container) runtimes. It is however unclear if this approach can completely replace system-wide firewalling, or if it will just coexist alongside other *tables tools (e.g. for NAT and forwarding policies).

As an additional note, a wish item for the medium-term future could be a tool similar to systemd-sysctl for static firewalling rules, capable of sourcing rule-snippet from multiple locations (under /usr, /run, and /etc) and merging them in order. However at this point we don’t expect having any hardcoded rules under /usr, so we won’t be actively working on designing and implementing such a tool (but we welcome experimentation in this area).

[dustymabe]

so in TL;DR bullet point form for the "short term":

• we're focused on static rules and letting runtimes control dynamic rules
• we'll ship tools for old and new firewalling technology and let users/applications choose which to use
• we'd like to ship "no firewall" by default

Did I miss anything?

I'll add the meeting label so we can discuss tomorrow.

[dustymabe]

Discussed this in the meeting today. We had no opposition to the current proposal (summarized here).

Remaining action items for this ticket:

• Commit decision and reasoning to Design.md doc
• Identify any missing software that we need to ship in FCOS that we aren't currently doing today to support legacy and new backends
• Make sure existing FCOS configs produce artifacts with no firewall enabled by default (when we move away from anaconda we should make sure we keep this)

@lucab - can you handle those three bullet points ^^

[cgwalters]

I think this ticket is great and RHCOS should inherit this.

One thing I want to add here is that Kubernetes changes the game around firewalling in a lot of ways, but an interesting specific point is that in my opinion, a lot of the idea of host-level firewalling is inherited from the original "classic Unix timesharing" model where you'd have unprivileged users ssh'ing into a big box.

In Kubernetes, only admins ssh into machines. User applications are run in a "overlay network" - you shouldn't be able to e.g. talk to a host's sshd from the cluster. Further they are only exposed via ingress/router (well ok there's NodePort too). Admins can have a lot of power/control over exposed services; in traditional Unix any process can just call bind() without much oversight and I think that was a motivator for traditional host-level firewalling.

So TL;DR: I assert that Kubernetes clusters really don't have a strong need host-level firewalling by default.

[dustymabe]

since coreos/fedora-coreos-config#40 and #103 have merged, I think we can close this.

cc @lucab @bgilbert

[danwinship]

Sorry, catching up belatedly and @cgwalters said I should add my comments here...

coreos/fedora-coreos-config#40 includes iptables-services, but that service is really cloud-unfriendly. In particular, if you edit its configuration and then systemctl restart iptables, it will delete all of the iptables rules on the system (including, in particular, the dynamic iptables rules created by kubernetes, etc). Currently kubernetes deals with this by frequently polling to see if the user has deleted all of its rules out from under it, but obviously that's not ideal and we'd like to not have to do that...

Possible fixes would include:

• drop iptables-services, provide some other simpler service that sets up admin-provided iptables rules at boot time, but doesn't provide any way to change them after that
• patch iptables-services to disable restart
• don't explicitly disable restart, but just document that you're not allowed to do it, and it will break things if you do
• modify iptables-services or create an alternative to it that can restart/reload without having to delete everyone else's rules. (This would mean that instead of letting the admin add rules directly to the INPUT, FILTER, etc, chains, it would just create rules like -I INPUT -j ADMIN-INPUT, -I FILTER -j ADMIN-FILTER, etc, and then put all the admin-specified rules in the subchains. Then, on reload, it could delete and recreate just the subchains, but leave the toplevel chains untouched so as to not interfere with rules created by kubernetes, etc.)
• modify iptables-services to emit a dbus signal when it restarts ("This is just to say, I have eaten the iptables rules that were in the icebox, and which you were probably saving for breakfast...") and then modify kube to just watch for that rather than constantly polling. (Firewalld does this, and kube watches for that signal, but only if firewalld is running, so iptables-service can't just spoof firewalld's signal and have it work.)

[lucab]

@danwinship I pretty much share those concerns. I think this followup mirror the discussion ongoing at https://bugzilla.redhat.com/show_bug.cgi?id=1667875 and your feedback would be valuable there (sorry if we keep bumping you). My personal hope was to get plain restore-only services (i.e. your first bullet item).

[lucab]

@danwinship I cross-linked your feedback to bugzilla, you may want to subscribe yourself and watch for followups there.