Option to allow containers to bind to specific interfaces

[TRPB]

This adds the following option in containers..forwardPorts:

    forwardPorts = [
      {
        containerPort = 80;
        hostPort = 8080;
        protocol = "tcp";
        interfaces = [
          "lo"
        ];
      }

A list of interfaces can be specified so that different services can listen on IP addresses.

In addition, it allows the loopback interface to be bound

Description of changes

Feedback is welcome, this is my first time doing anything in the nix language that isn't just defining data structures for system configuration so no doubt there are better ways to achieve what I've done here.

The current containers implementation has a few limitations when it comes to port forwarding:

• You cannot bind ports on different interfaces on the host, it's (nearly) all or nothing
• The loopback interface cannot be bound

The last one is a little surprising. Tools like docker make us used to being able to access a container via 127.0.0.1:80 or whatever.

There are quite a few times when I'd like to access a container locally but not expose it to the rest of the network. For example a web development machine. While the firewall can do this, why bind the port only to have the firewall block access to it?

Alternatively, a machine with a public and local interface on the network, I can differentiate between internet facing services and local only services.

Implementation

There is a new interfaces option in forwardPorts when declaring a container.

Example:

  containers.web = {
    autoStart = true;
    privateNetwork = true;
    config = ./lemp.nix;
    extraFlags = [ "--U ];

    hostAddress = "192.168.100.10";
    localAddress = "192.168.100.11";

    bindMounts = {
      "/app" = {
        hostPath = "/tmp";
        isReadOnly = true;
      };
    };

    forwardPorts = [
      {
        containerPort = 80;
        hostPort = 8080;
        protocol = "tcp";
        interfaces = [
          "lo"
        ];
      }
    ];
  };

Implementation choices

Nspawn's port options are very limited. This implementation skips it and manually configures port forwarding from the host to the container.

Interface names were chosen over IP addresses as IPs can change. For example, running a service on a machine on the local network and accessing it via the hostname is not uncommon. By using interface it doesn't matter if the IP changes machinename:port will still work to other machines on the network if the host IP changes.

I'm using socat to handle the forwarding. It may be possible to do this with IP tables but regardless what I tried I could not get forwarding to work on the loopback address. Though I'm no iptables expert. Similar issue described here

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[TRPB]

Can someone point me in the right direction for getting this test to work? It's the only part I'm struggling with.

I think I'd need to bring both interfaces up with the correct IPs for it to work but I'm unsure how to do that in the test

[TRPB]

I added this so that if the IP on the host changes the service will restart

[TRPB]

is there a nicer way of doing that?

[JustTNE]

For something more complicated like this we simply use the more advanced NAT features of nixos like here: https://gitlab.com/garuda-linux/infra-nix/-/blob/main/nixos/modules/nspawn-containers.nix?ref_type=heads#L150

[TRPB]

Thanks, I can't find very much documentation on that and the test coverage is minimal.

Do you have an example of binding a port to 127.0.0.1?

I have this setup:

  networking.nat = {
    enable = true;
    internalInterfaces = [ "lo" "eno2" ];
    externalInterface = "ve-web@if2";
    forwardPorts = [
      {
        sourcePort = 80;
        proto = "tcp";
        destination = "192.168.100.11:80";
      }
    ];
  };

I'll update the PR to add the NAT portforwarding rules instead of using socat if I can get it to work.

I'm trying to map localhost or the IP from eno2 to the container but can't connect.
curl 192.168.100.11 works, but curl localhost or curl [eno2 ip] does not. What am I missing here?

I have the various sysctl options set to allow forwarding:

  boot.kernel.sysctl = {
    "net.ipv4.conf.all.forwarding" = 1;
    "net.ipv4.conf.default.forwarding" = 1;
    "net.ipv4.conf.eno2.route_localnet" = 1;
  };

edit: And looking at the options, I don't think that approach will allow me to have different port forwarding rules for different external interfaces?

Seems I'm not the only one:

https://discourse.nixos.org/t/networking-nat-not-working-as-expected-for-nixos-container-networking/41050
https://discourse.nixos.org/t/struggling-to-forward-a-localhost-port-to-another-port-on-localhost/28278

[TRPB]

edit: Ignore below, I think there's a bug in how it works. There is a loopbackIps option:

    forwardPorts = [
      {
        sourcePort = 80;
        proto = "tcp";
        destination = "192.168.100.11:80";
        loopbackIPs = [ "127.0.0.1" "192.168.2.204" ];
      }
    ];

But it doesn't work unless nftables is enabled. There's a bug somewhere in nixos/modules/services/networking/nat-nftables.nix it looks like it is just missing masquerading. I'll see if I can fix it and open a separate PR.

I've managed to get that working with nftables. This was super unclear and I don't know enough about it to understand the security implications (or lack thereof) from doing it this way:

  networking.nat = {
    enable = true;
    internalInterfaces = [ "lo" "eno2" ];
    externalInterface = "eno2";
    forwardPorts = [
      {
        sourcePort = 80;
        proto = "tcp";
        destination = "192.168.100.11:80";
      }
    ];
  };

  networking.nftables = {
    enable = true;
    ruleset = ''
        table ip nat {
          # Needed for external connections connecting inbound on eno2
          chain PREROUTING {
            type nat hook prerouting priority dstnat; policy accept;
            iifname "eno2" tcp dport 80 dnat to 192.168.100.11:80
          }
          
          # The following are both needed for internal connections on 127.0.0.1:port
          chain POSTROUTING {
            type nat hook postrouting priority srcnat; policy accept;
            fib saddr type local counter masquerade
          }

          chain OUTPUT {
              type nat hook output priority dstnat; policy accept;
              ip daddr 127.0.0.1 tcp dport 80 counter dnat to 192.168.100.11:80
            }
        }
    '';
  };

source

So my questions are:

1. Why doesn't networking.nat set these up automatically? Without manually configuring prerouting nftables, no external connections are possible which seems to make the networking.nat option rather useless on its own. The wiki page here demonstrates the same thing. The output chain is required if I want to connect via 127.0.0.1 or the en02's ip from the local machine.
2. I'd like to automatically configure networking.nat based on containers..forwardPorts. Would that be accepted in a PR? My example with conainers..forwardPorts.interfaces would be used.
3. If my machine has two interfaces with two public IPs, I don't see how it's possible to use networking.nat to run, for example, different web servers on each of the two interfaces because port forwarding is applied to networking.nat.externalInterface. Defining different port forwarding rules for each interface doesn't seem to be an option unless I'm missing something. Wouldn't we need networking.nat.eth0.forwardPorts = [] for that?

[JustTNE]

I wouldn't suggest you change anything about the nat stuff right now, I have an open pull request that makes major changes and introduces more tests to nat here: #277016

If you have a test case for NAT that isn't being tested here that should work, please let me know and I'll add them to the PR.

Do you have an example of binding a port to 127.0.0.1?

I don't think you can do that directly using NAT honestly. What you can do is directly access the service through the bridge interface's address. If you do port forward and use `loopbackIPs then you can also make a connection from the server to its own public IP to access services behind NAT like that. You would need a custom iptables/nftables rule to allow 127.0.0.1 directly though.