CNI Plugins compatible with nftables

The plugins in this repository implement CNI Specification v0.4.0.

At the moment, the CNI Plugins maintained by the CNI team do not support nftables. The below plugins do.

This repository contains the following plugins:

• cni-nftables-portmap: port mapping with nftables
• cni-nftables-firewall: firewalling with nftables

Table of Contents

• CNI Plugins compatible with nftables
  • Table of Contents
  • Supported Operating Systems
  • Getting Started
  • Architecture
  • Miscellaneous
    • Known Issues

Supported Operating Systems

The plugins had been tested on the following systems:

• CentOS 8:
  • kernel version: 4.18.0-193.14.2.el8_2.x86_64
  • nftables version v0.9.3 (Topsy)

Getting Started

First, download the plugins:

go get -u github.com/greenpau/cni-plugins/cmd/cni-nftables-portmap
go get -u github.com/greenpau/cni-plugins/cmd/cni-nftables-firewall

Check the location of the downloaded plugins:

$ which cni-nftables-portmap
~/dev/go/bin/cni-nftables-portmap

Next, copy the plugins to /usr/local/lib/cni/ directory.

mv ~/dev/go/bin/cni-nftables-{portmap,firewall} /usr/local/lib/cni/

After the above, copy assets/net.d/87-podman-bridge.conflist to /etc/cni/net.d/.

sudo cp assets/net.d/87-podman-bridge.conflist /etc/cni/net.d/

The configuration is as follows:

{
  "cniVersion": "0.4.0",
  "name": "podman",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni-podman0",
      "isGateway": true,
      "ipMasq": false,
      "ipam": {
        "type": "host-local",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ],
        "ranges": [
          [
            {
              "subnet": "10.88.0.0/16",
              "gateway": "10.88.0.1"
            }
          ]
        ]
      }
    },
    {
      "type": "cni-nftables-portmap",
      "capabilities": {
        "portMappings": true
      }
    },
    {
      "type": "cni-nftables-firewall",
      "forward_chain_name": "forward"
    }
  ]
}

Please note the ipMasq key is being set to false.

Review network config:

$ sudo podman network ls
NAME     VERSION   PLUGINS
podman   0.4.0     bridge,cni-nftables-portmap,cni-nftables-firewall

Next, run the following command to place a container in the previously created podman network and query AWS Check IP website:

podman run --net=podman -it nicolaka/netshoot curl http://checkip.amazonaws.com/

Run the following commands to test port-mapping plugin by placing a container in podman network and exposing a web server in the container.

First, start the container:

podman run --net=podman -P -d nginxdemos/hello

Alternatively, map the container port to port 8080:

podman run --net=podman -p 8080:80/tcp -d nginxdemos/hello

Verify connectivity to the container:

curl -v http://HOST_IP:8080

For localhost port mapping to work it is necessary to set route_localnet sysctl

sysctl -w net.ipv4.conf.all.route_localnet=1

Also, don't limit your sending and receiving of 127.0.0.1 to the lo interface.

nft add rule filter input iifname != 'lo' ip saddr 127.0.0.0/8 counter drop
nft add rule filter input iifname != 'lo' ip daddr 127.0.0.0/8 counter drop

This will block communication between the localhost and the container network. Instead, match the physical interfaces explicitly, like this.

nft add rule filter input iifname eth0 ip saddr 127.0.0.0/8 counter drop
nft add rule filter input iifname eth0 ip daddr 127.0.0.0/8 counter drop
nft add rule filter input iifname eth1 ip saddr 127.0.0.0/8 counter drop
nft add rule filter input iifname eth1 ip daddr 127.0.0.0/8 counter drop

This allows the container network to communicate with localhost.

Architecture

TBD.

Miscellaneous

Known Issues

There could be an issue with checksums when using portmap plugin.

Specifically, packets would arrive to a container, but they would be disregarded and no SYN/ACK would be sent.

When running tcpdump inside a container, there is checksum error cksum 0xd776 (incorrect -> 0xd8b9).

$ tcpdump -i eth0 -vvv -nne
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:05:16.704789 ee:58:3f:4d:1f:23 > ea:56:b4:c6:4f:c7, ethertype IPv4 (0x0800), length 58: (tos 0x0, ttl 63, id 8844, offset 0, flags [none], proto TCP (6), length 44)
    10.0.2.2.54017 > 10.88.0.116.80: Flags [S], cksum 0xd776 (incorrect -> 0xd8b9), seq 2337032705, win 65535, options [mss 1460], length 0

See similar issue here.

The solution is upgrading to nftables v0.9.3 (Topsy).