VPN PoC

Signed-off-by: Manuel Buil <mbuil@suse.com>

docs/adrs/integrat-vpns.md

@@ -0,0 +1,74 @@
+# Integrate vpn in k3s
+
+Date: 2023-04-26
+
+## Status
+
+Under review
+
+## Context
+
+There are kubernetes use cases which require a kubernetes cluster to be deployed on a set of heterogeneous nodes, i.e. baremetal nodes, AWS VMs, Azure VMs, etc. Some of these use cases:
+	* Edge apps which are divided into two parts: a small footprint part deployed at the edge and the "non-edge" part deployed in the DC. These need to be always connected.
+	* Having a baremetal cluster that requires, only in certain periods, to be extended with hyperscalers VMs to cope out with the demand
+	* Require cluster to include nodes in different hyperscalers due to resiliency reasons or legal requirements, e.g. GDPR
+
+As of today, k3s allows to deploy a cluster on a set of heterogeneous nodes by a simple and robust solution. This is achieved by using the [websocket proxy](https://github.com/k3s-io/k3s/blob/master/pkg/agent/run.go#L277) to connect the control-plane of the cluster, i.e. kube-api <==> kubelet, and a vpn-type flannel backend, e.g. wireguard, to connect the data-plane, i.e. pod <==> pod/node.
+
+The current solution works well but has a few limitations:
+	* It requires the server to have a public IP
+	* It requires the server to open ports on that external IP (e.g. 6443)
+	* There is no central management point for your vpn. Therefore, it is impossible to:
+		1. Have a vpn topology view 
+		2. Monitor node status, performance, etc
+		3. Configure ACLs or other policies
+		4. Other features
+
+There are well known projects which can be used as an alternative to our solution. In general, these projects set up a vpn mesh that includes all nodes and thus we could deploy k3s as if all nodes belonged to the same network. Besides, these projects include a central management point that offer extra features and do not require a public IP to be available. Some of these projects are: tailscale, netmaker, nebula or zerotier.
+
+We already have users that are operating k3s on top of one of these vpn solutions. However, it is sometimes a pain for them because they are not necessarily network experts and run into integration problems such as: performance issues due to double encapsulation, [mtu issues due to vpn tunneling encapsulation](https://github.com/k3s-io/k3s/issues/4743), strange errors due to wrong vpn configuration, etc. Moreover, they need to first deploy the vpn, collect important information and then deploy k3s using that information. These three steps are not always easy to automate and automation is paramount in edge use cases.
+
+My proposal is to integrate the best or the two best of these projects into k3s. Integrating in the sense of setting up the vpn and configuring k3s accordingly, so that the user ends up with a working heterogeneous cluster in a matter of seconds by just deploying k3s. At this stage, the proposal is not to incorporate the vpn binaries or daemons inside the k3s binary, we require the user to have the vpn binary installed.
+
+Therefore, the user would have 1 or 2 alternatives to deploy k3s in an heterogeneous cluster:
+1 - Our simple and robust solution
+2 - vpn solution (e.g. tailscale)
+3 - (Optional) vpn solution 2 (e.g. netmaker)
+
+In terms of support, we could always offer support for alternative 1 and best effort support for alternative 2 (or 3). We don't control those projects and some of them have proprietary parts or are using a freemium business model (e.g. lmimited number nodes)
+
+In the first round, only tailscale will be integrated. Note that the code includes bits and pieces to integrate netmaker but we will add it in future iterations
+
+### Architecture
+
+Going a bit deeper into the code, this is a high-level summary of the changes applied to k3s:
+	* New flag is passed for both server and agent. This flag provide the name and the auth-keys required for the node to be accepted in the vpn and set node configs (e.g. allow routing of podCIDR via the VPN)
+	* Functions that start the vpn and provide information of its status in "package netutil" (pkg/netutil/vpn.go)
+	* VPNInfo struct in "package netutil" that includes important vpn endpoint information such as the vpn endpoint IP and the nodeID in the vpn context (for config purposes)
+	* The collection of vpn endpoint information and its start are implemented by calling the vpn binary. Tailscale has a "not-feature-complete" go client but netmaker does not, so calls to the binary is the common denominator
+	* In the agents, if a vpn config flag is detected, the vpn is started before the websocket proxy is created, so the agent can contact the server
+	* In the servers, if a vpn config flag is detected, the vpn is started before the apiserver is started, so that agents can contact the server. AdvertiseIP is changed to use the VPN IP
+	* If a vpn config flag is detected, the vpn info is collected and the nodeIP replaced before the kubelet certificate is created (due to SAN). This happens in func get(...) of pkg/agent/config/config.go
+	* Two new flannel backends are defined: tailscale and netmaker. These use the general purpose "extension" backend, which executes shell commands when certain events happen (e.g. new node is added)
+	* When a new node is added, flannel queries the subnet podCIDR for that node. The new backends, by executing the vpn binary with certain flags, allow traffic to/from that subnet podCIDR to flow via the VPN
+
+
+## Decision
+
+???
+
+## Consequences
+
+Good
+====
+* Users can automatically deploy vpn+k3s in seconds that seamlessly work and connect heterogeneous nodes
+* New exciting feature for the community
+* We offer not only our simple solution but some extra ones for heterogeneous clusters
+* Fills the gap for useful use cases in edge
+
+Bad
+===
+* Integration with 3rd party projects which we do not control and thus complete support is not possible (similar to CNI plugins)
+* Some of these projects are not 100% open source (e.g. tailscale) and some are in its infancy (i.e. buggy), e.g. netmaker.
+* Not possible to configure a set of heterogeneous nodes in Rancher Management. Therefore, it is currently impossible to deploy through it but could be deployed standalone
+

pkg/agent/config/config.go

@@ -26,6 +26,7 @@ import (
 	"github.com/k3s-io/k3s/pkg/clientaccess"
 	"github.com/k3s-io/k3s/pkg/daemons/config"
 	"github.com/k3s-io/k3s/pkg/daemons/control/deps"
+	"github.com/k3s-io/k3s/pkg/netutil"
 	"github.com/k3s-io/k3s/pkg/util"
 	"github.com/k3s-io/k3s/pkg/version"
 	"github.com/pkg/errors"
@@ -382,6 +383,19 @@ func get(ctx context.Context, envInfo *cmds.Agent, proxy proxy.Proxy) (*config.N
 		return nil, err
 	}
 
+	// If there is a VPN, we must overwrite NodeIP
+	var vpnInfo netutil.VPNInfo
+	if envInfo.VPNAuth != "" {
+		vpnInfo, err := netutil.GetVPNInfo(envInfo.VPNAuth)
+		if err != nil {
+			return nil, err
+		}
+		if len(vpnInfo.IPs) != 0 {
+			logrus.Debugf("Detected node-ips changed to %v due to VPN", vpnInfo.IPs)
+			nodeIPs = vpnInfo.IPs
+		}
+	}
+
 	nodeExternalIPs, err := util.ParseStringSliceToIPs(envInfo.NodeExternalIP)
 	if err != nil {
 		return nil, fmt.Errorf("invalid node-external-ip: %w", err)
@@ -532,6 +546,20 @@ func get(ctx context.Context, envInfo *cmds.Agent, proxy proxy.Proxy) (*config.N
 		nodeConfig.AgentConfig.CNIBinDir = filepath.Dir(hostLocal)
 		nodeConfig.AgentConfig.CNIConfDir = filepath.Join(envInfo.DataDir, "agent", "etc", "cni", "net.d")
 		nodeConfig.AgentConfig.FlannelCniConfFile = envInfo.FlannelCniConfFile
+
+		// It does not make sense to use VPN without its flannel backend
+		if envInfo.VPNAuth != "" {
+			nodeConfig.AgentConfig.NetmakerNodeID = vpnInfo.NodeID
+			nodeConfig.AgentConfig.NetmakerAPIKey = vpnInfo.VPNAPIKey
+
+			if strings.Contains(envInfo.VPNAuth, "tailscale") {
+				nodeConfig.FlannelBackend = "tailscale"
+			}
+
+			if strings.Contains(envInfo.VPNAuth, "netmaker") {
+				nodeConfig.FlannelBackend = "netmaker"
+			}
+		}
 	}
 
 	if nodeConfig.Docker {

pkg/agent/flannel/setup.go

@@ -68,6 +68,12 @@ const (
 	"PSK": "%psk%"
 }`
 
+	tailscaledBackend = `{
+	"Type": "extension",
+	"PostStartupCommand": "tailscale up --accept-routes --advertise-routes=$SUBNET",
+	"ShutdownCommand": "tailscale down"
+}`
+
 	wireguardNativeBackend = `{
 	"Type": "wireguard",
 	"PersistentKeepaliveInterval": %PersistentKeepaliveInterval%,
@@ -80,6 +86,14 @@ const (
 	ipv6
 )
 
+var (
+	netmakerBackend = `{
+		"Type": "extension",
+		"PostStartupCommand": "curl -H 'Authorization: Bearer %NETMAKERKEY%' -H 'Content-Type: application/json' -d '{\"interface\":\"cni0\",\"natenabled\":\"true\",\"ranges\":[\"'$SUBNET'\"]}'  https://api.mbuilnetmaker.ydns.eu/api/nodes/mynet/%NETMAKERNODEID%/creategateway",
+		"ShutdownCommand": "netclient disconnect && netclient leave -n mynet"
+	}`
+)
+
 func Prepare(ctx context.Context, nodeConfig *config.Node) error {
 	if err := createCNIConf(nodeConfig.AgentConfig.CNIConfDir, nodeConfig); err != nil {
 		return err
@@ -210,6 +224,12 @@ func createFlannelConf(nodeConfig *config.Node) error {
 		backendConf = hostGWBackend
 	case config.FlannelBackendIPSEC:
 		logrus.Fatal("The ipsec backend is deprecated and was removed in k3s v1.27; please switch to wireguard-native. Check our docs for information on how to migrate.")
+	case config.FlannelBackendTailscale:
+		backendConf = strings.ReplaceAll(tailscaledBackend, "%flannelConfDir%", filepath.Dir(nodeConfig.FlannelConfFile))
+	case config.FlannelBackendNetmaker:
+		backendConf = strings.ReplaceAll(netmakerBackend, "%flannelConfDir%", filepath.Dir(nodeConfig.FlannelConfFile))
+		backendConf = strings.ReplaceAll(backendConf, "%NETMAKERKEY%", nodeConfig.AgentConfig.NetmakerAPIKey)
+		backendConf = strings.ReplaceAll(backendConf, "%NETMAKERNODEID%", nodeConfig.AgentConfig.NetmakerNodeID)
 	case config.FlannelBackendWireguardNative:
 		mode, ok := backendOptions["Mode"]
 		if !ok {
@@ -225,6 +245,7 @@ func createFlannelConf(nodeConfig *config.Node) error {
 		return fmt.Errorf("Cannot configure unknown flannel backend '%s'", nodeConfig.FlannelBackend)
 	}
 	confJSON = strings.ReplaceAll(confJSON, "%backend%", backendConf)
+	confJSON = strings.ReplaceAll(confJSON, "%CIDR%", nodeConfig.AgentConfig.ClusterCIDR.String())
 
 	logrus.Debugf("The flannel configuration is %s", confJSON)
 	return util.WriteFile(nodeConfig.FlannelConfFile, confJSON)

pkg/cli/agent/agent.go

@@ -76,5 +76,13 @@ func Run(ctx *cli.Context) error {
 
 	contextCtx := signals.SetupSignalContext()
 
+	// Starts the VPN in the agent if config was set up
+	if cmds.AgentConfig.VPNAuth != "" {
+		err := netutil.StartVPN(cmds.AgentConfig.VPNAuth)
+		if err != nil {
+			return err
+		}
+	}
+
 	return agent.Run(contextCtx, cfg)
 }

pkg/cli/cmds/agent.go

@@ -30,6 +30,7 @@ type Agent struct {
 	FlannelIface             string
 	FlannelConf              string
 	FlannelCniConfFile       string
+	VPNAuth                  string
 	Debug                    bool
 	Rootless                 bool
 	RootlessAlreadyUnshared  bool
@@ -151,7 +152,12 @@ var (
 		Usage:       "(agent/networking) Override default flannel cni config file",
 		Destination: &AgentConfig.FlannelCniConfFile,
 	}
-	ResolvConfFlag = &cli.StringFlag{
+	VPNAuthInfo = cli.StringFlag{
+		Name:        "vpn-auth",
+		Usage:       "(agent/networking) VPN auth data for the external VPNs. It must include the VPN and joinKey in the format (name=foo,joinKey=var)",
+		Destination: &AgentConfig.VPNAuth,
+	}
+	ResolvConfFlag = cli.StringFlag{
 		Name:        "resolv-conf",
 		Usage:       "(agent/networking) Kubelet resolv.conf file",
 		EnvVar:      version.ProgramUpper + "_RESOLV_CONF",
@@ -254,6 +260,7 @@ func NewAgentCommand(action func(ctx *cli.Context) error) cli.Command {
 			PreferBundledBin,
 			// Deprecated/hidden below
 			DockerFlag,
+			VPNAuthInfo,
 		},
 	}
 }

pkg/cli/cmds/server.go

@@ -501,6 +501,7 @@ var ServerFlags = []cli.Flag{
 	FlannelIfaceFlag,
 	FlannelConfFlag,
 	FlannelCniConfFileFlag,
+	VPNAuthInfo,
 	ExtraKubeletArgs,
 	ExtraKubeProxyArgs,
 	ProtectKernelDefaultsFlag,

pkg/cli/server/server.go

@@ -91,6 +91,14 @@ func run(app *cli.Context, cfg *cmds.Server, leaderControllers server.CustomCont
 		}
 	}
 
+	// Starts the VPN in the server if config was set up
+	if cmds.AgentConfig.VPNAuth != "" {
+		err := netutil.StartVPN(cmds.AgentConfig.VPNAuth)
+		if err != nil {
+			return err
+		}
+	}
+
 	agentReady := make(chan struct{})
 
 	serverConfig := server.Config{}
@@ -218,6 +226,17 @@ func run(app *cli.Context, cfg *cmds.Server, leaderControllers server.CustomCont
 		serverConfig.ControlConfig.AdvertiseIP = util.GetFirstValidIPString(cmds.AgentConfig.NodeIP)
 	}
 
+	// if not set, try setting advertise-ip from agent VPN
+	if cmds.AgentConfig.VPNAuth != "" {
+		vpnInfo, err := netutil.GetVPNInfo(cmds.AgentConfig.VPNAuth)
+		if err != nil {
+			return err
+		}
+		if len(vpnInfo.IPs) != 0 {
+			serverConfig.ControlConfig.AdvertiseIP = vpnInfo.IPs[0].String()
+		}
+	}
+
 	// if we ended up with any advertise-ips, ensure they're added to the SAN list;
 	// note that kube-apiserver does not support dual-stack advertise-ip as of 1.21.0:
 	/// https://github.com/kubernetes/kubeadm/issues/1612#issuecomment-772583989

pkg/daemons/config/types.go

@@ -25,6 +25,8 @@ const (
 	FlannelBackendHostGW          = "host-gw"
 	FlannelBackendIPSEC           = "ipsec"
 	FlannelBackendWireguardNative = "wireguard-native"
+	FlannelBackendTailscale       = "tailscale"
+	FlannelBackendNetmaker        = "netmaker"
 	EgressSelectorModeAgent       = "agent"
 	EgressSelectorModeCluster     = "cluster"
 	EgressSelectorModeDisabled    = "disabled"
@@ -115,6 +117,8 @@ type Agent struct {
 	ImageCredProvConfig     string
 	IPSECPSK                string
 	FlannelCniConfFile      string
+	NetmakerNodeID          string
+	NetmakerAPIKey          string
 	PrivateRegistry         string
 	SystemDefaultRegistry   string
 	AirgapExtraRegistry     []string