nomad port-forward command

[schmichael]

Kubernetes has a port-forward command that allows operators to dynamically and ephemerally forward a local port to a remote pod for debugging and other operations.

This feature seems particularly useful when using Consul Connect as Connect's mTLS requirements make it difficult for operators to peek at Connectified services.

Implementation

Port forwarding should use Nomad's existing region-aware RPC infrastructure to allow forwarding ports across regions.

Implementation on the client-side (CNI? driver specific?) is TBD.

Security

A new ACL capability would be added: namespace:alloc-net (name TBD). While port forwarding offers a similarly high level of container access as namespace:alloc-exec, this feature should have a distinct ACL to avoid having to give operators remote execution privileges.

[lukluk]

port-forward but for tooling purposes, we can just deploy util job then exec and play around there, but if you want port-forward for connections to different cluster better deploy envoy or another proxy

[picatz]

I really wanted this feature this past Sunday after deploying a service with an HTTP API I didn't want to expose to my SSH bastion, or any other networked services within my cluster, or fiddle with additional load balancers, or edge router configs. I found myself using nomad alloc exec to get a shell within the running task. But, the container didn't include a lot of tooling I needed to interact with the service, and often times those tools are already installed or available in a local container on my host.

I just wanted to forward the traffic from a local listener on my host to an upstream service available to the task.

laptop-browser → nomad-cli-listener → load-balancer → nomad-server → nomad-client → nomad-driver → nomad-task → nomad-alloc → upstream-service-listener

So, I went down a fairly deep Nomad-shaped rabbit hole to understand how to make this a reality. Along the way, I learned a lot -- like, how it actually sort'of already exists, and how it could work better in the future. 🕳🐇

Port Forwarding with nomad alloc exec

It's already possible to wrap nomad alloc exec to facilitate what effectively gives you a port-forward command, but with three requirements:

• Your local nomad CLI has been configured with an ACL token (with alloc-exec permissions) if ACLs are enabled, with any other environment variables set to be able to connect, authenticate, and authorize access to the cluster.
• You will need to install, or already have installed socat (or equivalent byte-facilitator) so that you can run that command within the execution context of our target task in order to pass bytes through STDIN to whatever upstream service available available for the container and recv more back from STDOUT.
• You have a nomad command wrapper (like the one below) that can start a listener which sends those bytes to the STDIN of the nomad alloc exec command, and then read the response out of its STDOUT back to the client.

A shell's STDIN/STDOUT can just be a TCP proxy.

package main

import (
	"flag"
	"fmt"
	"log"
	"net"
	"os"
	"os/exec"
	"strings"
)

func main() {
	task := flag.String("task", "", "task name if alloc contains multiple")
	socatPath := flag.String("socat-path", "/usr/bin/socat", "path to socat binary in task")
	portMap := flag.String("p", "", "port mapping local_port:remote_port")
	flag.Parse()

	args := flag.Args()
	if len(args) != 1 {
		log.Fatalf("expected 1 alloc argument given %d", len(args))
	}

	portMapParts := strings.Split(*portMap, ":")
	if len(portMapParts) != 2 {
		log.Fatalf("expected 2 parts (local_port:remote_port) for -p flag, given %d", len(portMapParts))
	}

	ln, err := net.Listen("tcp", fmt.Sprintf("localhost:%s", portMapParts[0]))
	if err != nil {
		log.Fatalf("failed to create local listener: %v", err)
	}
	defer ln.Close()

	log.Printf("started local server: %v", ln.Addr())
	for {
		conn, err := ln.Accept()
		if err != nil {
			log.Fatalf("failed to accept new connection: %v", err)
		}
		log.Printf("accepted new connection: %v", conn.RemoteAddr())
		go func(conn net.Conn) {
			defer conn.Close()
			defer log.Printf("closed connection: %v", conn.RemoteAddr())

			argsStr := fmt.Sprintf("alloc exec -i -t=false -task=%s %s %s - TCP4:localhost:%s", *task, args[0], *socatPath, portMapParts[1])

			log.Printf("running command: nomad %s", argsStr)
			cmd := exec.Command("nomad", strings.Split(argsStr, " ")...)

			cmd.Stdin = conn
			cmd.Stdout = conn
			cmd.Stderr = os.Stderr

			err = cmd.Run()
			if err != nil {
				log.Printf("nomad exec command error: %v", err)
				return
			}
		}(conn)
	}
}

$ go run main.go -p 3100:3100 -task=$TASK_NAME $ALLOC_ID
2021/04/09 16:55:35 started local server: 127.0.0.1:3100
2021/04/09 16:55:37 accepted new connection: 127.0.0.1:60777
2021/04/09 16:55:37 running command: nomad alloc exec -i -t=false -task=promtail 0d253bda /usr/bin/socat - TCP4:localhost:3100
2021/04/09 16:55:39 closed connection: 127.0.0.1:60777

$ curl http://localhost:3100/config
...

☝️ From my laptop I can now curl the upstream Loki service running in a different Nomad task available to the promtail task exposed through an Envoy sidecar managed by Consul on localhost:3100 within the container running promtail. It's localhost turtles all the way down.

From an ACL security perspective, this means that alloc-exec is essentially alloc-net (or whatever it will be named) unless you explicitly prevent applications like socat from running in your cluster, which could be done in various ways.

Native Port Forwarding with nomad alloc port-forward

While an exec port-forward is one solution, I don't think it's ideal. For one, it would require having socat installed on any container/vm/host running your alloc/task. It also doesn't allow you to cleanly separate TCP port-forward access from exec access, even if those lines can be still be blurry for various reasons.

I started chipping away at figuring out all the pieces to support a "native" port forwarding experience following @notnoop's work in #5632, since it's so incredibly similar. I have some working tests, RPC endpoints, and task driver changes.

I plan to make a PR in the near future, and look forward to feedback!

[wilzbach]

Just wanted to chip on @picatz awesome existing workaround.
If someone doesn't want to use Go, it's possible to use socat on your local machine too:

socat tcp4-listen:1234,reuseaddr,fork system:'(nomad alloc exec <job/task params> -i socat - TCP4:yourhost:1234)'