GOnetstat/port resource fail to notice net.ipv6.bindv6only - validation fails when dual mode ipv4/6 port is available

[dominics]

When sysctl net.ipv6.bindv6only is 0 (the default value), ports shown as tcp6 in netstat output are may be actually listening on both protocols. If I start with a Goss check like this:

port:
  tcp:12345:
    listening: true

And I then create a simple container, exposing a port on all interfaces (ipv4 and ipv6), like so:

$ docker run -d -p 12345:80 --name goss-test nginx

Then I'll find that I can correctly use the listening port via ipv4:

$ nc -z 127.0.0.1 12345 ; echo $?
0

However, goss will not validate the port correctly:

$ goss validate
F

Failures/Skipped:

Port: tcp:12345: listening:
Expected
    <bool>: false
to equal
    <bool>: true

Total Duration: 0.016s
Count: 1, Failed: 1, Skipped: 0

What this boils down to is: goss is ensuring netstat's output, not the actual state of the ports.

Isn't netstat generally deprecated in favour of ss, because ss does a better job of not confusing the user in ipv6 situations like this one? e.g.

$ sudo ss -lapn | grep 12345
tcp    LISTEN     0      512                   :::12345                :::*      users:(("docker-proxy",25454,4))

versus

$ sudo netstat -lapn | grep 12345
tcp6       0      0 :::12345                :::*                    LISTEN      25454/docker-proxy

(ss is more correct; this is a tcp port, not a tcp6 specific one)

[aelsabbahy]

GOnetstat isn't really "netstat" per say, but rather a low level library that parses /proc/net/* files.

I'll look into the impact of net.ipv6.bindv6only on the proc files and see if there's a way to better handle this setting. But I can't make promises on timeframe, since I'm not sure how easy it will be for me to find this information.

That said, if you or anyone else knows how net.ipv6.bindv6only is handled behind the scenes in ss or impacts /proc/net/, it would be a major help and speed up the fix greatly.

[santiagax99]

Maybe it will be useful:
Try to get info about port 2200

$ netstat -lapn | grep 2200
tcp        0      0 0.0.0.0:2200            0.0.0.0:*               LISTEN      -
tcp        0      0 10.148.198.222:2200     10.128.104.133:53786    ESTABLISHED -
tcp        0    200 10.148.198.222:2200     10.128.82.234:52288     ESTABLISHED -
tcp6       0      0 :::2200                 :::*                    LISTEN      -

Convert decimal 2200 to hex

$ printf "%x\n" 2200
898

Find out information about 0898 in /proc/net/tcp

$ cat /proc/net/tcp | grep 0898
  16: 00000000:0898 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 2568782053 1 ffff88345d94e0c0 99 0 0 10 -1
  94: DEC6940A:0898 8568800A:D21A 01 00000000:00000000 02:000087B6 00000000     0        0 2255852539 2 ffff8801327640c0 48 3 30 10 -1
 242: DEC6940A:0898 EA52800A:CC40 01 000000BC:00000000 01:00000017 00000000     0        0 2476527846 4 ffff88188853c7c0 24 3 17 10 15

And in /proc/net/tcp6

$ cat /proc/net/tcp6 | grep 0898
   4: 00000000000000000000000000000000:0898 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 2568782055 1 ffff885fd7238080 99 0 0 2 -1

Headers of /proc/net/tcp

$ cat /proc/net/tcp | head -n 1
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode

Headers of /proc/net/tcp6

$ cat /proc/net/tcp6 | head -n 1
  sl  local_address                         remote_address                        st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode

st value for LISTENING state is 0A

and local_address keep IP:PORT data in hex

so try to find out listening ipv6 ports from /proc/net/tcp6:

$ cat /proc/net/tcp6 | grep " 0A " | awk '{print $2}' | cut -d: -f2  | xargs -I %% printf "%d\n" 0x%%
5672
3306
11211
22
2200

and verify this information with netstat:

$ netstat -tunap6 | grep LISTEN | awk '{print $4}' |rev |cut -d: -f1|rev
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
5672
3306
11211
22
2200

similar approach with /proc/net/tcp for ipv4

And with docker ports

# docker ps
CONTAINER ID        IMAGE                                                COMMAND                  CREATED             STATUS              PORTS                     NAMES
1f142a08c558        mysql:5.6                                            "docker-entrypoint.sh"   9 hours ago         Up 9 hours          0.0.0.0:29999->3306/tcp   some-mysql

# sudo netstat -lapn | grep 29999
tcp6       0      0 :::29999                :::*                    LISTEN      7631/docker-proxy

# cat /proc/net/tcp6 | grep " 0A " | awk '{print $2}' | cut -d: -f2  | xargs -I %% printf "%d\n" 0x%% | grep 29999
29999

[aelsabbahy]

ss does not print out the correct values either, it just says tcp for all, for example:

netstat:

$ netstat -lna|grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN

ss:

ss -lna |grep 8080
tcp    LISTEN     0      128      :::8080                 :::*                  
tcp    LISTEN     0      0        :::8080                 :::*                  

[aelsabbahy]

Copied from duplicate thread:
Random thought.. maybe it would be simpler to get rid of the whole tcp: vs tcp6: thing all together.

We can collapse the tests into two categories: tcp (default) and udp, if the user wants to make more specific assertions, they can use the ip: attribute.

The only downside is if a user wants to test one or the other, it would be an awkward syntax:

port:
  tcp:22:
    listening: true
    ip:
      and:
      - not:
          contain-element: '::'
      - contain-element: 0.0.0.0

Thoughts on this approach? I'm just thinking out-loud on this.. not 100% sure it's a good idea yet.

@frezbo Brought up a good point about this complicating the syntax. It seems like a trade off, the syntax would be simpler for just checking if a port is listening, but more complicated if you're trying to test that it's listening only tcp but not tcp6 and vice versa.

[pysysops]

Okay, I have an idea:

Things we know: If net.ipv6.bindv6only is 0 then anything bound to all interfaces will be presented to the OS as listening on the IPv6 address: "::" but it's also listening on the IPv4 address "0.0.0.0".

Processes bound to specific addresses aren't bound to both IPv6 and IPv4 unless specified.

So, my thought is that the simplest way to get the expected behaviour from Goss would be to do something like this in the port code:

....
	v6only, _ := sysctl.Get("net.ipv6.bindv6only")
	....
	net = "tcp6"
	for _, entry := range netstat {
		if entry.State == "LISTEN" {
			port := strconv.FormatInt(entry.Port, 10)
			ports[net+":"+port] = append(ports[net+":"+port], entry)
			if entry.Ip == "::" && v6only == "0" {
				entry.Ip = "0.0.0.0"
				ports["tcp:"+port] = append(ports["tcp:"+port], entry)
			}
		}
	}
....

I know this may not look ideal immediately, but considering the alternatives (try to find and integrate another go package or use the above syntax) I think it probably is the MVP of achieving the output we expect from the test. I would need to investigate around the errors thrown by sysctl.Get() and the value returned on error.

Any thoughts or critique?

[aelsabbahy]

Processes bound to specific addresses aren't bound to both IPv6 and IPv4 unless specified.

Are you sure this is true? I haven't tinkered enough with it to know.. but if the bindv6only affects only wildcard IPs, then your solution would be sufficient.

[pysysops]

By that statement I mean that if I run say tomcat bound to an interface such as: 192.168.10.10:8080 it wouldn't be (this is an assumption) bound to an IPv6 address unless specified.

I may have to do some testing around this though tbh. Maybe it's possible to bind to a specific IPv4 address but bind to all IPv6 interfaces which would result in "::" but shouldn't output 0.0.0.0 in tcp port binding. as I would expect the process to report that port 8080 is already bound to.

I'll double check it, I'm confused too now...

[jumping]

After added the "spew.Dump(system.Ports())" in the system/port.go file then could get all ports information on the machine.

func NewDefPort(port string, system *System, config util.Config) Port {
	p := normalizePort(port)
	spew.Dump(system.Ports())
	return &DefPort{
		port:     p,
		sysPorts: system.Ports(),
	}
}

For example, the output of the ss command:

ss -lna|grep 9090
LISTEN     0      128                      :::9090                    :::*

the above will print like as the flowing:

....
 (string) (len=9) "tcp6:9090": ([]GOnetstat.Process) (len=1 cap=1) {
  (GOnetstat.Process) {
   User: (string) (len=4) "root",
   Name: (string) "",
   Pid: (string) "",
   Exe: (string) "",
   State: (string) (len=6) "LISTEN",
   Ip: (string) (len=2) "::",
   Port: (int64) 9090,
   ForeignIp: (string) (len=2) "::",
   ForeignPort: (int64) 0
  }
....

So the goss.yaml could as

port:
  tcp6:9090:
    listening: true
    ip:
    - '::'

[apolegoshko]

Is it finally implemented ?

[aelsabbahy]

I've marked this ticket as "help wanted" a while back. I would be interested in knowing which tools handle this correctly. See my comment above regarding netstat and ss #149 (comment)