conntrack process executed twice

[alban]

On current master, when running scope without eBPF enabled, I notice two conntrack processes executed with the same parameters:

conntrack --buffer-size 212992 -E -o id -p tcp --any-nat

Both conntrack processes are long-running processes.

I added debug messages in Scope and I see that newConntrackFlowWalker / conntrackWalker is executed twice via:

1. probe/endpoint/reporter.go:NewReporter
2. probe/endpoint/connection_tracker.go:newProcfsConnectionTracker

/cc @2opremio @bboreham
other conntrack issues: #2488 ("Faster conntrack parser"), #2118 ("probe: conntrack: fix output parsing")

[2opremio]

We have always created two long running conntrack processes.

1. To obtain NAT information on connections
2. To obtain short-lived connections when ebpf isn't enabled.

The command invocations used to be different (--any-nat was only provided for (1)) before #2135. Specifically, the --any-nat argument was added here as opposed to how conntrack was originally invoked.

This is a problem. It means that, when eBPF is not enabled, which is the default, we are only showing short-lived connections that are NATed.

This has been happening since 1.3.0 (March 27) and it has prevented Scope from showing short-lived connections between containers living in the same host (which are not NATed).

I am frankly really surprised that nobody reported this explicitly before. Even worse, we didn't have a test for it :(

[2opremio]

As predicted, I can easily reproduce by running these two containers in the same host:

1. docker run --name nginx nginx
2. docker run --name curl tutum/curl bash -c "while true; do curl $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' nginx); sleep 2; done"

Before #2527

After #2527