connection to dead process associated with different process

[rade]

Running

$ scope launch --weave=false --probe.ebpf.connections=false
$ tcpserver -v 0 1122 echo foo

and then

$ nc localhost 1122

produces

i.e. the connection is erroneously associated with the chrome process.

[rade]

During other occurrences of the problem, the destination endpoint got associated with other processes I have running, such as firefox or dropbox. In all cases these are long-running processes, so I don't think the problem is due to pid re-use.

tcpserver forks for new connections, executing the specified program. In the above example the new process terminates quickly, so the connection really shouldn't show up at all in scope since the process associated with the destination is long gone (though the source process remains alive).

When instead running with

$ tcpserver -v 0 1122 cat

which keeps the process running, the endpoint gets associated with the cat process:

That kinda makes sense, but note that when running with eBPF connection tracking enabled...

$ scope launch --weave=false

the endpoint gets associated with the tcpserver process:

This happens for both the echo foo and cat tests.

Here are some relevant details from the reports when the problem occurs...

For the first ~16s the connection doesn't show up. During that time the report contains

"xps;127.0.0.1;55794": {
  "adjacency": ["xps;127.0.0.1;1122"],
  "latest": {
    "conntracked": "true"
  },
  "topology": "endpoint"
}
"xps;127.0.0.1;1122": {
  "latest": {
    "conntracked": "true"
  },
  "topology": "endpoint"
},

i.e. the endpoints are conntracked, and are lacking pids.

Once the connection does show up, the report contains this:

"xps-4026531969;127.0.0.1;55794": {
  "adjacency": ["xps-4026531969;127.0.0.1;1122"],
  "latest": {
    "pid": "20692",
    "procspied": "true"
  },
  "topology": "endpoint"
},
"xps-4026531969;127.0.0.1;1122": {
  "latest": {
    "pid": "17057",
    "procspied": "true"
  },
  "topology": "endpoint"
},

So the endpoints are procspied now and have pids.

The connection disappears after after another 70s, at which point the endpoints look like this:

"xps-4026531969;127.0.0.1;55794": {
  "adjacency": ["xps-4026531969;127.0.0.1;1122"],
  "latest": {
    "pid": "20692",
    "procspied": "true"
  },
  "topology": "endpoint"
},
"xps-4026531969;127.0.0.1;1122": {
  "latest": {
    "procspied": "true"
  },
  "topology": "endpoint"
},

i.e. the destination endpoint is no longer associated with a process.

Throughout, the process topology contains

"xps;20692": {
  "latest": {
    "name": "nc"
  },
  "topology": "process"
},"
xps;17057": {
  "latest": {
    "name": "/opt/google/chrome/chrome"
  },
  "topology": "process"
},

So, as we can see, while the connection was showing up in the UI, the destination endpoint was indeed mysteriously associated with the chrome process.