We need a way to capture short lived connections.

[tomwilkie]

Conntrack will do it, but we can't map it back to a pid (at least on the client side; for servers we could look at the output of lsof)

[tomwilkie]

There may be a way:

https://github.com/mk-fg/conntrack-logger

[tomwilkie]

Nope, that python project is just consulting /proc/net/tcp; their approach won't work with containers, and will only capture slight more connections that we already do.

[tomwilkie]

http://www.spinics.net/lists/netfilter/msg55872.html

[tomwilkie]

netstat seems to do it by walking all the entries in proc:

http://sourceforge.net/p/net-tools/code/ci/master/tree/netstat.c

[tomwilkie]

I think we're going to end up using ftrace on the connect system call https://www.kernel.org/doc/Documentation/trace/ftrace.txt

[tomwilkie]

Okay some more docs about ftrace / perf_events:

• https://www.kernel.org/doc/Documentation/trace/events.txt
• http://www.brendangregg.com/perf.html
• https://perf.wiki.kernel.org/index.php/Main_Page
• https://github.com/brendangregg/perf-tools
• http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html
• http://events.linuxfoundation.org/sites/events/files/slides/perf-collabsummit-2015.pdf
• https://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_rostedt.pdf
• https://lwn.net/Articles/410200/
• https://www.kernel.org/doc/Documentation/trace/uprobetracer.txt
• https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt

Summary: I can happily use ftrace to get a realtime stream of connect and accept syscalls, including the pid & fd. They do not include the ip address, as its a pointer to a userspace struct. I think we might be able to use bpfs to do this.

[tomwilkie]

More info:

• http://events.linuxfoundation.org/sites/events/files/slides/tracing-linux-ezannoni-linuxcon-ja-2015_0.pdf
• http://man7.org/linux/man-pages/man2/perf_event_open.2.html
• https://lkml.org/lkml/2015/6/1/219
• https://lwn.net/Articles/646058/
• https://www.kernel.org/doc/Documentation/networking/filter.txt

Summary: I'm quite confident ftrace + ebpf's will work well, but I think support for this was only merged into kernel 4.1. Kernel 4.1 will most likely be in ubuntu 15.10, so it seems like a dead end for the next ~6months at least

2 ideas left (1 new one):

• combine ftrace & conntrack - when ftrace sees a connect syscall it will know pid and fd; look this up in proc to get src ip and port, then use the asynchronously delivered conntrack info for the dst ip and port
• forget about pid, and do a foreign key join through ip address for the containers view

[tomwilkie]

Status update: got an ftrace prototype (#381), but short answer is it isn't going to work. It can't get the local addr and port number quickly enough from procfs:

Failed to get local addr for pid=708 fd=6: Fd 6 not found for proc 708
Failed to get local addr for pid=764 fd=6: Fd 6 not found for proc 764
Failed to get local addr for pid=708 fd=6: Fd 6 not found for proc 708
Failed to get local addr for pid=745 fd=6: readlink /proc/745/fd/6: no such file or directory
Failed to get local addr for pid=745 fd=8: Fd 8 not found for proc 745
Failed to get local addr for pid=764 fd=6: readlink /proc/764/fd/6: no such file or directory
Failed to get local addr for pid=708 fd=6: Fd 6 not found for proc 708
Failed to get local addr for pid=745 fd=6: Fd 6 not found for proc 745
Failed to get local addr for pid=764 fd=6: readlink /proc/764/fd/6: no such file or directory
Failed to get local addr for pid=745 fd=6: Fd 6 not found for proc 745
Failed to get local addr for pid=745 fd=6: readlink /proc/745/fd/6: no such file or directory
Failed to get local addr for pid=764 fd=6: readlink /proc/764/fd/6: no such file or directory
Failed to get local addr for pid=708 fd=6: readlink /proc/708/fd/6: no such file or directory
Failed to get local addr for pid=708 fd=8: Fd 8 not found for proc 708
Failed to get local addr for pid=746 fd=6: Fd 6 not found for proc 746
Failed to get local addr for pid=746 fd=6: Fd 6 not found for proc 746
Failed to get local addr for pid=1179 fd=6: readlink /proc/1179/fd/6: no such file or directory
Failed to get local addr for pid=1179 fd=6: readlink /proc/1179/fd/6: no such file or directory
Failed to get local addr for pid=764 fd=6: Fd 6 not found for proc 764
Failed to get local addr for pid=708 fd=6: Fd 6 not found for proc 708
Failed to get local addr for pid=745 fd=6: Fd 6 not found for proc 745
Failed to get local addr for pid=746 fd=6: Fd 6 not found for proc 746
Failed to get local addr for pid=746 fd=6: Fd 6 not found for proc 746
Failed to get local addr for pid=764 fd=6: readlink /proc/764/fd/6: no such file or directory

[tomwilkie]

Looks like option (2) is the current leading only candidate - @peterbourgon?

[inercia]

@tomwilkie What about using some conntrack library in Go like this one?

[tomwilkie]

@inercia checkout PR #386 - we are using exactly that library.

The problem is that aren't told the pid which is taking part in the connection by that library, so we're having to do some extra work.

[inercia]

@tomwilkie oh, I see.. and can't we obtain the PID from somewhere in /proc? /proc/<pid>/fd/<n> maps <pid:fd> to ports, so we could walk the whole /proc/<pid>s, build a map and correlate it with the conntrack info, right? Probably you already know all of this, or you even have a better solution in mind...

[tomwilkie]

@inercia Thats exactly what procspy does; the problem is we can't walk too often (every few seconds) as its quite expensive, so we miss particularly short connections. This ticket is/was about finding a different way of doing it, to capture those short lived connections.

[inercia]

oh, it seems I'm quite good at stating the obvious... do you need any help with anything of this?