Add eBPF connection tracking, without dependencies on kernel headers

[iaguis]

Based on work from Lorenzo, updated by Iago, Alban, and Alessandro

This PR adds connection tracking using eBPF. This feature is not enabled by default.
For now, you can enable it by launching scope with the following command:

sudo ./scope launch --probe.ebpf.connections=true

Scope Probe also falls back on the old /proc parsing if eBPF is not working
(e.g. too old kernel, or pre-compiled eBPF program missing for the current kernel).
This patch allows scope to get notified of every connection event, without relying
on the parsing of /proc/$pid/net/tcp{,6} and /proc/$pid/fd/*, and therefore
improve performance.

We vendor https://github.com/kinvolk/gobpf-elf-loader in Scope to load and
receive information from the ebpf programs. The Scope build fetches the pre-compiled
ebpf programs from https://hub.docker.com/r/kinvolk/tcptracer-bpf/
(see https://github.com/kinvolk/tcptracer-bpf)

At the moment, the eBPF programs are pre-compiled for:

• Arch 4.8.11-1
• fedora-24
• debian-testing
• coreos

The ebpf programs use kprobes on the following kernel functions:

• tcp_v4_connect
• tcp_v6_connect
• inet_csk_accept
• tcp_close

It generates "connect", "accept" and "close" events containing the
connection tuple but also pid and netns.
Note: the IPv6 events are not plugged in Scope.

probe/endpoint/ebpf.go maintains the list of connections. Similarly to
conntrack, it also keeps the dead connections for one iteration in order
to report short-lived connections.

The code for parsing /proc/$pid/net/tcp{,6} and /proc/$pid/fd/* is still
there and still used at start-up because eBPF only brings us the events
and not the initial state. However, the /proc parsing for the initial
state is now done in foreground instead of background, via newForegroundReader().

NAT resolution on connections from eBPF works in the same way as it did on connections
from /proc: by using conntrack. One of the two conntrack instances was removed since
eBPF detects short-lived connections.

The Scope Docker image size comparison:

• weaveworks/scope in current master: 22 MB (compressed), 68 MB
  (uncompressed)
• weaveworks/scope with this patchset: 24 MB (compressed), 70 MB
  (uncompressed)

Fixes #1168 (walking /proc to obtain connections is very expensive)
Fixes #1260 (Short-lived connections not tracked for containers in
shared networking namespaces)
Fixes #1962 (Port ebpf tracker to Go)
Fixes #1961 (Remove runtime kernel header dependency from ebpf tracker)

Supersedes #2024

[rade]

Scope fetches the pre-compiled ebpf program from https://hub.docker.com/r/kinvolk/tcptracer-bpf/
(see https://github.com/kinvolk/tcptracer-bpf)

Perhaps include all the binaries in the scope container image? Yes, it adds bloat, but dynamically downloading binaries (on every launch?) is gross.

[alban]

Perhaps include all the binaries in the scope container image? Yes, it adds bloat, but dynamically downloading binaries (on every launch?) is gross.

@rade It is already in the Scope Docker image. It is done at build-time (implemented in Scope's Makefile).

At the moment, the size of the compiled ebpf programs is very small, I think <50KB uncompressed.
https://hub.docker.com/r/kinvolk/tcptracer-bpf/tags/

[rade]

Ah, so when you wrote "Scope fetches" you didn't actually mean "Scope" but "The scope build". Also you wrote "Scope fetches the pre-compiled ebpf program" (note the singular). Taking into account your comments, I believe that sentence should have read "The scope build fetches the pre-compiled ebpf programs." Correct?

[alban]

@rade correct. Sorry for the impreceise wording. I or @iaguis will edit the PR text.

[alban]

I closed the two previous iterations of this work, in favor of this PR. For the record, this was:

• iteration 1: Add eBPF connection tracking #1967 (ebpf using bcc/python)
• iteration 2: Add eBPF connection tracking with gobpf #2024 (ebpf using bcc/gobpf)

[schu]

Two graphs comparing the performance between master and this branch (by running our test dialer, #2082):

With up to 4K connections (40 dialer container with 100 connections each):

With a pool of 100 containers, each holding a random number of connections between 1 and 20:

[iaguis]

Superseded by #2135