Add eBPF connection tracking with gobpf #2024
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
getWalkedProcPid() reads latestBuf every 3 seconds (for each report). But performWalk() writes latestBuf every 10 seconds or so. So we need to be able to read the same buffer several times.
This is done with the following command: $ gvt fetch github.com/iovisor/gobpf
Based on work from Lorenzo, updated by Iago and Alban This is the second attempt to add eBPF connection tracking. The first one was via weaveworks#1967 by forking a python script using bcc. This one is done in Golang directly thanks to [gobpf](https://github.com/iovisor/gobpf). This is not enabled by default. For now, it should be enabled manually with: ``` sudo ./scope launch --probe.ebpf.connections=true ``` Scope Probe also falls back on the the old /proc parsing if eBPF is not working (e.g. too old kernel, or missing kernel headers). This allows scope to get notified of every connection event, without relying on the parsing of /proc/$pid/net/tcp{,6} and /proc/$pid/fd/*, and therefore improve performance. The eBPF program is in probe/endpoint/ebpf.go. It was discussed in bcc via iovisor/bcc#762. It is using kprobes on the following kernel functions: - tcp_v4_connect - inet_csk_accept - tcp_close It generates "connect", "accept" and "close" events containing the connection tuple but also the pid and the netns. probe/endpoint/ebpf.go maintains the list of connections. Similarly to conntrack, we keep the dead connections for one iteration in order to report the short-lived connections. The code for parsing /proc/$pid/net/tcp{,6} and /proc/$pid/fd/* is still there and still used at start-up because eBPF only brings us the events and not the initial state. However, the /proc parsing for the initial state is now done in foreground instead of background, via newForegroundReader(). NAT resolutions on connections from eBPF works in the same way as it did on connections from /proc: by using conntrack. One of the two conntrack instances was removed since eBPF is able to get short-lived connections. The Scope Docker image is bigger because we need a few more packages for bcc: - weaveworks/scope in current master: 22 MB (compressed), 71 MB (uncompressed) - weaveworks/scope with this patchset: 83 MB (compressed), 223 MB (uncompressed) But @iaguis has ongoing work to reduce the size of the image. Limitations: - [ ] Does not support IPv6 - [ ] Sets `procspied: true` on connections coming from eBPF - [ ] Size of the Docker images - [ ] Requirement on kernel headers for now - [ ] Location of kernel headers: iovisor/bcc#743 Fixes weaveworks#1168 (walking /proc to obtain connections is very expensive) Fixes weaveworks#1260 (Short-lived connections not tracked for containers in shared networking namespaces)
The probe forks 2 conntracks process: - to catch short-lived connections - to apply NAT When eBPF is used to get the set of connections, we don't need the first conntrack anymore after initialization. This patch removes it after initialization.
Start from ubuntu:xenial and do all the RUN operations in one command. Remove unnecessary files like perl/python3 and the udev hwdb.
iaguis
force-pushed
the
alban/conn-perf-ebpf-pr-2
branch
from
4f87213 to
8ba0b07
Compare
|
This PR (ebpf using bcc/gobpf) is replaced by #2070 (ebpf in Go without bcc, to avoid run-time dependency on kernel headers). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the second attempt to add eBPF connection tracking. The first one was via #1967 by forking a python script using bcc. This one is done in Golang directly thanks to gobpf.
This is not enabled by default. For now, it should be enabled manually with:
Scope Probe also falls back on the the old /proc parsing if eBPF is not working (e.g. too old kernel, or missing kernel headers).
This allows scope to get notified of every connection event, without relying on the parsing of
/proc/$pid/net/tcp{,6}and/proc/$pid/fd/*, and therefore improve performance.The eBPF program is in
probe/endpoint/ebpf.go. It was discussed in bcc via iovisor/bcc#762.It is using kprobes on the following kernel functions:
It generates "connect", "accept" and "close" events containing the connection tuple but also the pid and the netns.
probe/endpoint/ebpf.gomaintains the list of connections. Similarly to conntrack, we keep the dead connections for one iteration in order to report the short-lived connections.The code for parsing
/proc/$pid/net/tcp{,6}and/proc/$pid/fd/*is still there and still used at start-up because eBPF only brings us the events and not the initial state. However, the /proc parsing for the initial state is now done in foreground instead of background, vianewForegroundReader().NAT resolutions on connections from eBPF works in the same way as it did on connections from /proc: by using conntrack. One of the two conntrack instances was removed since eBPF is able to get short-lived connections.
The Scope Docker image is bigger because we need a few more packages
for bcc:
But @iaguis still has ongoing work to reduce the size of the image.
Limitations:
procspied: trueon connections coming from eBPFFixes #1168 (walking /proc to obtain connections is very expensive)
Fixes #1260 (Short-lived connections not tracked for containers in shared networking namespaces)