NETOBSERV-1061: Add TCP drop and DNS tracking hooks #115

msherif1234 · 2023-04-26T19:35:53Z

Add skb_free tracepoint hook to detect when TCP flows are dropped and update flow metrics with tcp socket info.
To view ebpf tracepoints with bpftool

bpftool perf show
pid 119418  fd 8: prog_id 134  tracepoint  kfree_skb

For reference about tcpdrop https://www.brendangregg.com/blog/2018-05-31/linux-tcpdrop.html
Steps for manual testing using flowsdump collector tool

- using ncat
=========
server: ncat -l 8080
client: ncat 192.168.122.69 8080

use iptables rule to drop
============================

sudo iptables -I INPUT 1 -m tcp --proto tcp --dst 192.168.122.69/32 --dport 8080 -j DROP
switch back
sudo iptables -I INPUT 1 -m tcp --proto tcp --dst 192.168.122.69/32 --dport 8080 -j ACCEPT

once the drop iptable rule is installed we can see tcpdrop stats been updated

 },{
                "cpu": 2,
                "value": {
                    "packets": 1,
                    "bytes": 67,
                    "start_mono_time_ts": 5481943029602,
                    "end_mono_time_ts": 5481943029602,
                    "flags": 16,
                    "errno": 0,
                    "tcp_drops": {
                        "packets": 14,
                        "bytes": 800,
                        "flags": 16,
                        "state": 1,
                        "drop_cause": 8  <<<<< SKB_DROP_REASON_NETFILTER_DROP
                    }
                }	
            }

add net_dev_queue trace point hook to implement light weight DNS tracker
testing using dig google.coom

DNS id 23099
Query
 "dns_record": {
                "id": 23099,
                "flags": 288,
                "req_mono_time_ts": 3559391813403,
                "rsp_mono_time_ts": 0
}

Response
 "dns_record": {
                "id": 23099,
                "flags": 33152,
                "req_mono_time_ts": 0,
                "rsp_mono_time_ts": 3559392215340
}
~.4 msec latency

NOTE:

here are all possible TCP state values https://elixir.bootlin.com/linux/v6.3/source/include/net/tcp_states.h#L12
here are all drop causes
https://elixir.bootlin.com/linux/latest/source/include/net/dropreason-core.h#L88

Related PRs:
netobserv/network-observability-operator#331
netobserv/flowlogs-pipeline#429
netobserv/network-observability-console-plugin#324

codecov · 2023-04-26T19:43:48Z

Codecov Report

Merging #115 (2cffd02) into main (2d63d90) will decrease coverage by 0.38%.
The diff coverage is 44.66%.

@@            Coverage Diff             @@
##             main     #115      +/-   ##
==========================================
- Coverage   40.48%   40.11%   -0.38%     
==========================================
  Files          31       31              
  Lines        2060     2124      +64     
==========================================
+ Hits          834      852      +18     
- Misses       1186     1227      +41     
- Partials       40       45       +5

Flag	Coverage Δ
unittests	`40.11% <44.66%> (-0.38%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/agent/agent.go	`39.20% <0.00%> (ø)`
pkg/ebpf/tracer.go	`0.00% <0.00%> (ø)`
pkg/exporter/proto.go	`82.73% <62.06%> (-17.27%)`	⬇️
pkg/flow/record.go	`71.42% <66.66%> (-4.58%)`	⬇️
pkg/flow/account.go	`82.69% <100.00%> (ø)`
pkg/flow/tracer_map.go	`79.41% <100.00%> (ø)`

openshift-ci-robot · 2023-04-26T19:55:20Z

@msherif1234: This pull request references NETOBSERV-979 which is a valid jira issue.

In response to this:

add skb_free tracepoint hook

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-04-27T16:19:39Z

@msherif1234: This pull request references NETOBSERV-979 which is a valid jira issue.

In response to this:

add skb_free tracepoint hook
to view ebpf tracepoints with bpftool
bpftool perf show
pid 119418  fd 8: prog_id 134  tracepoint  kfree_skb

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-04-27T17:15:02Z

@msherif1234: This pull request references NETOBSERV-979 which is a valid jira issue.

In response to this:

inspired by https://www.brendangregg.com/blog/2018-05-31/linux-tcpdrop.html

add skb_free tracepoint hook
to view ebpf tracepoints with bpftool
bpftool perf show
pid 119418  fd 8: prog_id 134  tracepoint  kfree_skb

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-04-27T17:26:44Z

@msherif1234: This pull request references NETOBSERV-979 which is a valid jira issue.

In response to this:

Add skb_free tracepoint hook to detect when TCP flows are dropped and update flow metrics with tcp socket info.

To view ebpf tracepoints with bpftool
bpftool perf show
pid 119418  fd 8: prog_id 134  tracepoint  kfree_skb
For reference about tcpdrop https://www.brendangregg.com/blog/2018-05-31/linux-tcpdrop.html

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-04-27T17:29:17Z

@msherif1234: This pull request references NETOBSERV-979 which is a valid jira issue.

In response to this:

Add skb_free tracepoint hook to detect when TCP flows are dropped and update flow metrics with tcp socket info.

To view ebpf tracepoints with bpftool
bpftool perf show
pid 119418  fd 8: prog_id 134  tracepoint  kfree_skb
For reference about tcpdrop https://www.brendangregg.com/blog/2018-05-31/linux-tcpdrop.html

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msherif1234 · 2023-04-27T17:30:12Z

/assign @praveingk @jotak @jpinsonneau @ronensc

github-actions · 2023-04-28T07:38:31Z

New image: ["quay.io/netobserv/netobserv-ebpf-agent:8197b19"]. It will expire after two weeks.

jpinsonneau · 2023-04-28T07:51:14Z

@msherif1234 I tried to deploy this on a clusterbot using launch 4.13 aws,large but got the following error:

time="2023-04-28T07:46:46Z" level=debug msg="agent IP: 10.0.149.138" component=agent.Flows
time="2023-04-28T07:46:47Z" level=info msg="failed to attach the BPF program to kfree_skb tracepoint: reading file \"/sys/kernel/debug/tracing/events/skb/kfree_skb/id\": open /sys/kernel/debug/tracing/events/skb/kfree_skb/id: no such file or directory" component=ebpf.FlowFetcher
time="2023-04-28T07:46:47Z" level=fatal msg="can't instantiate NetObserv eBPF Agent" error="reading file \"/sys/kernel/debug/tracing/events/skb/kfree_skb/id\": open /sys/kernel/debug/tracing/events/skb/kfree_skb/id: no such file or directory"

I thought using privileged option in the CRD would solve it but it's not 😿

apiVersion: flows.netobserv.io/v1beta1
kind: FlowCollector
...
spec:
  agent:
    ebpf:
...
      privileged: true
...

Am I missing something ?

msherif1234 · 2023-04-28T10:23:27Z

@msherif1234 I tried to deploy this on a clusterbot using launch 4.13 aws,large but got the following error:

time="2023-04-28T07:46:46Z" level=debug msg="agent IP: 10.0.149.138" component=agent.Flows
time="2023-04-28T07:46:47Z" level=info msg="failed to attach the BPF program to kfree_skb tracepoint: reading file \"/sys/kernel/debug/tracing/events/skb/kfree_skb/id\": open /sys/kernel/debug/tracing/events/skb/kfree_skb/id: no such file or directory" component=ebpf.FlowFetcher
time="2023-04-28T07:46:47Z" level=fatal msg="can't instantiate NetObserv eBPF Agent" error="reading file \"/sys/kernel/debug/tracing/events/skb/kfree_skb/id\": open /sys/kernel/debug/tracing/events/skb/kfree_skb/id: no such file or directory"

I thought using privileged option in the CRD would solve it but it's not 😿

apiVersion: flows.netobserv.io/v1beta1
kind: FlowCollector
...
spec:
  agent:
    ebpf:
...
      privileged: true
...

Am I missing something ?

Yes similar to what I added to e2e manifests we need to add the same mount volume to the crd

openshift-ci-robot · 2023-04-28T15:54:38Z

@msherif1234: This pull request references NETOBSERV-979 which is a valid jira issue.

In response to this:

Add skb_free tracepoint hook to detect when TCP flows are dropped and update flow metrics with tcp socket info.
To view ebpf tracepoints with bpftool

bpftool perf show
pid 119418  fd 8: prog_id 134  tracepoint  kfree_skb

For reference about tcpdrop https://www.brendangregg.com/blog/2018-05-31/linux-tcpdrop.html
Steps for manual testing using flowsdump collector tool

using ncat
=========
server ncat -l 8080
client ncat 192.168.122.69 8080


use iptables rule to drop
============================
sudo iptables -I INPUT 1 -m tcp --proto tcp --dst 192.168.122.69/32 --dport 8080 -j DROP
switch back
sudo iptables -I INPUT 1 -m tcp --proto tcp --dst 192.168.122.69/32 --dport 8080 -j ACCEPT

run local flow collector with long time to allow flow to stay longer (2Minutes)
===============================================================================
sudo FLOWS_TARGET_HOST=127.0.0.1 FLOWS_TARGET_PORT=9999 CACHE_ACTIVE_TIMEOUT="120s" ./bin/netobserv-ebpf-agent


use bpftool to find the map and dump it
=======================================
sudo bpftool map list
sudo bpftool map dump id 500 | grep dst_port

use bpftool to look at prog traces
=================================
sudo bpftool prog tracelog

use bpftool to check tracepoint is attached
===========================================
bpftool perf show

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-04-28T19:16:39Z

@msherif1234: This pull request references NETOBSERV-979 which is a valid jira issue.

In response to this:

Add skb_free tracepoint hook to detect when TCP flows are dropped and update flow metrics with tcp socket info.
To view ebpf tracepoints with bpftool

bpftool perf show
pid 119418  fd 8: prog_id 134  tracepoint  kfree_skb

For reference about tcpdrop https://www.brendangregg.com/blog/2018-05-31/linux-tcpdrop.html
Steps for manual testing using flowsdump collector tool

using ncat
=========
server ncat -l 8080
client ncat 192.168.122.69 8080


use iptables rule to drop
============================
sudo iptables -I INPUT 1 -m tcp --proto tcp --dst 192.168.122.69/32 --dport 8080 -j DROP
switch back
sudo iptables -I INPUT 1 -m tcp --proto tcp --dst 192.168.122.69/32 --dport 8080 -j ACCEPT

run local flow collector with long time to allow flow to stay longer (2Minutes)
===============================================================================
sudo FLOWS_TARGET_HOST=127.0.0.1 FLOWS_TARGET_PORT=9999 CACHE_ACTIVE_TIMEOUT="120s" ./bin/netobserv-ebpf-agent


use bpftool to find the map and dump it
=======================================
sudo bpftool map list
sudo bpftool map dump id 500 | grep dst_port

use bpftool to look at prog traces
=================================
sudo bpftool prog tracelog

use bpftool to check tracepoint is attached
===========================================
bpftool perf show

once the drop iptable rule is installed we can see tcpdrop stats been updated

},{
               "cpu": 10,
               "value": {
                   "packets": 7,
                   "bytes": 540,
                   "start_mono_time_ts": 1047585604832,
                   "end_mono_time_ts": 1054159237862,
                   "flags": 16,
                   "errno": 0,
                   "tcp_drops": {
                       "packets": 7,
                       "bytes": 442,
                       "flags": 16,
                       "state": 1
                   }
               }

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msherif1234 · 2023-04-28T20:39:09Z

/ok-to-test

msherif1234 · 2023-06-20T20:55:39Z

/ok-to-test

msherif1234 · 2023-06-20T20:59:13Z

/ok-to-test

github-actions · 2023-06-20T21:00:40Z

New image: quay.io/netobserv/netobserv-ebpf-agent:5aac091. It will expire after two weeks.

jotak · 2023-06-21T14:58:57Z

/lgtm

bpf/utils.h

dushyantbehl · 2023-06-21T18:01:20Z

bpf/utils.h

+    } else if (id->eth_protocol == ETH_P_IPV6) {
+        *family = AF_INET6;
+    }
+ }


Not urgent for this commit but there's functionality being repeated in these functions and the fill_** functions so maybe we should look to just consolidate them.

sure we can do so

Signed-off-by: msherif1234 <mmahmoud@redhat.com>

fix lint errors flatten icmp block Signed-off-by: msherif1234 <mmahmoud@redhat.com>

msherif1234 · 2023-06-23T16:49:00Z

/ok-to-test

github-actions · 2023-06-23T16:50:17Z

New image: quay.io/netobserv/netobserv-ebpf-agent:cfcda6b. It will expire after two weeks.

dushyantbehl

/lgtm

msherif1234 · 2023-06-24T13:27:06Z

/approve

openshift-ci · 2023-06-24T13:27:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msherif1234

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [msherif1234]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

msherif1234 changed the title ~~Add TCP drop hook and update flows metrics~~ WIP: Add TCP drop hook and update flows metrics Apr 26, 2023

openshift-ci bot added the do-not-merge/work-in-progress label Apr 26, 2023

msherif1234 changed the title ~~WIP: Add TCP drop hook and update flows metrics~~ WIP: NETOBSERV-979: Add TCP drop hook and update flows metrics Apr 26, 2023

openshift-ci-robot added the jira/valid-reference label Apr 26, 2023

msherif1234 changed the title ~~WIP: NETOBSERV-979: Add TCP drop hook and update flows metrics~~ NETOBSERV-979: Add TCP drop hook and update flows metrics Apr 27, 2023

openshift-ci bot removed the do-not-merge/work-in-progress label Apr 27, 2023

openshift-ci bot assigned jotak, jpinsonneau, praveingk and ronensc Apr 27, 2023

msherif1234 added the enhancement New feature or request label Apr 27, 2023

msherif1234 requested a review from praveingk April 27, 2023 18:36

msherif1234 changed the title ~~NETOBSERV-979: Add TCP drop hook and update flows metrics~~ NETOBSERV-979: Add TCP drop hook and update flows metric Apr 27, 2023

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 28, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 28, 2023

jpinsonneau mentioned this pull request Apr 28, 2023

NETOBSERV-1064: TCP drop & DNS tracking netobserv/network-observability-console-plugin#324

Closed

6 tasks

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 28, 2023

openshift-ci bot removed the lgtm label Jun 20, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 20, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 20, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 20, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 20, 2023

msherif1234 mentioned this pull request Jun 20, 2023

WIP Fix memory and cpu scale issue work around in #133 #135

Closed

openshift-ci bot added the lgtm label Jun 21, 2023

dushyantbehl reviewed Jun 21, 2023

View reviewed changes

bpf/utils.h Outdated Show resolved Hide resolved

dushyantbehl reviewed Jun 21, 2023

View reviewed changes

openshift-ci bot removed the lgtm label Jun 22, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 22, 2023

msherif1234 added 4 commits June 23, 2023 12:43

Add TCP drop hook and update flows metrics

67fd5c6

Signed-off-by: msherif1234 <mmahmoud@redhat.com>

update agent e2e manifest to mount kernel debug volume

47f2c0f

Signed-off-by: msherif1234 <mmahmoud@redhat.com>

Add light weight DNS tracker hook

5e5aae3

Signed-off-by: msherif1234 <mmahmoud@redhat.com>

rename tcpdrop fields to reflect its the latest drop

2cffd02

fix lint errors flatten icmp block Signed-off-by: msherif1234 <mmahmoud@redhat.com>

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 23, 2023

dushyantbehl approved these changes Jun 24, 2023

View reviewed changes

openshift-ci bot assigned dushyantbehl Jun 24, 2023

openshift-ci bot added the lgtm label Jun 24, 2023

openshift-ci bot added the approved label Jun 24, 2023

openshift-merge-robot merged commit 27baf29 into netobserv:main Jun 24, 2023
9 checks passed

jpinsonneau added the breaking-change This pull request has breaking changes. They should be described in PR description. label Jun 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-1061: Add TCP drop and DNS tracking hooks #115

NETOBSERV-1061: Add TCP drop and DNS tracking hooks #115

msherif1234 commented Apr 26, 2023 •

edited

Loading

codecov bot commented Apr 26, 2023 •

edited

Loading

openshift-ci-robot commented Apr 26, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 27, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 27, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 27, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 27, 2023 •

edited by openshift-ci bot

Loading

msherif1234 commented Apr 27, 2023

github-actions bot commented Apr 28, 2023

jpinsonneau commented Apr 28, 2023

msherif1234 commented Apr 28, 2023

openshift-ci-robot commented Apr 28, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 28, 2023 •

edited by openshift-ci bot

Loading

msherif1234 commented Apr 28, 2023

msherif1234 commented Jun 20, 2023

msherif1234 commented Jun 20, 2023

github-actions bot commented Jun 20, 2023

jotak commented Jun 21, 2023

dushyantbehl Jun 21, 2023

msherif1234 Jun 21, 2023

msherif1234 commented Jun 23, 2023

github-actions bot commented Jun 23, 2023

dushyantbehl left a comment

msherif1234 commented Jun 24, 2023

openshift-ci bot commented Jun 24, 2023

NETOBSERV-1061: Add TCP drop and DNS tracking hooks #115

NETOBSERV-1061: Add TCP drop and DNS tracking hooks #115

Conversation

msherif1234 commented Apr 26, 2023 • edited Loading

codecov bot commented Apr 26, 2023 • edited Loading

Codecov Report

openshift-ci-robot commented Apr 26, 2023 • edited by openshift-ci bot Loading

openshift-ci-robot commented Apr 27, 2023 • edited by openshift-ci bot Loading

openshift-ci-robot commented Apr 27, 2023 • edited by openshift-ci bot Loading

openshift-ci-robot commented Apr 27, 2023 • edited by openshift-ci bot Loading

openshift-ci-robot commented Apr 27, 2023 • edited by openshift-ci bot Loading

msherif1234 commented Apr 27, 2023

github-actions bot commented Apr 28, 2023

jpinsonneau commented Apr 28, 2023

msherif1234 commented Apr 28, 2023

openshift-ci-robot commented Apr 28, 2023 • edited by openshift-ci bot Loading

openshift-ci-robot commented Apr 28, 2023 • edited by openshift-ci bot Loading

msherif1234 commented Apr 28, 2023

msherif1234 commented Jun 20, 2023

msherif1234 commented Jun 20, 2023

github-actions bot commented Jun 20, 2023

jotak commented Jun 21, 2023

dushyantbehl Jun 21, 2023

Choose a reason for hiding this comment

msherif1234 Jun 21, 2023

Choose a reason for hiding this comment

msherif1234 commented Jun 23, 2023

github-actions bot commented Jun 23, 2023

dushyantbehl left a comment

Choose a reason for hiding this comment

msherif1234 commented Jun 24, 2023

openshift-ci bot commented Jun 24, 2023

msherif1234 commented Apr 26, 2023 •

edited

Loading

codecov bot commented Apr 26, 2023 •

edited

Loading

openshift-ci-robot commented Apr 26, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 27, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 27, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 27, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 27, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 28, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Apr 28, 2023 •

edited by openshift-ci bot

Loading