NETOBSERV-1233: Remove L3 fields for ARP ethtype packets and update unit-test #471

msherif1234 · 2023-08-10T12:37:15Z

remove l3 fields fom FLP storgae for layer2 EthTypes protocol like ARP
extend unit-test to test various protocols and L2 protocols
update conntrack to skip l2 flows

Signed-off-by: Mohamed Mahmoud <mmahmoud@redhat.com>

codecov · 2023-08-10T13:07:54Z

Codecov Report

Merging #471 (fa5ecbf) into main (5912edf) will decrease coverage by 0.06%.
Report is 1 commits behind head on main.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #471      +/-   ##
==========================================
- Coverage   66.08%   66.03%   -0.06%     
==========================================
  Files          94       94              
  Lines        6877     6921      +44     
==========================================
+ Hits         4545     4570      +25     
- Misses       2071     2092      +21     
+ Partials      261      259       -2

Flag	Coverage Δ
unittests	`66.03% <100.00%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
pkg/config/generic_map.go	`100.00% <100.00%> (ø)`
pkg/pipeline/decode/decode_protobuf.go	`29.78% <100.00%> (-0.73%)`	⬇️
pkg/pipeline/extract/conntrack/conntrack.go	`93.68% <100.00%> (+0.27%)`	⬆️

msherif1234 · 2023-08-10T13:10:58Z

/ok-to-test

github-actions · 2023-08-10T13:13:41Z

New image:
quay.io/netobserv/flowlogs-pipeline:4e8b12e

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=4e8b12e make set-flp-image

Signed-off-by: Mohamed Mahmoud <mmahmoud@redhat.com>

msherif1234 · 2023-08-10T13:57:29Z

/ok-to-test

github-actions · 2023-08-10T14:00:17Z

New image:
quay.io/netobserv/flowlogs-pipeline:dcf47f6

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=dcf47f6 make set-flp-image

jpinsonneau · 2023-08-10T14:02:40Z

pkg/pipeline/extract/conntrack/conntrack.go

@@ -61,6 +61,11 @@ func (ct *conntrackImpl) Extract(flowLogs []config.GenericMap) []config.GenericM

 	var outputRecords []config.GenericMap
 	for _, fl := range flowLogs {
+		if !fl.IsValidProtocol() {
+			log.Debugf("skipping layer2 protocols flow log %v", fl)
+			ct.metrics.inputRecords.WithLabelValues("discarded").Inc()


Suggested change

ct.metrics.inputRecords.WithLabelValues("discarded").Inc()

ct.metrics.inputRecords.WithLabelValues("rejected").Inc()

Any reason not using the same metric as below ? Else maybe we should make the difference clear for the user

I felt rejected because of any error but in this case there is no error just l2 protocol that we aren't supporting but I am up for better suggestion ?

What about a note in the README explaining the diffs then ?

Hi @jpinsonneau I checked readme it doesn't really explain any of the metrics ? is there another place where this is already doc ?

nope @msherif1234 it's something we should add around the line I mentionned I guess

We have docs about the operational metrics in
https://github.com/netobserv/flowlogs-pipeline/blob/5912edfd278091cd07c693baf27bfbdc8f8408dd/docs/operational-metrics.md
but it is auto-generated and doesn't include the details about label values.

jpinsonneau

Small comment, otherwise looks good ! thanks @msherif1234 !

msherif1234 · 2023-08-10T19:24:10Z

/ok-to-test

github-actions · 2023-08-10T19:35:40Z

New image:
quay.io/netobserv/flowlogs-pipeline:9a62d90

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=9a62d90 make set-flp-image

openshift-ci-robot · 2023-08-11T11:20:59Z

@msherif1234: This pull request references NETOBSERV-1233 which is a valid jira issue.

In response to this:

remove l3 fields fom FLP storgae for layer2 EthTypes protocol like ARP

extend unit-test to test various protocols and L2 protocols

update conntrack to skip l2 flows

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jpinsonneau

LGTM thanks @msherif1234

github-actions · 2023-08-14T13:26:24Z

New image:
quay.io/netobserv/flowlogs-pipeline:c26bbca

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=c26bbca make set-flp-image

ronensc · 2023-08-14T14:22:00Z

pkg/config/generic_map.go

+	return false
+}
+
+func (m GenericMap) FilterFLowLog() bool {


Suggested change

func (m GenericMap) FilterFLowLog() bool {

func (m GenericMap) FilterFlowLog() bool {

I think FilterFlowLog() should be a method conntrackImpl and get a flowLog as an argument. Sorry for not making it clear in my previous comment.

ronensc · 2023-08-14T14:30:17Z

pkg/config/generic_map.go

-const duplicateFieldName = "Duplicate"
+const (
+	duplicateFieldName = "Duplicate"
+	protoFieldName     = "Proto"


My point is:
What if we get flow logs from a source that gives the protocol field a name different from Proto, for example Protocol?

msherif1234 · 2023-08-17T10:22:31Z

/ok-to-test

msherif1234 · 2023-08-17T10:23:02Z

@Amoghrd can u pls check out this fix and add the required tags if changes are ok ?

github-actions · 2023-08-17T10:24:51Z

New image:
quay.io/netobserv/flowlogs-pipeline:6236dec

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=6236dec make set-flp-image

Amoghrd · 2023-08-17T21:24:30Z

/label qe-approved

pkg/pipeline/extract/conntrack/conntrack.go

Signed-off-by: Mohamed Mahmoud <mmahmoud@redhat.com>

msherif1234 · 2023-08-21T13:58:29Z

/approve

openshift-ci · 2023-08-21T13:58:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msherif1234

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [msherif1234]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

msherif1234 added 2 commits August 10, 2023 08:52

Remove L3 info for layer 2 EthType protocols like ARP

23efa77

Signed-off-by: Mohamed Mahmoud <mmahmoud@redhat.com>

update golang packages to include ethernet package

e0eef85

Signed-off-by: Mohamed Mahmoud <mmahmoud@redhat.com>

msherif1234 force-pushed the test_latest branch from 038696f to e0eef85 Compare August 10, 2023 12:58

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 10, 2023

update conntrack to skip l2 protocol flows

16ab32c

Signed-off-by: Mohamed Mahmoud <mmahmoud@redhat.com>

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 10, 2023

msherif1234 requested review from jpinsonneau and ronensc August 10, 2023 13:56

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 10, 2023

jpinsonneau reviewed Aug 10, 2023

View reviewed changes

jpinsonneau previously approved these changes Aug 10, 2023

View reviewed changes

openshift-ci bot assigned jpinsonneau Aug 10, 2023

openshift-ci bot added the lgtm label Aug 10, 2023

msherif1234 dismissed jpinsonneau’s stale review via 977bc15 August 10, 2023 19:23

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 10, 2023

openshift-ci bot added ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. and removed lgtm labels Aug 10, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 11, 2023

msherif1234 requested a review from jpinsonneau August 11, 2023 11:15

msherif1234 changed the title ~~Remove L3 fields for ARP ethtype packets and update unit-test~~ NETOBSERV-1233: Remove L3 fields for ARP ethtype packets and update unit-test Aug 11, 2023

openshift-ci-robot added the jira/valid-reference label Aug 11, 2023

jpinsonneau previously approved these changes Aug 11, 2023

View reviewed changes

msherif1234 force-pushed the test_latest branch from 206b646 to 9983b83 Compare August 14, 2023 14:00

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 14, 2023

msherif1234 force-pushed the test_latest branch from 9983b83 to eacc980 Compare August 14, 2023 14:28

ronensc reviewed Aug 14, 2023

View reviewed changes

msherif1234 force-pushed the test_latest branch from eacc980 to 4f0be98 Compare August 14, 2023 15:04

jpinsonneau previously approved these changes Aug 17, 2023

View reviewed changes

openshift-ci bot added the lgtm label Aug 17, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 17, 2023

openshift-ci bot added the qe-approved QE has approved this pull request label Aug 17, 2023

ronensc previously approved these changes Aug 21, 2023

View reviewed changes

pkg/pipeline/extract/conntrack/conntrack.go Outdated Show resolved Hide resolved

openshift-ci bot assigned ronensc Aug 21, 2023

msherif1234 added 2 commits August 21, 2023 07:43

skip none transport protocols (icmpv4/6)

7f8045e

Signed-off-by: Mohamed Mahmoud <mmahmoud@redhat.com>

update readme with connection tracking mertics

fa5ecbf

Signed-off-by: Mohamed Mahmoud <mmahmoud@redhat.com>

msherif1234 dismissed stale reviews from ronensc and jpinsonneau via fa5ecbf August 21, 2023 11:44

msherif1234 force-pushed the test_latest branch from 4f0be98 to fa5ecbf Compare August 21, 2023 11:44

openshift-ci bot removed the lgtm label Aug 21, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 21, 2023

msherif1234 requested a review from ronensc August 21, 2023 11:58

ronensc approved these changes Aug 21, 2023

View reviewed changes

openshift-ci bot added the lgtm label Aug 21, 2023

openshift-ci bot added the approved label Aug 21, 2023

openshift-merge-robot merged commit 914d470 into netobserv:main Aug 21, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-1233: Remove L3 fields for ARP ethtype packets and update unit-test #471

NETOBSERV-1233: Remove L3 fields for ARP ethtype packets and update unit-test #471

msherif1234 commented Aug 10, 2023 •

edited

Loading

codecov bot commented Aug 10, 2023 •

edited

Loading

msherif1234 commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

msherif1234 commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

jpinsonneau Aug 10, 2023

msherif1234 Aug 10, 2023

jpinsonneau Aug 10, 2023

msherif1234 Aug 10, 2023

jpinsonneau Aug 11, 2023

ronensc Aug 14, 2023

jpinsonneau left a comment

msherif1234 commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

openshift-ci-robot commented Aug 11, 2023 •

edited by openshift-ci bot

Loading

jpinsonneau left a comment

github-actions bot commented Aug 14, 2023

ronensc Aug 14, 2023

ronensc Aug 14, 2023

ronensc Aug 14, 2023

msherif1234 commented Aug 17, 2023

msherif1234 commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

Amoghrd commented Aug 17, 2023

msherif1234 commented Aug 21, 2023

openshift-ci bot commented Aug 21, 2023

	ct.metrics.inputRecords.WithLabelValues("discarded").Inc()
	ct.metrics.inputRecords.WithLabelValues("rejected").Inc()

	func (m GenericMap) FilterFLowLog() bool {
	func (m GenericMap) FilterFlowLog() bool {

NETOBSERV-1233: Remove L3 fields for ARP ethtype packets and update unit-test #471

NETOBSERV-1233: Remove L3 fields for ARP ethtype packets and update unit-test #471

Conversation

msherif1234 commented Aug 10, 2023 • edited Loading

codecov bot commented Aug 10, 2023 • edited Loading

Codecov Report

msherif1234 commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

msherif1234 commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpinsonneau left a comment

Choose a reason for hiding this comment

msherif1234 commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

openshift-ci-robot commented Aug 11, 2023 • edited by openshift-ci bot Loading

jpinsonneau left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msherif1234 commented Aug 17, 2023

msherif1234 commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

Amoghrd commented Aug 17, 2023

msherif1234 commented Aug 21, 2023

openshift-ci bot commented Aug 21, 2023

msherif1234 commented Aug 10, 2023 •

edited

Loading

codecov bot commented Aug 10, 2023 •

edited

Loading

openshift-ci-robot commented Aug 11, 2023 •

edited by openshift-ci bot

Loading