NETOBSERV-1235 Reporter option issues #366

jpinsonneau · 2023-08-07T17:36:40Z

Followup to fix #311 issues

skip dedup on conversations
metrics queries on ingress by default
expose FlowDirection as filter to let user choose metrics seen
mergeFlowReporters 5 tuple + ingress as favorite (FlowDirection filter override this behavior)

See netobserv/network-observability-operator#402 for metrics consistency

codecov · 2023-08-07T17:40:33Z

Codecov Report

Merging #366 (5a89060) into main (f2a289f) will decrease coverage by 0.60%.
The diff coverage is 53.12%.

❗ Current head 5a89060 differs from pull request most recent head bd7ffc5. Consider uploading reports for the commit bd7ffc5 to get more accurate results

@@            Coverage Diff             @@
##             main     #366      +/-   ##
==========================================
- Coverage   57.83%   57.24%   -0.60%     
==========================================
  Files         166      167       +1     
  Lines        7679     7774      +95     
  Branches      918      924       +6     
==========================================
+ Hits         4441     4450       +9     
- Misses       2971     3056      +85     
- Partials      267      268       +1

Flag	Coverage Δ
uitests	`58.11% <52.00%> (-0.06%)`	⬇️
unittests	`54.83% <57.14%> (-2.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...rc/components/dropdowns/query-options-dropdown.tsx	`82.35% <ø> (ø)`
web/src/components/netflow-traffic.tsx	`55.77% <ø> (ø)`
web/src/model/filters.ts	`72.30% <ø> (ø)`
web/src/utils/columns.ts	`66.53% <ø> (ø)`
web/src/utils/filter-definitions.ts	`66.66% <0.00%> (-4.52%)`	⬇️
pkg/handler/flows.go	`62.02% <25.00%> (-2.45%)`	⬇️
web/src/utils/filter-options.ts	`51.85% <50.00%> (-0.21%)`	⬇️
pkg/model/filters/filters.go	`82.22% <100.00%> (ø)`
web/src/utils/flows.ts	`100.00% <100.00%> (ø)`
web/src/utils/ids.ts	`94.44% <100.00%> (+0.69%)`	⬆️

... and 1 file with indirect coverage changes

github-actions · 2023-08-09T10:50:42Z

New image:
quay.io/netobserv/network-observability-console-plugin:846bd0a

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=846bd0a make set-plugin-image

msherif1234 · 2023-08-09T14:24:22Z

pkg/model/filters/filters.go

-		NewMatch(fields.FlowDirection, `"`+constants.Ingress+`"`),
-		NewMatch(fields.SrcOwnerName, `""`),
+		NewMatch(fields.FlowDirection, `"`+constants.Egress+`"`),
+		NewMatch(fields.DstOwnerName, `""`),


why would reversing the logic works now, I think u saw that same issue either way ?

DNS response codes are in INGRESS flows so I want these first and then fill the gap with the EGRESS ones.
pktDrop seems to be more consistent but I also saw differences

I'm investigating both download / upload from pod cases since I have issue with old implementation and new one

Here are my tests for consistency:

using sampling 1

Download a ubuntu.iso of 1000.73 MB and observe transferts between the IP and Node

Upload the downloaded image and see the differences with the new IP

--- download --- curl https://releases.ubuntu.com/22.04.2/ubuntu-22.04.2-desktop-amd64.iso --output /tmp/ubuntu.iso reinterpret + Egress (old implementation) pod 1.2GB node 1.3GB => same results using ingress no reinterpret + Egress pod 922.7MB node 625.1MB no reinterpret + Ingress (new implementation) pod 9.2MB ??? node 630.4MB --- upload --- curl bashupload.com -T /tmp/ubuntu.iso reinterpret + Egress (old implementation) pod 2.8GB node 4.1GB => same results using ingress no reinterpret + Egress pod 4.4MB ?? node 2.1GB no reinterpret + Ingress (new implementation) pod 2.1GB node 2.1GB

Note:

If you download a file from a pod and set Destination Namespace, you don't see the expected bytes in topology view with my update.
With the original one it worked:

If you upload a file from a pod and set Source Namespace, you see your flows with my update.
With the original one it didn't worked:

So in conclusion we can't do that simplification as is if we want to keep Src / Dst consistency and track both download & upload amounts !

see #366 (comment) solution 😸

pkg/model/filters/filters_test.go

msherif1234 · 2023-08-09T14:27:30Z

web/src/components/netflow-traffic.tsx

@@ -1319,7 +1319,7 @@ export const NetflowTraffic: React.FC<{
          setShowDuplicates,
          allowFlow: isFlow(),
          allowConnection: isConnectionTracking(),
-          allowShowDuplicates: selectedViewId === 'table',
+          allowShowDuplicates: selectedViewId === 'table' && recordType !== 'allConnections',


can u explain allconnections check ?

On frontend side, you can only switch between 'flowLogs' or 'allConnections'.

However, conntrack events doesn't have the "Duplicate" field. It actually doesn't make sense to filter on this so I disable the checkbox in the interface.

FYI the connection tracking automatically skip duplicates while doing the aggregates
https://github.com/netobserv/flowlogs-pipeline/blob/main/pkg/pipeline/extract/conntrack/conntrack.go#L71-L73

github-actions · 2023-08-10T09:38:56Z

New image:
quay.io/netobserv/network-observability-console-plugin:c9dbdcb

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=c9dbdcb make set-plugin-image

jpinsonneau · 2023-08-21T11:19:45Z

I finally got the expected behavior adding the FlowDirection filter and using netobserv/network-observability-operator#402 to remove the reinterpret direction.
That approach is based on Ingress by default but allow the user to add a filter on Ingress, Egress or both.

Download scenario:
Upload scenario:

The numbers are also correct here (using sampling 50):

sh-4.4$ curl bashupload.com -T /tmp/ubuntu.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 99 4699M    0     0   99 4688M      0  15.6M  0:05:00  0:04:59  0:00:01 15.2M

92.8*50 = 4640 (node) / 93.2*50 = 4660 (pod) which is almost the expected 4688M

github-actions · 2023-08-21T11:24:47Z

New image:
quay.io/netobserv/network-observability-console-plugin:27681b6

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=27681b6 make set-plugin-image

jpinsonneau · 2023-08-21T14:54:09Z

Also took the opportunity to manage routing for topology view bd7ffc5

When both pod <> external & node<>external edges are available, the final render will be pod<>node<>external

Else it will fallback on previous behavior:

github-actions · 2023-08-21T15:02:24Z

New image:
quay.io/netobserv/network-observability-console-plugin:47254cc

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=47254cc make set-plugin-image

jotak · 2023-08-22T15:07:28Z

/lgtm
thanks!

jpinsonneau · 2023-08-23T14:06:18Z

I'm merging this as the impact is quite huge. Let's open followups if needed.

jpinsonneau · 2023-08-23T14:06:23Z

/approve

openshift-ci · 2023-08-23T14:06:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpinsonneau

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jpinsonneau]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress label Aug 7, 2023

jpinsonneau changed the title ~~WIP Reporter option issues~~ WIP NETOBSERV-1235 Reporter option issues Aug 7, 2023

jpinsonneau force-pushed the 1235 branch 2 times, most recently from fceb0c5 to e9f939a Compare August 9, 2023 10:30

jpinsonneau marked this pull request as ready for review August 9, 2023 10:44

jpinsonneau changed the title ~~WIP NETOBSERV-1235 Reporter option issues~~ NETOBSERV-1235 Reporter option issues Aug 9, 2023

openshift-ci bot removed the do-not-merge/work-in-progress label Aug 9, 2023

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 9, 2023

jpinsonneau requested review from msherif1234 and OlivierCazade August 9, 2023 10:51

msherif1234 reviewed Aug 9, 2023

View reviewed changes

pkg/model/filters/filters_test.go Outdated Show resolved Hide resolved

msherif1234 reviewed Aug 9, 2023

View reviewed changes

pkg/model/filters/filters_test.go Outdated Show resolved Hide resolved

msherif1234 reviewed Aug 9, 2023

View reviewed changes

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 9, 2023

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 10, 2023

jpinsonneau mentioned this pull request Aug 10, 2023

NETOBSERV-1109 remove reinterpret direction netobserv/network-observability-operator#402

Closed

jpinsonneau added 6 commits August 21, 2023 12:25

skip dedup on conversations

f9c1597

metrics query on ingress

86af2d7

mergeFlowReporters tuple + sort

6a7b2b0

fix tests

53b8881

use constants

bd032f1

direction filter

5a89060

jpinsonneau force-pushed the 1235 branch from 0d28ee6 to 5a89060 Compare August 21, 2023 11:12

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 21, 2023

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 21, 2023

jpinsonneau requested a review from jotak August 21, 2023 11:27

manage routing for topology

bd7ffc5

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 21, 2023

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 21, 2023

openshift-ci bot assigned jotak Aug 22, 2023

openshift-ci bot added the lgtm label Aug 22, 2023

openshift-ci bot added the approved label Aug 23, 2023

openshift-merge-robot merged commit 57f6e0f into netobserv:main Aug 23, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-1235 Reporter option issues #366

NETOBSERV-1235 Reporter option issues #366

jpinsonneau commented Aug 7, 2023 •

edited

Loading

codecov bot commented Aug 7, 2023 •

edited

Loading

github-actions bot commented Aug 9, 2023

msherif1234 Aug 9, 2023

jpinsonneau Aug 9, 2023

jpinsonneau Aug 10, 2023

jpinsonneau Aug 10, 2023 •

edited

Loading

jpinsonneau Aug 21, 2023

msherif1234 Aug 9, 2023

jpinsonneau Aug 9, 2023 •

edited

Loading

jpinsonneau Aug 9, 2023

github-actions bot commented Aug 10, 2023

jpinsonneau commented Aug 21, 2023 •

edited

Loading

github-actions bot commented Aug 21, 2023

jpinsonneau commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

jotak commented Aug 22, 2023

jpinsonneau commented Aug 23, 2023

jpinsonneau commented Aug 23, 2023

openshift-ci bot commented Aug 23, 2023

NETOBSERV-1235 Reporter option issues #366

NETOBSERV-1235 Reporter option issues #366

Conversation

jpinsonneau commented Aug 7, 2023 • edited Loading

codecov bot commented Aug 7, 2023 • edited Loading

Codecov Report

github-actions bot commented Aug 9, 2023

msherif1234 Aug 9, 2023

Choose a reason for hiding this comment

jpinsonneau Aug 9, 2023

Choose a reason for hiding this comment

jpinsonneau Aug 10, 2023

Choose a reason for hiding this comment

jpinsonneau Aug 10, 2023 • edited Loading

Choose a reason for hiding this comment

jpinsonneau Aug 21, 2023

Choose a reason for hiding this comment

msherif1234 Aug 9, 2023

Choose a reason for hiding this comment

jpinsonneau Aug 9, 2023 • edited Loading

Choose a reason for hiding this comment

jpinsonneau Aug 9, 2023

Choose a reason for hiding this comment

github-actions bot commented Aug 10, 2023

jpinsonneau commented Aug 21, 2023 • edited Loading

github-actions bot commented Aug 21, 2023

jpinsonneau commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

jotak commented Aug 22, 2023

jpinsonneau commented Aug 23, 2023

jpinsonneau commented Aug 23, 2023

openshift-ci bot commented Aug 23, 2023

jpinsonneau commented Aug 7, 2023 •

edited

Loading

codecov bot commented Aug 7, 2023 •

edited

Loading

jpinsonneau Aug 10, 2023 •

edited

Loading

jpinsonneau Aug 9, 2023 •

edited

Loading

jpinsonneau commented Aug 21, 2023 •

edited

Loading