NETOBSERV-1216 PacketDrop enhancements #362

jpinsonneau · 2023-07-26T10:27:28Z

Requires:
netobserv/flowlogs-pipeline#470
netobserv/network-observability-operator#401

This is a followup on NETOBSERV-1064 to improve Packet drop functionnality:

moved ICMP type / code to protocol
moved drop cause to packets
improved descriptions with clickable popovers including link to documentations
allow filter on drop cause / tcp state
removed green color when when drop info are not provided
- flows containing the info will keep green / red colors even if the feature is disabled

codecov · 2023-07-26T10:30:48Z

Codecov Report

Merging #362 (c15abab) into main (13e24e4) will increase coverage by 0.03%.
Report is 1 commits behind head on main.
The diff coverage is 54.78%.

❗ Current head c15abab differs from pull request most recent head 6e698a5. Consider uploading reports for the commit 6e698a5 to get more accurate results

@@            Coverage Diff             @@
##             main     #362      +/-   ##
==========================================
+ Coverage   57.15%   57.18%   +0.03%     
==========================================
  Files         163      167       +4     
  Lines        7543     7738     +195     
  Branches      921      913       -8     
==========================================
+ Hits         4311     4425     +114     
- Misses       2967     3046      +79     
- Partials      265      267       +2

Flag	Coverage Δ
uitests	`58.09% <54.78%> (+0.06%)`	⬆️
unittests	`54.69% <ø> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
web/src/components/filters/filters-toolbar.tsx	`64.17% <ø> (ø)`
...c/components/netflow-overview/netflow-overview.tsx	`67.02% <ø> (+1.39%)`	⬆️
web/src/components/netflow-traffic.tsx	`55.77% <ø> (+3.10%)`	⬆️
web/src/model/filters.ts	`72.30% <ø> (ø)`
web/src/utils/filter-options.ts	`52.05% <30.00%> (-2.64%)`	⬇️
web/src/components/filters/filter-hints.tsx	`63.63% <33.33%> (-11.37%)`	⬇️
web/src/components/netflow-record/record-field.tsx	`69.56% <39.13%> (-5.20%)`	⬇️
...eb/src/components/netflow-overview/panel-kebab.tsx	`60.86% <41.66%> (ø)`
web/src/utils/icmp.ts	`57.69% <58.82%> (-23.44%)`	⬇️
web/src/utils/pkt-drop.ts	`88.88% <88.88%> (ø)`
... and 4 more

... and 19 files with indirect coverage changes

msherif1234 · 2023-07-26T12:21:14Z

web/locales/en/plugin__netobserv-plugin.json

+  "A _LINUX_TCP_STATES_H TCP name like ESTABLISHED, SYN_SENT, SYN_RECV": "A _LINUX_TCP_STATES_H TCP name like ESTABLISHED, SYN_SENT, SYN_RECV",
+  "Packet drop latest cause": "Packet drop latest cause",
+  "Specify a single TCP drop cause.": "Specify a single TCP drop cause.",
+  "Specify a single TCP drop cause like:": "Specify a single TCP drop cause like:",


this isn't TCP specific here and below ?

sorry my bad

msherif1234 · 2023-07-26T12:24:13Z

web/src/utils/filter-definitions.ts

@@ -448,6 +449,34 @@ export const getFilterDefinitions = (
        encoder: simpleFiltersEncoder('_HashId'),
        overlap: false
      },
+      {
+        id: 'pkt_drop_state',


well this filter be off when the dropped packet is for UDP as an example ?

No specific mechanism is done here. The user can set 2 filters as PktDropLatestState=TCP_XXX, Proto=UDP with match any query option to get both dropped TCP & UDP flows

msherif1234 · 2023-07-26T12:25:07Z

web/src/utils/filter-options.ts

@@ -6,6 +6,7 @@ import { splitResource, SplitStage } from '../model/resource';
 import { autoCompleteCache } from './autocomplete-cache';
 import { DNS_RCODES } from './dns';
 import { getPort, getService } from './port';
+import { DROP_CAUSES, DROP_STATES } from './tcp-drop';


pls rename the pkg to pkt-drop instead ?

msherif1234 · 2023-07-26T12:26:30Z

web/src/utils/icmp.ts

+    case 5: // ICMP_REDIRECT:
+      return ICMP_REDIRECT_CODES.find(v => v.value === c);
+    case 11: // ICMP_TIME_EXCEEDED:
+      return ICMP_TIME_EXCEEDED_CODES.find(v => v.value === c);


what about icmp6 codes and types they aren't the same as in icmp4 ?

Correct:
https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml#icmp-parameters-types
https://www.iana.org/assignments/icmpv6-parameters/icmpv6-parameters.xhtml#icmpv6-parameters-2

I should implement both and switch according to for protocol field

1 | ICMP | Internet Control Message vs 58 | IPv6-ICMP | ICMP for IPv6

msherif1234 · 2023-07-26T12:28:54Z

web/src/utils/tcp-drop.ts

+  { value: 72, name: 'SKB_DROP_REASON_IPV6_NDISC_HOP_LIMIT', description: 'invalid hop limit' },
+  { value: 73, name: 'SKB_DROP_REASON_IPV6_NDISC_BAD_CODE', description: 'invalid NDISC icmp6 code' },
+  { value: 74, name: 'SKB_DROP_REASON_IPV6_NDISC_BAD_OPTIONS', description: 'invalid NDISC options' },
+  { value: 75, name: 'SKB_DROP_REASON_IPV6_NDISC_NS_OTHERHOST', description: 'NEIGHBOUR SOLICITATION' }


do we have to maintain this translation from number to string in two components ? if it makes more sense to have it here we probably removed from FLP and just keep it as a number to reduce future efforts adding new drops ?

That's a tradeoff to decide. If we want the user to be able to type either number or name in the filters, we need these in console plugin.

I think we need to be able to filter on both as u have

jotak · 2023-07-27T07:56:15Z

web/src/components/netflow-record/record-field.tsx

+            child = clickableContent(
+              flow.fields.PktDropLatestDropCause,
+              getDropCauseDescription(flow.fields.PktDropLatestDropCause as DROP_CAUSES_NAMES),
+              'https://elixir.bootlin.com/linux/latest/source/include/net/dropreason-core.h#L88'


is there a specific reason to use a URL from elixir.bootlin.com ? (I don't know about this site)
This one sounds more official / primary source: https://github.com/torvalds/linux/blob/master/include/net/dropreason-core.h ?

sure github sounds better 👍

jotak · 2023-07-27T07:57:36Z

web/src/utils/filter-definitions.ts

+        - ${t('A _LINUX_TCP_STATES_H number like 1, 2, 3')}
+        - ${t('A _LINUX_TCP_STATES_H TCP name like ESTABLISHED, SYN_SENT, SYN_RECV')}`,


Like you put a link to the drop reasons, we may have a link to these states definitions? https://github.com/torvalds/linux/blob/master/include/net/tcp_states.h

added doc url in examples popover pointing to github ressources

74edded

jotak · 2023-07-27T08:10:10Z

looks good to me in general (two comments about the external links)
You could also add a couple of tests, for example in https://github.com/netobserv/network-observability-console-plugin/blob/main/web/src/components/filters/__tests__/autocomplete-filter.spec.tsx#L29-L51 we could have similar tests for the new autocompleted fields

jpinsonneau · 2023-07-27T09:08:29Z

You could also add a couple of tests, for example in https://github.com/netobserv/network-observability-console-plugin/blob/main/web/src/components/filters/__tests__/autocomplete-filter.spec.tsx#L29-L51 we could have similar tests for the new autocompleted fields

Currently the new filters are just rejecting empty values validate: rejectEmptyValue,
Do you think we should force the user to the defined list only ? This was made on purpose to allow any new value to be filtered on, even if we didn't defined these.

I tried to check for the popover suggestions but the wrapper.find function can't find it 😞
It seems that the waitForComponentToPaint is not enough to wait for the popover to show or maybe the popover doesn't even show in these tests.

msherif1234 · 2023-07-31T11:11:37Z

/lgtm

Amoghrd · 2023-07-31T14:51:46Z

/ok-to-test

github-actions · 2023-07-31T14:55:43Z

New image:
quay.io/netobserv/network-observability-console-plugin:f125544

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=f125544 make set-plugin-image

jpinsonneau · 2023-08-03T07:06:07Z

Rebased without changes

github-actions · 2023-08-03T07:10:11Z

New image:
quay.io/netobserv/network-observability-console-plugin:32d3f9d

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=32d3f9d make set-plugin-image

msherif1234 · 2023-08-03T11:55:23Z

/lgtm

openshift-ci · 2023-08-04T08:19:00Z

New changes are detected. LGTM label has been removed.

jpinsonneau · 2023-08-04T08:27:33Z

@Amoghrd Addressing feedback from QE doc:

Filters still available when feature is disabled 03d6282
Show others -- can't reproduce all the time

I changed the stat out of show others and it seems to be more consistent d8ab9fd Can you please double check ?
Also took the opportunity to remove others on DNS latency since it's an average and add kebab on all graphs (export was missing on the ones without options)

Weird IP in total panel -- seems to work fine on my cluster; any tips to reproduce ?

Missing DNS options filtering c15abab

jpinsonneau · 2023-08-07T09:45:16Z

While testing I noticed Bytes and Packets fields are not skipped when zero on FLP side.
That create inconsistency between flows and conversations. I've added a commit for that
06ed39b

That makes this PR dependant of netobserv/flowlogs-pipeline#470 one. Waiting for feedback about that approach

github-actions · 2023-08-07T09:52:17Z

New image:
quay.io/netobserv/network-observability-console-plugin:0219ec4

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=0219ec4 make set-plugin-image

openshift-ci · 2023-08-08T07:02:05Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress label Jul 26, 2023

jpinsonneau force-pushed the 1216 branch from de633b7 to b2e54a6 Compare July 26, 2023 11:07

jpinsonneau marked this pull request as ready for review July 26, 2023 11:13

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 26, 2023

jpinsonneau requested review from jotak, msherif1234 and OlivierCazade July 26, 2023 11:13

msherif1234 reviewed Jul 26, 2023

View reviewed changes

jotak reviewed Jul 27, 2023

View reviewed changes

openshift-ci bot assigned msherif1234 Jul 31, 2023

openshift-ci bot added the lgtm label Jul 31, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 31, 2023

jpinsonneau added 6 commits August 3, 2023 09:04

addressed feedback + improved code

5782329

remove tcp mentions

9ffd0d6

doc url on filters examples

17a89b6

icmp v6

5485cb6

extra doc urls

763a135

fmt & lint

575b9b0

jpinsonneau force-pushed the 1216 branch from fa59bb5 to 575b9b0 Compare August 3, 2023 07:05

openshift-ci bot removed the lgtm label Aug 3, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 3, 2023

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 3, 2023

openshift-ci bot added the lgtm label Aug 3, 2023

pkt drop filters availability

03d6282

openshift-ci bot removed the lgtm label Aug 4, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 4, 2023

show others fix + kebab

d8ab9fd

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 4, 2023

missing options DNS filtering

c15abab

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 7, 2023

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 7, 2023

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 7, 2023

bytes / packets zeros

6e698a5

jpinsonneau force-pushed the 1216 branch from 06ed39b to 6e698a5 Compare August 7, 2023 09:47

netobserv deleted a comment from github-actions bot Aug 7, 2023

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Aug 7, 2023

jpinsonneau added lgtm approved labels Aug 8, 2023

openshift-merge-robot merged commit 99ba7d1 into netobserv:main Aug 8, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-1216 PacketDrop enhancements #362

NETOBSERV-1216 PacketDrop enhancements #362

jpinsonneau commented Jul 26, 2023 •

edited

Loading

codecov bot commented Jul 26, 2023 •

edited

Loading

msherif1234 Jul 26, 2023

jpinsonneau Jul 27, 2023

jpinsonneau Jul 27, 2023

msherif1234 Jul 26, 2023

jpinsonneau Jul 27, 2023

msherif1234 Jul 26, 2023

jpinsonneau Jul 27, 2023

msherif1234 Jul 26, 2023

jpinsonneau Jul 27, 2023

jpinsonneau Jul 27, 2023

msherif1234 Jul 26, 2023

jpinsonneau Jul 26, 2023

msherif1234 Jul 26, 2023

jotak Jul 27, 2023

jpinsonneau Jul 27, 2023

jotak Jul 27, 2023

jpinsonneau Jul 27, 2023

jotak commented Jul 27, 2023

jpinsonneau commented Jul 27, 2023

msherif1234 commented Jul 31, 2023

Amoghrd commented Jul 31, 2023

github-actions bot commented Jul 31, 2023

jpinsonneau commented Aug 3, 2023

github-actions bot commented Aug 3, 2023

msherif1234 commented Aug 3, 2023

openshift-ci bot commented Aug 4, 2023

jpinsonneau commented Aug 4, 2023 •

edited

Loading

jpinsonneau commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

openshift-ci bot commented Aug 8, 2023

		- ${t('A _LINUX_TCP_STATES_H number like 1, 2, 3')}
		- ${t('A _LINUX_TCP_STATES_H TCP name like ESTABLISHED, SYN_SENT, SYN_RECV')}`,

NETOBSERV-1216 PacketDrop enhancements #362

NETOBSERV-1216 PacketDrop enhancements #362

Conversation

jpinsonneau commented Jul 26, 2023 • edited Loading

codecov bot commented Jul 26, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jotak commented Jul 27, 2023

jpinsonneau commented Jul 27, 2023

msherif1234 commented Jul 31, 2023

Amoghrd commented Jul 31, 2023

github-actions bot commented Jul 31, 2023

jpinsonneau commented Aug 3, 2023

github-actions bot commented Aug 3, 2023

msherif1234 commented Aug 3, 2023

openshift-ci bot commented Aug 4, 2023

jpinsonneau commented Aug 4, 2023 • edited Loading

jpinsonneau commented Aug 7, 2023

github-actions bot commented Aug 7, 2023

openshift-ci bot commented Aug 8, 2023

jpinsonneau commented Jul 26, 2023 •

edited

Loading

codecov bot commented Jul 26, 2023 •

edited

Loading

jpinsonneau commented Aug 4, 2023 •

edited

Loading