NETOBSERV-1025: include back and forth traffic #343

jotak · 2023-06-13T13:39:45Z

Replace "Common" filters with an option to include return traffic.
Add a "swap src/dst filters" button

Outdated screenshots: for up-to-date screenshot, cf comment #343 (comment)

New stuff on the UI:

New filters per source/dest in topology:

codecov · 2023-06-13T13:46:15Z

Codecov Report

Merging #343 (443f93b) into main (e6d08c3) will decrease coverage by 5.90%.
The diff coverage is 37.07%.

@@            Coverage Diff             @@
##             main     #343      +/-   ##
==========================================
- Coverage   58.61%   52.71%   -5.90%     
==========================================
  Files         151       29     -122     
  Lines        6635     1857    -4778     
  Branches      804        0     -804     
==========================================
- Hits         3889      979    -2910     
+ Misses       2524      783    -1741     
+ Partials      222       95     -127

Flag	Coverage Δ
uitests	`?`
unittests	`52.71% <37.07%> (-5.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/handler/lokiclientmock/loki_client_mock.go	`0.00% <0.00%> (ø)`
pkg/model/fields/fields.go	`100.00% <ø> (ø)`
pkg/loki/flow_query.go	`73.83% <36.17%> (-11.12%)`	⬇️
pkg/loki/filter.go	`62.85% <41.66%> (-20.80%)`	⬇️
pkg/loki/topology_query.go	`65.60% <48.27%> (-8.24%)`	⬇️
pkg/handler/topology.go	`55.43% <83.33%> (+0.48%)`	⬆️
pkg/handler/flows.go	`66.03% <100.00%> (+0.32%)`	⬆️
pkg/handler/frontend-config.go	`35.48% <100.00%> (+4.44%)`	⬆️

... and 123 files with indirect coverage changes

jotak · 2023-06-13T14:35:02Z

Just figured out that loading UI state from URL params seem quite broken but not sure yet if it's my PR or if also broken on main

jotak · 2023-06-30T15:45:20Z

Still some issues to fix:

It seems modifying filters from the main toolbar isn't reflected on topology node filters
I changed some stuff regarding to hovers as it was triggering complete refresh, recreating the whole nodes; need to make sure if that's ok. Not only it was making rendering less efficient, but also it was preventing the contextual menu to stay open, being hidden immediately after appearing.

Edit: fixed

openshift-ci-robot · 2023-07-04T14:44:37Z

@jotak: This pull request references NETOBSERV-1025 which is a valid jira issue.

In response to this:

Replace "Common" filters with an option to include return traffic.

Add a "swap src/dst filters" button

New stuff on the UI:

New filters per source/dest in topology:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jpinsonneau · 2023-07-07T10:25:18Z

web/src/components/filters/filter-action-links.tsx

+      {showSwap && (
+        <Tooltip content={t('Swap source and destination filters')}>
+          <Button id="swap-filters-button" data-test="swap-filters-button" variant="link" onClick={swapFilters}>
+            {t('Swap filters')}
+          </Button>
+        </Tooltip>
+      )}


These items are taking too much width so we should consider changing the way we display them to allow being column flexed

network-observability-console-plugin/web/src/components/netflow-traffic.tsx

Lines 844 to 872 in f37382e

const filtersExtraControl = () => {

return (

<Dropdown

data-test="more-options-dropdown"

id="more-options-dropdown"

onSelect={() => setFiltersOverflowMenuOpen(false)}

toggle={

<Button

data-test="more-options-button"

id="more-options-button"

variant="link"

className="overflow-button"

icon={<EllipsisVIcon />}

onClick={() => setFiltersOverflowMenuOpen(!isFilterOverflowMenuOpen)}

>

{t('More options')}

</Button>

}

isOpen={isFilterOverflowMenuOpen}

dropdownItems={[

<DropdownGroup key="fullscreen-group" label={t('View')}>

<DropdownItem key="fullscreen" onClick={() => setFullScreen(!isFullScreen)}>

{isFullScreen ? t('Collapse') : t('Expand')}

</DropdownItem>

</DropdownGroup>

]}

/>

);

};

or having a responsive dropdown similar to "more option"

jpinsonneau · 2023-07-07T10:27:53Z

web/src/components/filters/filters-toolbar.tsx

+              <ToolbarItem className="flex-start" id="back-and-forth-item">
+                <Tooltip content={t('Include return traffic, with swapped source and destination filters')}>
+                  <Checkbox
+                    data-test="back-and-forth"
+                    id="back-and-forth"
+                    onChange={setBackAndForth}
+                    isChecked={filters.backAndForth}
+                    label={t('Back and forth')}
+                    aria-label={t('Back and forth')}
                  />
-                  {getFilterControl()}
-                </InputGroup>
-                <FilterHints def={selectedFilter} />
-              </div>
-            </Tooltip>
-          </ToolbarItem>
+                </Tooltip>
+              </ToolbarItem>


Same here maybe it's the time to allow back and forth and show / hide filters in the "more options" dropdown ?

or maybe place the back and forth check somewhere else ?

in the query options (but would be less visible)

at swap filters level (would be more consistent to me since it's related to existing filters)

jpinsonneau · 2023-07-07T11:29:39Z

web/src/model/topology.ts

 const getFilterDefValue = (nodeType: NodeType, fields: Partial<TopologyMetricPeer>, dir: FilterDir, t: TFunction) => {
  let def: FilterDefinition | undefined;
  let value: string | undefined;
  if (fields.resource && fields.namespace) {
-    def = findFilter(t, dir === 'src' ? 'src_resource' : dir === 'dst' ? 'dst_resource' : 'resource')!;
+    def = findFilter(t, dir === 'src' ? 'src_resource' : 'dst_resource')!;


Suggested change

def = findFilter(t, dir === 'src' ? 'src_resource' : 'dst_resource')!;

def = findFilter(t, `${dir}_resource`)!;

jpinsonneau · 2023-07-07T11:30:00Z

web/src/model/topology.ts

+    def = findFilter(t, dir === 'src' ? 'src_host_name' : 'dst_host_name')!;
    value = `"${fields.hostName || fields.resource?.name}"`;
  } else if (nodeType === 'namespace' && (fields.namespace || fields.resource)) {
-    def = findFilter(t, dir === 'src' ? 'src_namespace' : dir === 'dst' ? 'dst_namespace' : 'namespace')!;
+    def = findFilter(t, dir === 'src' ? 'src_namespace' : 'dst_namespace')!;
    value = `"${fields.namespace || fields.resource?.name}"`;
  } else if (nodeType === 'resource' && fields.resource) {
-    def = findFilter(t, dir === 'src' ? 'src_name' : dir === 'dst' ? 'dst_name' : 'name')!;
+    def = findFilter(t, dir === 'src' ? 'src_name' : 'dst_name')!;
    value = `"${fields.resource.name}"`;
  } else if (nodeType === 'owner' && fields.owner) {
-    def = findFilter(t, dir === 'src' ? 'src_owner_name' : dir === 'dst' ? 'dst_owner_name' : 'owner_name')!;
+    def = findFilter(t, dir === 'src' ? 'src_owner_name' : 'dst_owner_name')!;
    value = `"${fields.owner.name}"`;
  } else if (fields.addr) {
-    def = findFilter(t, dir === 'src' ? 'src_address' : dir === 'dst' ? 'dst_address' : 'address')!;
+    def = findFilter(t, dir === 'src' ? 'src_address' : 'dst_address')!;


Similar comment than above

jpinsonneau · 2023-07-07T11:41:33Z

web/src/model/filters.ts

 export type FilterId =
-  | 'namespace'
  | 'src_namespace'
  | 'dst_namespace'
-  | 'name'
  | 'src_name'
  | 'dst_name'
-  | 'kind'
  | 'src_kind'
  | 'dst_kind'
-  | 'owner_name'
  | 'src_owner_name'
  | 'dst_owner_name'
-  | 'resource'
  | 'src_resource'
  | 'dst_resource'
-  | 'address'
  | 'src_address'
  | 'dst_address'
-  | 'mac'
  | 'src_mac'
  | 'dst_mac'
-  | 'port'
  | 'src_port'
  | 'dst_port'
-  | 'host_address'
  | 'src_host_address'
  | 'dst_host_address'
-  | 'host_name'
  | 'src_host_name'
  | 'dst_host_name'
  | 'protocol'


Instead of managing each src / dst you can defined targeted filters separately and then build the final type as:

export type TargetedFilterId = | 'namespace' | 'name' | 'kind' | 'owner_name' | 'resource' | 'address' | 'mac' | 'port' | 'host_address' | 'host_name'; export type FilterId = | `src_${TargetedFilterId}` | `dst_${TargetedFilterId}` | 'protocol' | 'type' | 'id';

waw I didn't know we could infer a type like that, that's cool!

jpinsonneau · 2023-07-07T11:44:59Z

web/src/utils/filter-definitions.ts

-  srcEncoder: FiltersEncoder,
-  dstEncoder: FiltersEncoder
-): FilterDefinition[] => {
+const withDest = (src: FilterDefinition, dstEncoder: FiltersEncoder): FilterDefinition[] => {


Following TargetedFilterId approach you can keep the previous peers approach that takes TargetedFilterId + srcEncoder + dstEncoder as arguments

I feel like it's mor ecomplicated... e.g. here's what I would have:

const peers = ( base: Omit<Omit<FilterDefinition, 'id'>, 'encoder'> & { id: TargetedFilterId }, srcEncoder: FiltersEncoder, dstEncoder: FiltersEncoder ): FilterDefinition[] => { return [ { ...base, id: ('src_' + base.id) as FilterId, category: FilterCategory.Source, encoder: srcEncoder }, { ...base, id: ('dst_' + base.id) as FilterId, category: FilterCategory.Destination, encoder: dstEncoder } ]; };

The double Omit is a bit ugly, I felt like the new approach that takes a fully built FilterDefinition and mirroring it was nicer

What about split and extend the FilterDefinition interface like:

export interface FilterPeerDefinition { targetedId: TargetedFilterId; name: string; component: FilterComponent; category: FilterCategory; getOptions: (value: string) => Promise<FilterOption[]>; validate: (value: string) => { val?: string; err?: string }; checkCompletion?: (value: string, selected: string) => { completed: boolean; option: FilterOption }; autoCompleteAddsQuotes?: boolean; hint?: string; examples?: string; placeholder?: string; // overlap tells if the type of entity referred to with this filter may have overlapping (duplicated) // flows when querying for returned traffic (back and forth) - they result in slightly more complicated queries. overlap: boolean; } export interface FilterDefinition extends FilterPeerDefinition { id: FilterId; encoder: FiltersEncoder; }

Then you don't need the Omit part anymore. You can also use back quotes for ids to avoid the casts:

const peers = ( base: FilterPeerDefinition, srcEncoder: FiltersEncoder, dstEncoder: FiltersEncoder ): FilterDefinition[] => { return [ { ...base, id: `src_${base.targetedId}`, category: FilterCategory.Source, encoder: srcEncoder }, { ...base, id: `dst_${base.targetedId}`, category: FilterCategory.Destination, encoder: dstEncoder }, ]; };

I'll let you pick what you prefer 😉 no blocks either way for me

jpinsonneau · 2023-07-07T11:55:42Z

web/src/model/flow-query.ts

+
+const determineOverlap = (orig: Filter[], swapped: Filter[]): { overlaps: Filter[]; cancelSwap: boolean } => {
+  // With "back and forth", the input query is "doubled" with an analoguous query that captures the return traffic
+  // Overlap detection consists in excluding from that added query the overlapping part, by adding the opposite of the 1st query to the second


Nice ASCII art 😄

I'm fine increasing the max-len to 170 if needed

jpinsonneau · 2023-07-07T12:56:10Z

Thanks for this improvment @jotak ! It really brings a better UX. I feel it needs some small changes to makes it good to go:

The Back and forth option position may be better
The chips display should adapt accordingly
- especially using match any, we should remove the Source / Destination texts and merge filters accordingly
- but also for all using a new icon and grouping by filter

jotak · 2023-07-11T16:31:37Z

Thanks @jpinsonneau for the feedback!
I'll dig more into the layout issues tomorrow
On chips display, sounds like a good idea to have a bidirectional arrow between elements but I think it raises a couple of questions / problems: how to do that when source and dest are from different chip groups (e.g. traffic between a pod and a namespace). I guess we would need to rearrange all the chips layout ... I'd rather keep that for a follow-up

jotak · 2023-07-12T14:06:03Z

@jpinsonneau I took the opportunity to change a little bit the design, looks quite better I think:

jpinsonneau · 2023-07-13T14:11:41Z

@jpinsonneau I took the opportunity to change a little bit the design, looks quite better I think

That's cool ! thanks ! Do you want to address the chips refactoring in a followup ?

jotak · 2023-07-13T16:28:43Z

@jpinsonneau I took the opportunity to change a little bit the design, looks quite better I think

That's cool ! thanks ! Do you want to address the chips refactoring in a followup ?

Yeah I guess so, let me create a task
Edit: https://issues.redhat.com/browse/NETOBSERV-1189

Replace "Common" filters with an option to include return traffic. Add swapping feature reuse swapFilters func Split filters chips / toolbar in two files Fix filters not reloaded from url Allow faster frontend builds New make commands to build faster: - Build locally with `make build-frontend-fast` - Build image with `BUILDSCRIPT=:fast make images` The drawback is potentially not catching errors related to using dependencies (e.g. patternfly ...); that's the reason why I prefer to keep the default build as a "full compiling" build. But these new targets can save developer's time when doing frequent iterative changes. In any case, if a compile error is uncaught by the developer, it should be caught by the CI build, which is unchanged. Create filter menu in topology - Filter by source or dest - Stepping into turns on back'n forth - Split decorators in new files / new react components - Make more use of react callbacks - Avoid redrawing everything on hover Fix issues with back & forth Need to process differently symetric filters vs non-symetric ones, ie. a filter such as "srcns=A, dstns=B", when applied back and forth, becomes "srcns=A, dstns=B OR (dstns=A, srcns=B, srcns!=A, dstns!=B)" (the "!=" bits are to exclude overlaps ie. counting traffic twice). But when the query is symetric (src==dst, such as "srcns=A, dstns=A") we can't apply the same logic; in this particular example, the "swapped" query will be ignored because fully overlapping the initial query. Fix missing useEffect for filters drilldown in tpology review feedback Move "back and forth" to chip links and more ... - Move to chips links - Change design, using link+icon - Use Overflow menu - Create a new component LinksOverflow - Create a new component MaybeTooltip Fix fmt

OlivierCazade

LGTM

jotak · 2023-07-19T12:13:55Z

/approve

jotak · 2023-07-19T12:14:36Z

/approve cancel

jotak · 2023-07-19T12:15:42Z

.. sorry .. cancelling approval (I was too eager to merge it and forgot QE approval 😇)

openshift-ci-robot · 2023-07-19T12:18:46Z

@jotak: This pull request references NETOBSERV-1025 which is a valid jira issue.

In response to this:

Replace "Common" filters with an option to include return traffic.

Add a "swap src/dst filters" button

Outdated screenshots: for up-to-date screenshot, cf comment #343 (comment)

New stuff on the UI:

New filters per source/dest in topology:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-07-19T12:19:01Z

@jotak: This pull request references NETOBSERV-1025 which is a valid jira issue.

In response to this:

Replace "Common" filters with an option to include return traffic.

Add a "swap src/dst filters" button

Outdated screenshots: for up-to-date screenshot, cf comment #343 (comment)

New stuff on the UI:

New filters per source/dest in topology:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

memodi · 2023-07-19T17:06:00Z

/ok-to-test

github-actions · 2023-07-19T17:09:37Z

New image:
quay.io/netobserv/network-observability-console-plugin:6174fd8

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=6174fd8 make set-plugin-image

jotak · 2023-07-20T15:24:41Z

/approve
=> any additional feedback can be done via jira

openshift-ci · 2023-07-20T15:24:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jotak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jotak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress label Jun 13, 2023

jotak force-pushed the bnf branch from 479afe5 to dbca032 Compare June 15, 2023 05:56

openshift-merge-robot added the needs-rebase label Jun 16, 2023

jotak force-pushed the bnf branch from dbca032 to 1df7525 Compare June 19, 2023 12:03

openshift-merge-robot removed the needs-rebase label Jun 19, 2023

jotak marked this pull request as ready for review June 19, 2023 12:03

openshift-ci bot removed the do-not-merge/work-in-progress label Jun 19, 2023

jotak requested review from OlivierCazade and jpinsonneau and removed request for OlivierCazade and jpinsonneau June 19, 2023 12:03

jotak force-pushed the bnf branch from 1df7525 to fd5ca5d Compare June 22, 2023 16:35

jotak force-pushed the bnf branch from fd5ca5d to 2b147ae Compare June 30, 2023 15:32

jotak requested review from jpinsonneau and OlivierCazade July 3, 2023 09:40

jotak changed the title ~~NETOBSERV-1025 include back and forth traffic~~ NETOBSERV-1025: include back and forth traffic Jul 4, 2023

openshift-ci-robot added the jira/valid-reference label Jul 4, 2023

jpinsonneau reviewed Jul 7, 2023

View reviewed changes

openshift-merge-robot added the needs-rebase label Jul 11, 2023

jotak force-pushed the bnf branch from 8d87f37 to 77b64f2 Compare July 12, 2023 07:46

openshift-merge-robot removed the needs-rebase label Jul 12, 2023

jotak force-pushed the bnf branch from 77b64f2 to 875ffb1 Compare July 12, 2023 14:04

jotak added 2 commits July 17, 2023 17:25

Fix issues post-rebase

443f93b

jotak force-pushed the bnf branch from 1e25daa to 443f93b Compare July 17, 2023 16:22

OlivierCazade approved these changes Jul 19, 2023

View reviewed changes

openshift-ci bot assigned OlivierCazade Jul 19, 2023

openshift-ci bot added the lgtm label Jul 19, 2023

openshift-ci bot added the approved label Jul 19, 2023

openshift-ci bot removed the approved label Jul 19, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 19, 2023

openshift-ci bot added the approved label Jul 20, 2023

openshift-merge-robot merged commit e069f27 into netobserv:main Jul 20, 2023
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-1025: include back and forth traffic #343

NETOBSERV-1025: include back and forth traffic #343

jotak commented Jun 13, 2023 •

edited

Loading

codecov bot commented Jun 13, 2023 •

edited

Loading

jotak commented Jun 13, 2023

jotak commented Jun 30, 2023 •

edited

Loading

openshift-ci-robot commented Jul 4, 2023 •

edited by openshift-ci bot

Loading

jpinsonneau Jul 7, 2023

jpinsonneau Jul 7, 2023

jpinsonneau Jul 7, 2023

jpinsonneau Jul 7, 2023

jpinsonneau Jul 7, 2023

jpinsonneau Jul 7, 2023

jotak Jul 11, 2023

jpinsonneau Jul 7, 2023

jotak Jul 11, 2023

jpinsonneau Jul 12, 2023

jpinsonneau Jul 7, 2023

jpinsonneau commented Jul 7, 2023

jotak commented Jul 11, 2023

jotak commented Jul 12, 2023

jpinsonneau commented Jul 13, 2023

jotak commented Jul 13, 2023 •

edited

Loading

OlivierCazade left a comment

jotak commented Jul 19, 2023

jotak commented Jul 19, 2023

jotak commented Jul 19, 2023

openshift-ci-robot commented Jul 19, 2023 •

edited by openshift-ci bot

Loading

Outdated screenshots: for up-to-date screenshot, cf comment #343 (comment)

openshift-ci-robot commented Jul 19, 2023 •

edited by openshift-ci bot

Loading

memodi commented Jul 19, 2023

github-actions bot commented Jul 19, 2023

jotak commented Jul 20, 2023

openshift-ci bot commented Jul 20, 2023

	const filtersExtraControl = () => {
	return (
	<Dropdown
	data-test="more-options-dropdown"
	id="more-options-dropdown"
	onSelect={() => setFiltersOverflowMenuOpen(false)}
	toggle={
	<Button
	data-test="more-options-button"
	id="more-options-button"
	variant="link"
	className="overflow-button"
	icon={<EllipsisVIcon />}
	onClick={() => setFiltersOverflowMenuOpen(!isFilterOverflowMenuOpen)}
	>
	{t('More options')}
	</Button>
	}
	isOpen={isFilterOverflowMenuOpen}
	dropdownItems={[
	<DropdownGroup key="fullscreen-group" label={t('View')}>
	<DropdownItem key="fullscreen" onClick={() => setFullScreen(!isFullScreen)}>
	{isFullScreen ? t('Collapse') : t('Expand')}
	</DropdownItem>
	</DropdownGroup>
	]}
	/>
	);
	};

	def = findFilter(t, dir === 'src' ? 'src_resource' : 'dst_resource')!;
	def = findFilter(t, `${dir}_resource`)!;

NETOBSERV-1025: include back and forth traffic #343

NETOBSERV-1025: include back and forth traffic #343

Conversation

jotak commented Jun 13, 2023 • edited Loading

codecov bot commented Jun 13, 2023 • edited Loading

Codecov Report

jotak commented Jun 13, 2023

jotak commented Jun 30, 2023 • edited Loading

openshift-ci-robot commented Jul 4, 2023 • edited by openshift-ci bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpinsonneau commented Jul 7, 2023

jotak commented Jul 11, 2023

jotak commented Jul 12, 2023

jpinsonneau commented Jul 13, 2023

jotak commented Jul 13, 2023 • edited Loading

OlivierCazade left a comment

Choose a reason for hiding this comment

jotak commented Jul 19, 2023

jotak commented Jul 19, 2023

jotak commented Jul 19, 2023

openshift-ci-robot commented Jul 19, 2023 • edited by openshift-ci bot Loading

Outdated screenshots: for up-to-date screenshot, cf comment #343 (comment)

openshift-ci-robot commented Jul 19, 2023 • edited by openshift-ci bot Loading

memodi commented Jul 19, 2023

github-actions bot commented Jul 19, 2023

jotak commented Jul 20, 2023

openshift-ci bot commented Jul 20, 2023

jotak commented Jun 13, 2023 •

edited

Loading

codecov bot commented Jun 13, 2023 •

edited

Loading

jotak commented Jun 30, 2023 •

edited

Loading

openshift-ci-robot commented Jul 4, 2023 •

edited by openshift-ci bot

Loading

jotak commented Jul 13, 2023 •

edited

Loading

openshift-ci-robot commented Jul 19, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jul 19, 2023 •

edited by openshift-ci bot

Loading