Doc: regenerate API refs (FlowCollector + Flow Format) #550

jotak · 2024-01-30T12:54:05Z

Flow format is now generated from a fields.yaml file stored here, instead of the console plugin's type definition previously used, as it became more likely to be outdated

A check is added to make sure any field referenced in the frontend config is also defined in fields.yaml

jotak · 2024-01-30T12:57:37Z

FYI @skrthomas
(let's wait a bit that the dev team reviews it, but that should set the ground for 1.5 API doc)
Also, you'll see that I completely changed the layout of the Flow Format spec: this is because I don't use anymore the previous tooling and now we're more free for designing the way we want. So I've moved away from the previous flat text format, to use a table instead. Pls let me know how it plays with the doc integration...

jotak · 2024-01-30T12:58:46Z

to see the new Flow Format spec in github's renderer: https://github.com/netobserv/network-observability-operator/blob/a5606eef057a8e40c798d1cc2609cc8998468ef0/docs/flows-format.adoc

jpinsonneau · 2024-01-30T14:01:37Z

docs/fields.yaml

@@ -0,0 +1,129 @@
+# Fields definition, used to generate documentation


Can we add this to the static-frontend-config ?

I would like to get rid of ipfix.ts file from console plugin repo in the end.
We can also rely on these definitions to check if field is a label and add types since the query engine check for numbers.

I can surely move that to the config, ok, but it will stay unused in code for the time being, right? at least for this PR

well then I can also change my flow-format-validation script into a go test

Sure, that's a long run refactor 😉

skrthomas · 2024-01-31T18:56:43Z

docs/flows-format.adoc

+
+The "Filter ID" column shows which related name to use when defining Quick Filters (see `spec.consolePlugin.quickFilters` in the `FlowCollector` specification).
+
+The "Loki label" column is useful when querying Loki directly: label fields need to be selected using link:https://grafana.com/docs/loki/latest/logql/log_queries/#log-stream-selector[stream selectors].


Do we still need to write this too:

If you are reading this specification as a reference for the Kafka export feature, you must treat all Labels and Fields as regular fields and ignore any distinctions between them that are specific to Loki.

No I removed that because now it's not organized by Fields and Labels anymore - it's directly written for Kafka exporter users

Thanks Joel. I re-copied your latest IBMSG tweaks to my PR.

re-copied and pushed latest to my PR.

jpinsonneau · 2024-02-02T09:20:09Z

controllers/flowcollector_controller_certificates_test.go

+		// Flaky assertion here, quite annoying, some weirdness with `envtest` maybe?
+		// Anyway it doesn't bring much value. Disabling it for the time being


I wonder if this is not tied to the parallelization of the tests. Currently, we are not setting any parameter wich means tests from different packages can run concurently.

-parallel n Allow parallel execution of test functions that call t.Parallel, and fuzz targets that call t.Parallel when running the seed corpus. The value of this flag is the maximum number of tests to run simultaneously. While fuzzing, the value of this flag is the maximum number of subprocesses that may call the fuzz function simultaneously, regardless of whether T.Parallel is called. By default, -parallel is set to the value of GOMAXPROCS. Setting -parallel to values higher than GOMAXPROCS may cause degraded performance due to CPU contention, especially when fuzzing. Note that -parallel only applies within a single test binary. The 'go test' command may run tests for different packages in parallel as well, according to the setting of the -p flag (see 'go help build').

Adding -p 1 to the go test command ensure each tests to be run separately but highly increase the testing time.
We could also try to use the t.Parallel call in each test checking for Cleanup
Lastly, we can also force different FlowCollector instance name. It's hacky but may work here.

FYI this PR moved FLP and Monitoring controller tests outside of controllers package so we parallelize creation / deletion of FlowCollector between the 3 packages.

So, after many tests, I didn't find issues with parallelism (although I don't guarantee that there aren't any). My guess is that it's ok with parallelism because each controller set up its own envtest environment so they run in isolation. And tbh if this isn't the case, I'm quite surprised to not see many more failures at random places, whereas here this is always the same single test that is failing.
Anyway my understanding of this test failure is finally that this is a timing issue, the CR taking sometimes more time to be actually deleted than the test timeout. So I have this PR for a fix: #557 (and I'll remove my change here)

I reverted the change here, cf the mentioned PR above

controllers/consoleplugin/config/static-frontend-config.yaml

jpinsonneau · 2024-02-02T09:25:42Z

controllers/consoleplugin/config/static-frontend-config.yaml

+  - name: PktDropLatestState
+    type: string
+    description: TCP state on last dropped packet
+    filter: pkt_drop_state # couldn't guess from config
+  - name: PktDropLatestDropCause
+    type: string
+    description: Latest drop cause
+    filter: pkt_drop_cause # couldn't guess from config


Suggested change

- name: PktDropLatestState

type: string

description: TCP state on last dropped packet

filter: pkt_drop_state # couldn't guess from config

- name: PktDropLatestDropCause

type: string

description: Latest drop cause

filter: pkt_drop_cause # couldn't guess from config

- name: PktDropLatestState

type: string

description: TCP state on last dropped packet

- name: PktDropLatestDropCause

type: string

description: Latest drop cause

We keep filter definition in the columns right ?

jpinsonneau · 2024-02-02T09:30:36Z

controllers/consoleplugin/config/config.go

+type FieldConfig struct {
+	Name        string `yaml:"name" json:"name"`
+	Type        string `yaml:"type" json:"type"`
+	Description string `yaml:"description" json:"description"`
+	LokiLabel   bool   `yaml:"lokiLabel,omitempty" json:"lokiLabel,omitempty"`
+	Filter      string `yaml:"filter,omitempty" json:"filter,omitempty"`
+}


Suggested change

type FieldConfig struct {

Name string `yaml:"name" json:"name"`

Type string `yaml:"type" json:"type"`

Description string `yaml:"description" json:"description"`

LokiLabel bool `yaml:"lokiLabel,omitempty" json:"lokiLabel,omitempty"`

Filter string `yaml:"filter,omitempty" json:"filter,omitempty"`

}

type FieldConfig struct {

Name string `yaml:"name" json:"name"`

Type string `yaml:"type" json:"type"`

Description string `yaml:"description" json:"description"`

LokiLabel bool `yaml:"lokiLabel,omitempty" json:"lokiLabel,omitempty"`

}

I don't think we should define filters in the field configuration. Filters are already defined in their own section and tied to columns when needed.

they aren't defined from there, it's only a reference. In fact, for almost all the fields I didn't need that reference as I could retrieve them from the columns. But there are two exceptions: for drop state and drop cause. These ones don't have the filter defined in column, I guess they must be some kind of special cases in the front implementation? Because of them, as I want to document the existence of these filters, I had to add an explicit reference.

I can rename the field filterRef to clarify

There are multiple exceptions indeed such as:

TimeFlowStartMs / TimeFlowEndMs

PktDropLatestState / PktDropLatestDropCause

Duplicate

So either we refactor these in the plugin to rely on filters or we force these filter ids from the field config as you initially planned.

I don't see filters for TimeFlowStartMs / TimeFlowEndMs or Duplicate, while they exist for PktDrop stuff (in the filters yaml)

Perhaps a refactoring will be necessary moving forward, but for this PR I didn't intend to do any code change, just extracting whatever data I can for the doc

hack/asciidoc-flows-gen.sh

openshift-ci · 2024-02-02T14:04:31Z

@jotak: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-operator	`66920aa`	link	false	`/test e2e-operator`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

codecov · 2024-02-02T17:35:18Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (5d91297) 57.87% compared to head (c6b6db9) 57.89%.

❗ Current head c6b6db9 differs from pull request most recent head 8f04d2b. Consider uploading reports for the commit 8f04d2b to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #550      +/-   ##
==========================================
+ Coverage   57.87%   57.89%   +0.02%     
==========================================
  Files          73       73              
  Lines        9543     9543              
==========================================
+ Hits         5523     5525       +2     
  Misses       3684     3684              
+ Partials      336      334       -2

Flag	Coverage Δ
unittests	`57.89% <ø> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Flow format is now generated from a fields.yaml file stored here, instead of the console plugin's type definition previously used, as it became more likely to be outdated A check is added to make sure any field referenced in the frontend config is also defined in fields.yaml

Removed hack/flows-format-validation.sh, replaced with a go test Add type information

- for instance => for example - may => might Also use head directive ":_mod-docs-content-type: REFERENCE"

Co-authored-by: Julien Pinsonneau <91894519+jpinsonneau@users.noreply.github.com>

jotak · 2024-02-05T08:32:16Z

@jpinsonneau PR updated (doc regenerated after your updates, reverted flaky test change)

jpinsonneau

Looks good for the doc, thanks @jotak !

jotak · 2024-02-05T09:14:28Z

thanks
/approve

openshift-ci · 2024-02-05T09:14:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jotak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jotak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Doc: regenerate API refs (FlowCollector + Flow Format) Flow format is now generated from a fields.yaml file stored here, instead of the console plugin's type definition previously used, as it became more likely to be outdated A check is added to make sure any field referenced in the frontend config is also defined in fields.yaml * fix validation script * Merge fields.yaml in static-frontend-config.yaml Removed hack/flows-format-validation.sh, replaced with a go test Add type information * Minor tweaks for IBM style guide - for instance => for example - may => might Also use head directive ":_mod-docs-content-type: REFERENCE" * Update controllers/consoleplugin/config/static-frontend-config.yaml Co-authored-by: Julien Pinsonneau <91894519+jpinsonneau@users.noreply.github.com> * Update hack/asciidoc-flows-gen.sh Co-authored-by: Julien Pinsonneau <91894519+jpinsonneau@users.noreply.github.com> * Revert changes on flaky test * regen doc --------- Co-authored-by: Julien Pinsonneau <91894519+jpinsonneau@users.noreply.github.com>

* Doc: regenerate API refs (FlowCollector + Flow Format) Flow format is now generated from a fields.yaml file stored here, instead of the console plugin's type definition previously used, as it became more likely to be outdated A check is added to make sure any field referenced in the frontend config is also defined in fields.yaml * fix validation script * Merge fields.yaml in static-frontend-config.yaml Removed hack/flows-format-validation.sh, replaced with a go test Add type information * Minor tweaks for IBM style guide - for instance => for example - may => might Also use head directive ":_mod-docs-content-type: REFERENCE" * Update controllers/consoleplugin/config/static-frontend-config.yaml * Update hack/asciidoc-flows-gen.sh * Revert changes on flaky test * regen doc --------- Co-authored-by: Julien Pinsonneau <91894519+jpinsonneau@users.noreply.github.com>

jotak requested review from msherif1234, OlivierCazade and jpinsonneau January 30, 2024 12:54

jpinsonneau reviewed Jan 30, 2024

View reviewed changes

jotak force-pushed the regen-doc branch 2 times, most recently from 3473a2e to 946b71d Compare January 30, 2024 15:01

jotak requested a review from jpinsonneau January 30, 2024 15:10

jpinsonneau mentioned this pull request Jan 31, 2024

NETOBSERV-1382 - Custom metrics from config netobserv/network-observability-console-plugin#431

Merged

10 tasks

skrthomas reviewed Jan 31, 2024

View reviewed changes

This was referenced Feb 1, 2024

Network Observability API documentation updates openshift/openshift-docs#71033

Merged

Doc: regenerate API refs (FlowCollector + Flow Format) [1.5 backport] #556

Merged

jpinsonneau reviewed Feb 2, 2024

View reviewed changes

hack/asciidoc-flows-gen.sh Show resolved Hide resolved

jotak and others added 8 commits February 5, 2024 09:29

fix validation script

f0dc71f

Merge fields.yaml in static-frontend-config.yaml

091ce11

Removed hack/flows-format-validation.sh, replaced with a go test Add type information

Minor tweaks for IBM style guide

528dee8

- for instance => for example - may => might Also use head directive ":_mod-docs-content-type: REFERENCE"

Update controllers/consoleplugin/config/static-frontend-config.yaml

680d5f0

Co-authored-by: Julien Pinsonneau <91894519+jpinsonneau@users.noreply.github.com>

Update hack/asciidoc-flows-gen.sh

b41140e

Co-authored-by: Julien Pinsonneau <91894519+jpinsonneau@users.noreply.github.com>

Revert changes on flaky test

6f0834c

regen doc

8f04d2b

jotak force-pushed the regen-doc branch from c6b6db9 to 8f04d2b Compare February 5, 2024 08:30

jpinsonneau approved these changes Feb 5, 2024

View reviewed changes

openshift-ci bot assigned jpinsonneau Feb 5, 2024

openshift-ci bot added the lgtm label Feb 5, 2024

openshift-ci bot added the approved label Feb 5, 2024

openshift-merge-bot bot merged commit 34fd0ae into netobserv:main Feb 5, 2024
9 of 10 checks passed

jotak deleted the regen-doc branch February 8, 2024 09:10

jpinsonneau mentioned this pull request Feb 19, 2024

NETOBSERV-1503 Console plugin fields config netobserv/network-observability-console-plugin#476

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Doc: regenerate API refs (FlowCollector + Flow Format) #550

Doc: regenerate API refs (FlowCollector + Flow Format) #550

jotak commented Jan 30, 2024

jotak commented Jan 30, 2024

jotak commented Jan 30, 2024

jpinsonneau Jan 30, 2024

jotak Jan 30, 2024

jotak Jan 30, 2024

jpinsonneau Jan 30, 2024

skrthomas Jan 31, 2024

jotak Feb 1, 2024

skrthomas Feb 1, 2024

skrthomas Feb 7, 2024

jpinsonneau Feb 2, 2024

jpinsonneau Feb 2, 2024

jotak Feb 2, 2024

jotak Feb 5, 2024

jpinsonneau Feb 2, 2024

jpinsonneau Feb 2, 2024

jotak Feb 2, 2024

jotak Feb 2, 2024

jpinsonneau Feb 2, 2024

jotak Feb 5, 2024

jotak Feb 5, 2024

openshift-ci bot commented Feb 2, 2024

codecov bot commented Feb 2, 2024 •

edited

Loading

jotak commented Feb 5, 2024

jpinsonneau left a comment

jotak commented Feb 5, 2024

openshift-ci bot commented Feb 5, 2024

		@@ -0,0 +1,129 @@
		# Fields definition, used to generate documentation


		The "Filter ID" column shows which related name to use when defining Quick Filters (see `spec.consolePlugin.quickFilters` in the `FlowCollector` specification).

		The "Loki label" column is useful when querying Loki directly: label fields need to be selected using link:https://grafana.com/docs/loki/latest/logql/log_queries/#log-stream-selector[stream selectors].

		// Flaky assertion here, quite annoying, some weirdness with `envtest` maybe?
		// Anyway it doesn't bring much value. Disabling it for the time being

Doc: regenerate API refs (FlowCollector + Flow Format) #550

Doc: regenerate API refs (FlowCollector + Flow Format) #550

Conversation

jotak commented Jan 30, 2024

jotak commented Jan 30, 2024

jotak commented Jan 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Feb 2, 2024

codecov bot commented Feb 2, 2024 • edited Loading

Codecov Report

jotak commented Feb 5, 2024

jpinsonneau left a comment

Choose a reason for hiding this comment

jotak commented Feb 5, 2024

openshift-ci bot commented Feb 5, 2024

codecov bot commented Feb 2, 2024 •

edited

Loading